vagrant box add fails to unpack a large box

[barkingfoodog]

Vagrant version

Vagrant 2.0.2

Host operating system

macOS 10.11.6 and macOS 10.13.3

Guest operating system

macOS 10.11.6

Vagrantfile

Vagrant.configure("2") do |config|
  config.vm.box = "suran/macos1011"
  config.vm.box_url = "http://download.suran.com/developer/macos1011-0.1.0.box"
  config.vm.box_download_checksum = "b1bb86a7b56bc9a29bc6066d78ae22a76061e158"
  config.vm.box_download_checksum_type = "sha1"
end

Debug output

debug.log

Expected behavior

The machine is booted and accessible.

Actual behavior

An error occurs:

There was an error while executing `VBoxManage`, a CLI used by Vagrant
for controlling VirtualBox. The command and stderr is shown below.

Command: ["import", "/Users/administrator/.vagrant.d/boxes/suran-VAGRANTSLASH-macos1011/0/virtualbox/box.ovf", "--vsys", "0", "--vmname", "macos1011_1520735715622_34133", "--vsys", "0", "--unit", "9", "--disk", "/Users/administrator/VirtualBox VMs/macos1011_1520735715622_34133/macos1011-disk001.vmdk"]

Stderr: 0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Interpreting /Users/administrator/.vagrant.d/boxes/suran-VAGRANTSLASH-macos1011/0/virtualbox/box.ovf...
OK.
0%...
Progress state: VBOX_E_FILE_ERROR
VBoxManage: error: Appliance import failed
VBoxManage: error: Could not open the medium storage unit '/Users/administrator/.vagrant.d/boxes/suran-VAGRANTSLASH-macos1011/0/virtualbox/macos1011-disk001.vmdk'.
VBoxManage: error: VMDK: error reading the magic number in '/Users/administrator/.vagrant.d/boxes/suran-VAGRANTSLASH-macos1011/0/virtualbox/macos1011-disk001.vmdk' (VERR_EOF).
VBoxManage: error: VD: error VERR_VD_VMDK_INVALID_HEADER opening image file '/Users/administrator/.vagrant.d/boxes/suran-VAGRANTSLASH-macos1011/0/virtualbox/macos1011-disk001.vmdk' (VERR_VD_VMDK_INVALID_HEADER)
VBoxManage: error: Details: code VBOX_E_FILE_ERROR (0x80bb0004), component ApplianceWrap, interface IAppliance
VBoxManage: error: Context: "RTEXITCODE handleImportAppliance(HandlerArg *)" at line 886 of file VBoxManageAppliance.cpp

Contents of /Users/administrator/.vagrant.d/boxes/suran-VAGRANTSLASH-macos1011/0/virtualbox/:

total 40
drwxr-xr-x  6 administrator  staff   204 Mar 10 19:43 .
drwxr-xr-x  3 administrator  staff   102 Mar 10 19:43 ..
-rw-r--r--  1 administrator  staff  3276 Mar 10 19:42 Vagrantfile
-rw-r--r--  1 administrator  staff  8993 Mar 10 19:42 box.ovf
-rw-r--r--  1 administrator  staff     0 Mar 10 19:42 macos1011-disk001.vmdk
-rw-r--r--  1 administrator  staff    26 Mar 10 19:43 metadata.json

Steps to reproduce

1. Create the above vagrantfile
2. vagrant up

Workaround

Use GNU tar.

1. brew install gnu-tar
2. sudo cp /usr/local/bin/gtar /opt/vagrant/embedded/bin/bsdtar

Notes

It appears the issue is related to the embedded bsdtar utility being unable to extract this box. The box was created with packer 1.2.1 on the same box using the macOS boxcutter template with a modified cmtool.sh. The only modifications to this script are to pre-install homebrew and salt.

Here is the output from using the embedded bsdtar utility to unpack the box:

$ /opt/vagrant/embedded/bin/bsdtar -v -xzf macos1011-0.1.0.box
x Vagrantfile
x box.ovf
x macos1011-disk001.vmdk
x metadata.json
$ ls -lh Vagrantfile box.ovf macos1011-disk001.vmdk metadata.json
-rw-r--r--  1 administrator  staff   3.2K Mar  6 19:48 Vagrantfile
-rw-r--r--  1 administrator  staff   8.8K Mar  6 19:48 box.ovf
-rw-r--r--  1 administrator  staff     0B Mar  6 19:48 macos1011-disk001.vmdk
-rw-r--r--  1 administrator  staff    26B Mar  6 19:48 metadata.json

Here is the same output using GNU tar:

$ /usr/local/bin/gtar -v -zxf macos1011-0.1.0.box
Vagrantfile
box.ovf
macos1011-disk001.vmdk
metadata.json
$ ls -lh Vagrantfile box.ovf macos1011-disk001.vmdk metadata.json
-rw-r--r--  1 administrator  staff   3.2K Mar  6 19:48 Vagrantfile
-rw-r--r--  1 administrator  staff   8.8K Mar  6 19:48 box.ovf
-rw-r--r--  1 administrator  staff   9.5G Mar  6 19:48 macos1011-disk001.vmdk
-rw-r--r--  1 administrator  staff    26B Mar  6 19:48 metadata.json

This is a recent problem and is not exhibited with a macOS 10.12 box built from the same templates. I suspect this error is to due to the size of the macOS 10.11 box growing from recent security updates.

This may be a packer issue, but the above workaround applies to vagrant, so I figured I'd start here.

[DanHam]

Hi,

@chrisroberts Chiming in as this has the needs-repro tag...

I am experiencing the same issue with vagrant 2.0.3 (and 2.0.2). I suspect @barkingfoodog has already found the root cause - namely the embedded bsdtar failing to unpack the box properly.

My box is a big fat Windows box (approx. 6Gb) that has been built locally with Packer. I can manually extract the box files (using the system tar utility) and run the VM successfully using vmrun -T fusion blah.vmx so this is not an issue with the box files or Packer.

If I manually extract the box files using the system tar utility to the required folder under .vagrant.d/boxes then vagrant up works just fine...

This is the box directory after vagrant box add has done it's thing...

.
└── [        306]  vmware_desktop
    ├── [        983]  Vagrantfile
    ├── [         30]  metadata.json
    ├── [       8684]  windows2016-vmware-iso.nvram
    ├── [          0]  windows2016-vmware-iso.vmdk
    ├── [          0]  windows2016-vmware-iso.vmsd
    ├── [       3515]  windows2016-vmware-iso.vmx
    └── [        277]  windows2016-vmware-iso.vmxf

1 directory, 7 files

... vagrant up unsurprisingly fails since the vmdk file is a little on the slim side!

This is the box directory after manually extracting the contents of the box with tar xvf...
EDIT: My system uses GNU tar (tar (GNU tar) 1.29) by default - not bsdtar

.
└── [        340]  vmware_desktop
    ├── [        983]  Vagrantfile
    ├── [         30]  metadata.json
    ├── [       8684]  windows2016-vmware-iso.nvram
    ├── [13725794304]  windows2016-vmware-iso.vmdk
    ├── [          0]  windows2016-vmware-iso.vmsd
    ├── [       3515]  windows2016-vmware-iso.vmx
    └── [        277]  windows2016-vmware-iso.vmxf

1 directory, 7 files

A vagrant up then runs successfully!

As noted smaller boxes are unaffected by this bug and unpack successfully.

[mwhooker]

Hey folks, some info from the linked packer issue. There was a bug in libarchive < v3.3.2. Not sure when this will make it out to osx, but as @barkingfoodog mentioned, either using gnutar or updating bsdtar to a newer version should solve this. libarchive/libarchive#880

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.