[RFC] Support for running runv without 9pfs

[harche]

Hi,

We love runv for the level of isolation it offers for the critical workloads. Being able to use the docker workflow and combining it with qemu for increased isolation is just brilliant!

As you already know, runv depends on 9p filesystem for passing the container rootfs to the virtual machine. This works great, well for most of the time. But reluctance showed by distributions to officially support it makes it difficult to get this through enterprise customers and hence effectively their critical workloads which runv aims to provide higher isolation. Some of the valid concerns raised for using 9p can be found here, https://access.redhat.com/discussions/1119043.

So as a quick workaround we started thinking of ways to eliminate the use of 9p. Turns out, it's really not that simple to get rid of 9p if you want to pass a folder to the VM. But what if you bundle this rootfs folder into a qcow2 image and attach it to VM? Sure it will add to the bootup time of the VM because now you have to create a new qcow2 image of the rootfs (this can be optimized by caching the qcow2 images).

This PR has changes that add build time support for making runv work without depending on 9p. The regular flow of runv with 9p is not altered and it continues to function the way it is right now. But if you configure it with --without-9p and compile it, resulting runv binary will not use 9p. Overall the modified build instructions will look like,

$ cd $GOPATH/src/github.com/hyperhq
$ git clone https://github.com/hyperhq/runv/
$ cd runv
$ ./autogen.sh
$ ./configure --without-xen --without-9p
$ make
$ sudo make install

In order to generate qcow2 image, the added dependency here is libguestfs-tools. On ubuntu just do, apt-get install libguestfs-tools.

Of course if you want don't want to use 9p, it will require changes in hyperstart too. Right after creating this PR I will raise a PR in hyperstart project to support working without 9p. If you want to use runv with 9p, as is the case right now, the existing hyperstart will work just fine with this runv patch. The way it's implemented is, runv sends a json to hyperstart containing a key sharedDir. This is not required in case you aren't using 9p, so we set it to an empty string. This let's hyperstart know that instead of mounting 9p filesystem, mount the SCSI device that has the rootfs.

Corresponding hyperstart PR, hyperhq/hyperstart#344

Signed-off-by: Harshal Patil harshal.patil@in.ibm.com

[laijs]

Disks should be inserted via special APIs()
the disk path should be specified in the container config.
This will result in doing disk hotplug.
(Avoiding disk hotplug is possible, but it requires some refactor to the code)

[harche]

By disk hotplug, are you suggesting that we could attach a disk to pod VM after it has started? If yes, could you help me point to that API?

Also, the main motivation behind this PR is to make 9pfs optional. The way the code is structured right now makes all pod VMs to have 9pfs.

[laijs]

yes. This is the way how hyperd/hyperctl/hyper.sh work on the images/volumes. We have the APIs ready, only except that runv-cli hasn't call these APIs so far, which can be changed. hyperd/hyperctl/hyper.sh doesn't necessary depend on 9pfs either, we are glad to make it allow to be configured out.

To hotplug add the rootfs image, you can just put the image info into RootVolume
of the ContainerDescription.
https://github.com/hyperhq/runv/blob/master/api/descriptions.pb.go#L99

The code for setting RootVolume might be here:
https://github.com/hyperhq/runv/blob/master/cli/container.go#L76-L77

[gnawux]

@harche here when we create container, we created it with a spec here by protobuf, in which defined a rootVolume with VolumeDescription:

message VolumeDescription {
    string name = 1;
    string source = 2;
    string format = 3; //"raw" (or "qcow2" later) for volume, "vfs" for dir path
    string fstype = 4; //"xfs", "ext4" etc. for block dev, or "dir" for dir path
    VolumeOption options = 8;
    bool dockerVolume = 9;
    bool readOnly = 10;
}

For block devices, you could specify Source as the path of device, Format as raw, and Fstype as xfs or ext4 or any other fs in case the guest kernel supports it.

[harche]

I am trying to implement the suggested approach above, but I am getting following error, https://pastebin.com/raw/UsfXNfdS

The change I made was, https://pastebin.com/raw/KLMxdJ63

[laijs]

any update on the PR? @harche

[harche]

Working on it. I was focusing on hyperhq/hyperstart#345. Now that one is merged, I will spend some time on this one.

[laijs]

@harche I'm struggling to find out a proper user interface for the block device support, do you have any suggestion?

Command line option like runv create --root-device xxx.
Annotations like com.github.kata-containers.storage.root.path=xxx, com.github.kata-containers.storage.root.driver=qcow2
Filetype detection: check the root path to see if it is a block/qcow2/ceph-config-file/...
other spec extension ...

[harche]

@laijs How would this work seamlessly with docker or k8s (with CRI-O)?

IMHO, the disk creation should be part of runv (maybe using some config file to determine if VM should use 9pfs or not). If you make --root-device as a separate option then it won't work out of the box with docker because that argument will be absent.

Our aim is to remove the 9pfs dependency from the entire docker workflow. So as I mentioned earlier, ideally whether to use 9pfs or use a root disk is the decision runv should take just before launching the VM maybe using a config file. If it decides to use root disk then it should create it (from rootfs dir) before proceeding ahead.

You can always have an option --root-disk as you mentioned above for the existing root disks, but I am not sure how it will get used when you plug runv with docker or k8s.

[harche]

@laijs it could be added during the build time, like the changes in this PR.