Ubuntu/Upstart issues as a container/full machine

[kiorky]

One again, i am on my upstart-related bunch of issues while putting the stuff in a docker container, so i think it would be good to have a meta bug for upstart issues and a starter for further documentation.
For now:

• configuration of container is not that editable after start (ports, volumes), see Changeable/Editable volumes / ports #2045
• you need to tweak a bit some jobs, disable others, and emit some virtual events, see along Dockerfiles and shipped with upstart jobs & scripts
• FIXED in "master: restart/reboot/poweroff from within the container won't work unless we keep sys_boot lxc CAP, the only mean to kill/restart if from a docker restart/kill/stop call, see init system issues with shutdown/reboot #1960
• docker top doesnt show any process even if the container is running and well. Indeed when you ssh'in the container, processes are there up and running., see Container started with upstart init wont show anything in docker top #2277
• /etc/hostname & /etc/resolv.conf as they are mounted-readonly are also problematic (status of /etc/hosts and hostname as readwrite (writable) #2068)
• Some base packages wont install/update either because of CAPS or ReadOnly Files. We repackage them on the fly in the dockerfile and mark them as non upgradable.
  • resolvconf
  • ntp
  • fuse
• ntpd from container hit apparmor breakage from host, see : ntpd from ubuntu container is blocked from the host (ubuntu) apparmor policy (surely an aufs+apparmor bug) #2800

To reproduce all this stuff, you can find my debug env here:

• https://github.com/makinacorpus/vms/blob/master/docker/make.sh : this shell help makes images from ubuntu cloudarchives images, plus some modifications (see dockerfiles) to run smoothly on docker.
• notice : https://github.com/makinacorpus/vms/blob/master/docker/makinacorpus/ubuntu_template/lxc-setup.conf
• notice : https://github.com/makinacorpus/vms/blob/master/docker/makinacorpus/ubuntu_template/lxc-stop.conf
• notice : https://github.com/makinacorpus/vms/blob/master/docker/makinacorpus/ubuntu_template/lxc-cleanup.sh

Use the script like this:

git clone https://github.com/makinacorpus/vms.git
cd docker
./make.sh make_image ubuntu

then run it:

 docker run -d makinacorpus/ubuntu /sbin/init
 docker ps
 ./make.sh dattach $dockerid

dups: #1024
links:#3182

[kiorky]

/cc @jpetazzo @regilero

[jpetazzo]

Would love to know what you tweaked; I just spent some time trying to get upstart to work, and here's what I found:

• to get any kind of logging, rm /dev/console ; ln -s tty /dev/console then execute init --verbose and --debug
• I tried to emit the filesystem event but it wasn't enough
• it looks like mountall tries to mount everything (as the time implies) but it fails, because the container doesn't have the required privileges (cap_sysadmin); and everything is already mounted anyway
• don't try to run with -privileged or you might blow up your system :-)

[kiorky]

Look the makinacorpus/ubuntu dockerfile or even play, i do not run it privileged. Did you noticed in the other bugs that we have a nearly working vm after the tweaks (left the reboot/stop & ro files & docker top problems)

[jpetazzo]

/cc @joshk

[kiorky]

note to myself: replace getty by the ln tty in console.conf

[ghost]

nice

[chymian]

nice work @kiorky
unfortunately I have two issues during the build process:
./make.sh make_image ubuntu
it stops with:

[docker make] Building docker image /root/dockers/makina/vm-ubuntu/docker/makinacorpus/ubuntu_saucy from imported tarball
Uploading context 1.536 kB
Uploading context 
2014/03/17 22:43:13 Error: Can't build a directory with no Dockerfile
building current ubuntu failed

manualy building in

.../makinacorpus/ubuntu
docker build -rm -t makinacorpus/ubuntu .

works, but attaching fails:

docker run -p 222:22 -d makinacorpus/ubuntu /sbin/init
./make.sh dattach 34c3fb3f89ad
 [docker make] Executing lxc-attach -n 34c3fb3f89ad5e9f5a7b572320b16f48cdba6568851a5ea91ffb566e0db4a43b -- bash
lxc-attach: failed to get the init pid

as well, connecting to ssh does not work. assuming that docker top is working correctly, there is nothing running in the container exept /sbin/init

dio top 34c3fb3f89ad
UID                 PID                 PPID                C                   STIME               TTY                 TIME                CMD
root                6045                694                 0                   23:06               ?                   00:00:00            /sbin/init

has it to do whith /sbin/init not running as PID 1?

host: debian jessie,
docker 0.9.0+dfsg1-1
lxc 0.9.0~alpha3-2+deb8u1

[crosbymichael]

ping @tianon I believe that we can close this issue now because we have an official ubuntu-upstart image that works. Can you confirm?

[tianon]

I think so, but I believe @kiorky disagrees?

[kiorky]

Exactly,@tianon i disagree:), having upstart begin to work does not make a real system work smoothly, all the subparts and linked bugs must also be addressed for this one to be fully solved.

[kiorky]

@tianon, as i gave up the docker frond a bit, can you give me a pointer to the official ubuntu-upstart image build scripts for me to give a shot please. I hope i wont see any dpkg-divert things for initctl in there ;)
May it be for @makinacorpus the time to give docker another try.

The problem with a full vmcontainer is that the initsystem is 50% of it but you need all the other bugs to be addressed, like the uplevel network/firewalls ones and the DNS ones (for resolv.conv & other readonly files)

[iwinux]

@kiorky this?

[MichaelSp]

Just for reference: https://github.com/phusion/baseimage-docker

[kiorky]

I really dislike phusion approach, for reference.

[kiorky]

We, for the moment are still waiting for @dotcloud to work on the linked bugs ;)
We are using just pain lxc ATM.

[mike503]

+1. I was hoping to also use docker as well instead of LXC.

Mainly so that the provisioning scripts and everything can be reused for Packer.

Packer can then produce Docker and Vagrant images for our development and staging, and Docker and/or EC2 for production images... for some reason Packer does not support LXC directly and still refuses to support it officially.

hashicorp/packer#790

Docker still seems more suitable for shippable images, not long running "instances" like a VPS would provide (KVM, Xen, LXC, OpenVZ, etc)

[jessfraz]

what is the actionable item here?

[jessfraz]

ping @kiorky ^

[kiorky]

See all open bugs ?

[kiorky]

(Plus the ones closes without fixing, like the ntp one)

[jessfraz]

can we close this now

[kiorky]

well this isnt fixed ;)

[pradeepchhetri]

I tried using ubuntu-upstart images but i am unable to start a service. Even I wasn't able to start sshd daemon. I followed the workaround which i found in another github issue as well but it didn't work too:

dpkg-divert --local --rename --add /sbin/initctl
ln -s /bin/true /sbin/initctl

Can you help me out in debugging this. I am starting the container like this:

docker run --rm -it ubuntu-upstart:14.04 /bin/bash

[thaJeztah]

@pradeepchhetri by running /bin/bash you're overriding the default command of the image, and upstart will not be started, and not be the primary process of the container, see https://github.com/tianon/dockerfiles/blob/4d24a12b54b75b3e0904d8a285900d88d3326361/sbin-init/ubuntu/upstart/14.04/Dockerfile#L53

If you start it without a custom command, ssh will be running and you can login using the default password docker.io;

docker run -d -it -p 2200:22 ubuntu-upstart:14.04
efd1555328e2034cb8a237a46798ec4a5a4fccd2da3d8518add1bf7aaeda2029

ssh root@192.168.99.100 -p 2200
The authenticity of host '[192.168.99.100]:2200 ([192.168.99.100]:2200)' can't be established.
RSA key fingerprint is 5b:e5:be:07:05:47:06:2a:f8:8d:42:4a:50:3c:61:ca.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '[192.168.99.100]:2200' (RSA) to the list of known hosts.
root@192.168.99.100's password:
Welcome to Ubuntu 14.04 LTS (GNU/Linux 3.19.0-31-generic x86_64)
....

[pradeepchhetri]

@thaJeztah Ahh, thank you for the reply. I want to run a curl command after container starts which will install a package that depends on upstart. Should i first ssh and then execute the command on it ?

[thaJeztah]

@pradeepchhetri why not build an image using a Dockerfile that uses ubuntu-upstart as parent? e.g.

FROM ubuntu-upstart:14.04
RUN apt-get update && apt-get install something

However, the GitHub issue tracker is not the best place to discuss this; can you ask this question in the #docker IRC channel? There's probably someone there to help you further.

[LK4D4]

@thaJeztah do you know what's going on here? I'm certainly not :/

[vdemeester]

Given the activity level on this issue, I'm going to close it as it's either fixed, a duplicate or not a request anymore. If you think I'm mistaken, feel free to discuss it there 😉