dlite not starting correctly after reboot #214

getninjaN · 2016-12-01T16:22:54Z

Bug Reports

dlite version in use (run dlite --version):
dlite version 2.0.0-beta8
expected behavior:
dlite should start correctly after reboot and make my day the best day ever.
actual behavior:
dlite doesn't start correctly after reboot and makes the reboot day the worst day ever.
steps to reproduce
I haven't got a clue

TL;DR

Something seems to be wrong with extractUser, lookupUser or proxy on my machine, I don't really know...

My story

After the first install dlite starts without any problems and runs great, but after a reboot it won't start correctly.
Directly after I log in I can find a dlite process but no hyperkit process in Activity Monitor.
The dlite process is using 1-2 MB of RAM, which sounds small but probably isn't anything weird.

docker ps returns an error

$ docker ps
Error response from daemon: Unable to connect to the virtual machine

dlite start runs into a timeout. (two dlite processes during this time and when it's done one process is terminated and the original process persists)

$ dlite start
Starting the virtual machine: ERROR!
Timed out waiting for virtual machine

dlite stop runs into infinity and beyond until I press ctrl-c. (the dlite process is still running)

Running dlite stop again after this:

$ dlite stop
Stopping the virtual machine: done

(the dlite process is still running)

Debug mode activated

So I start digging and I find out that a some commands makes a HTTP POST request to http://127.0.0.1:1050/[command].

Running curl -X POST http://127.0.0.1:1050/start returns Unauthorized
Running curl -X POST --header "X-Username: emil" http://127.0.0.1:1050/start returns Timed out waiting for virtual machine
Running curl -X POST http://127.0.0.1:1050/stopreturns Virtual machine is not running (which is expected)
Using Chrome and visiting http://127.0.0.1:1050/status also returns Unauthorized.

It seems like there's something wrong with extractUser, lookupUser or proxy.

I have tried to uninstall everything (I think) and reinstall dlite but with the same results.
I have tried to unload local.docker.plist and loading it again but with the same results.

The text was updated successfully, but these errors were encountered:

kakoni · 2016-12-09T21:58:17Z

I'm experiencing same symptoms, except http://127.0.0.1:1050/status returns valid looking status

{"id":"a54321e06-be54-11e6-9769-7056818e1367","hostname":"local.docker","disk_size":20,"disk_path":"/Users/kakoni/.dlite/disk.qcow","cpu_cores":2,"memory":2,"dns_server":"192.168.64.1","docker_version":"latest","docker_args":"","route":true,"started":true,"ip":"192.168.64.7","pid":8549}

bibendi · 2016-12-13T18:13:37Z

My story

reboot

$ docker ps
Cannot connect to the Docker daemon. Is the docker daemon running on this host?

$ ls -l /var/run/docker.sock
srwxrwxrwx 1 root daemon 0 13 дек 23:03 /var/run/docker.sock

$ sudo rm -rf /var/run/docker.sock
$ sudo launchctl stop local.dlite
$ sudo launchctl start local.dlite
$ dlite start
Starting the virtual machine: ERROR!
Timed out waiting for virtual machine
$ dlite status
vm_state: started
ip_address: 192.168.64.4
pid: 963
id: aceb2746-a7d0-11e6-affc-80e6502222b0
hostname: local.docker
disk_size: 25
disk_path: /Users/merkushin/.dlite/disk.qcow
cpu_cores: 2
memory: 3
dns_server: 192.168.64.1
docker_version: latest
docker_args: --bip=172.17.0.1/24 --dns=172.17.0.1

$ docker ps
Error response from daemon: Unable to connect to the virtual machine

Waiting 1 minute...

$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
7b5e00b68d69 aacebedo/dnsdock:latest-amd64 "dnsdock" 4 weeks ago Up 2 seconds 172.17.0.1:53->53/udp dnsdock

nlf · 2016-12-13T18:16:14Z

can those of you experiencing this issue run dlite ssh and tell me the version of dlite-os shown?

if you have a version earlier than 1.0.0-beta3 this problem should be fixed by re-running dlite init, if you have dlite-os 1.0.0-beta3 and you're still experiencing this issue i have more debugging to do

getninjaN · 2016-12-14T08:57:20Z

$ dlite ssh
dlite-os version 1.0.0-beta3
Docker version 1.12.3, build 6b644ec

bibendi · 2016-12-19T11:43:16Z

$ dlite ssh
merkushin@local.docker's password:

ctrl + c =)

$ ssh docker@$(dlite ip)
docker@192.168.64.4's password:
dlite-os version 1.0.0-beta3
Docker version 1.12.3, build 6b644ec

getninjaN · 2016-12-28T12:49:38Z

A thing just hit me, I downloaded the binary from "Releases" and did not build it myself.
Could this be a thing causing me trouble?

nlf · 2016-12-31T18:48:11Z

@getninjaN that shouldn't be causing a problem, that's the binary i run on my own laptop without issues.

interesting that the vm seems to be coming up and just isn't phoning home correctly.. can someone who is able to login to their vm run df and tell me available disk space on their vm?

maeldur · 2017-01-22T23:10:19Z

i'm getting this same issue (beta8) and

$ dlite ssh
ssh: connect to host local.docker port 22: Operation timed out

getninjaN · 2017-02-10T13:48:04Z

I managed to fix this.

TL;DR

PEBKAC 🙃

The long story

$ dlite stop (2.0.0-beta8 or beta9)
Run Docker Toolbox Uninstall Script
$ brew uninstall docker docker-machine
$ dlite uninstall (2.0.0-beta8 or beta9)
$ brew uninstall dlite (2.0.0-beta8 or beta9)
$ brew install dlite (1.1.5)
$ brew uninstall dlite (1.1.5)
Restart macOS
$ brew install docker-compose
Download dlite 2.0.0-beta9
$ cp dlite /usr/local/bin
$ dlite init
Got DISK ERROR! as in Creating disk: ERROR! #217
$ brew install libev (To fix DISK ERROR!)
$ dlite init
docker-compose up and when this was done...
... restart macOS
$ dlite start
...
PROFIT!

Conclusion

Now everything is working like clockwork again.
What the problem was from the beginning is probably a combination of having used Kitematic, Docker for Mac and dlite-1.1.5, without properly uninstalling them first and in between use.

bibendi · 2017-02-13T05:31:45Z

I'm tired of waiting for fix of the problem 😸
docker-machine-driver-xhyve is working like a charm

getninjaN · 2017-02-15T16:08:11Z

Well sh*t... Ran into another problem now.
My Mac was acting up and I had to kill it with the power switch.

Now when I try to run dlite start I get this error

Starting the virtual machine: ERROR!
chown /Users/emil/.dlite/vm.tty: no such file or directory

In my console I get this for InternetSharing (/usr/libexec/InternetSharing)

2017-02-15 17:03:20.636867
com.docker.hyperkit: com.apple.NetworkSharing.broadcast-1 has been started

2017-02-15 17:03:20.650614
com.docker.hyperkit: com.apple.NetworkSharing.broadcast-1 (idle) has been stopped

Peace and love

nlf · 2017-02-15T16:11:58Z

well that's a new one.. i've actually seen the no such file or directory for the vm.tty before, though, so i'll open an issue for that specifically.

the InternetSharing stuff though, that's a new one. is there anything interesting in /Users/emil/.dlite/vm.log? likely at the very bottom

getninjaN · 2017-02-15T16:17:44Z

Nope.. vm.log wasn't modified at all.
Checked it when I first got the problem and after a new reboot, to see if that made it work, it had nothing new in it.

I can try to see if I'm able to reproduce this and check again.
Tried a whole bunch of stupid things without any success so I just reinstalled.

KiSchulte · 2017-03-13T15:05:46Z

Hi,

still an issue.
Is there any progress on this issue?

Just downloaded the binary today.

dlite ssh dlite-os version 1.0.0-beta3 Docker version 1.12.3, build 6b644ec

synic · 2017-04-05T16:29:48Z

This is the biggest issue I have with dlite ATM. Any progress here? Is dlite still being developed?

nlf · 2017-04-05T16:51:00Z

@synic sorry, yes. i'm still working on this one. doing some refactoring to make things more testable and also make it easier to handle error cases, and log more debugging information.

not being able to reproduce this one makes fixing it like playing a game of whack-a-mole in the dark with a blindfold on, rather than doing that i'm going to shuffle things around to try to isolate pieces of logic as much as possible. with that and some additional logging it should become a lot more clear when things go wrong. plus it means i can start actually writing unit tests for things, which will be nice.

it is, however, slow going. i promise it'll all be worth it in the long run though!

synic · 2017-04-05T17:01:16Z

Anything I can do to help, let me know!

…

On Wed, Apr 5, 2017, 10:51 AM Nathan LaFreniere ***@***.***> wrote: @synic <https://github.com/synic> sorry, yes. i'm still working on this one. doing some refactoring to make things more testable and also make it easier to handle error cases, and log more debugging information. not being able to reproduce this one makes fixing it like playing a game of whack-a-mole in the dark with a blindfold on, rather than doing that i'm going to shuffle things around to try to isolate pieces of logic as much as possible. with that and some additional logging it should become a lot more clear when things go wrong. plus it means i can start actually writing unit tests for things, which will be nice. it is, however, slow going. i promise it'll all be worth it in the long run though! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#214 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAB4ut3f0CA9Ymx7sz1qlMS23vbtRinoks5rs8Z1gaJpZM4LBl2o> .

WoodrowShigeru · 2018-11-29T10:47:03Z

Not sure if just a coincidence but … after over an hour of dlite start and dlite stop, I just deactivated my WLAN and ran dlite start again – it started on the first try.

Maybe this helps.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

dlite not starting correctly after reboot #214

dlite not starting correctly after reboot #214

getninjaN commented Dec 1, 2016

kakoni commented Dec 9, 2016

bibendi commented Dec 13, 2016

nlf commented Dec 13, 2016

getninjaN commented Dec 14, 2016

bibendi commented Dec 19, 2016

getninjaN commented Dec 28, 2016

nlf commented Dec 31, 2016

maeldur commented Jan 22, 2017

getninjaN commented Feb 10, 2017

bibendi commented Feb 13, 2017

getninjaN commented Feb 15, 2017

nlf commented Feb 15, 2017

getninjaN commented Feb 15, 2017

KiSchulte commented Mar 13, 2017

synic commented Apr 5, 2017

nlf commented Apr 5, 2017

synic commented Apr 5, 2017 via email

WoodrowShigeru commented Nov 29, 2018

dlite not starting correctly after reboot #214

dlite not starting correctly after reboot #214

Comments

getninjaN commented Dec 1, 2016

Bug Reports

TL;DR

My story

Debug mode activated

kakoni commented Dec 9, 2016

bibendi commented Dec 13, 2016

nlf commented Dec 13, 2016

getninjaN commented Dec 14, 2016

bibendi commented Dec 19, 2016

getninjaN commented Dec 28, 2016

nlf commented Dec 31, 2016

maeldur commented Jan 22, 2017

getninjaN commented Feb 10, 2017

TL;DR

The long story

Conclusion

bibendi commented Feb 13, 2017

getninjaN commented Feb 15, 2017

nlf commented Feb 15, 2017

getninjaN commented Feb 15, 2017

KiSchulte commented Mar 13, 2017

synic commented Apr 5, 2017

nlf commented Apr 5, 2017

synic commented Apr 5, 2017 via email

WoodrowShigeru commented Nov 29, 2018