Error response from daemon: can not get logs from container which is dead or marked for removal

[Deepak-Enc]

Docker image is running fine on my system but when run it on edge orchestration it shows me the errors:

Running perfect on my system:

Here is the command which I am running:
curl -X POST "192.168.43.11:56001/api/v1/orchestration/services" -H "accept: application/json" -H "Content-Type: application/json" -d "{ "ServiceName": "age_detection", "ServiceInfo": [{ "ExecutionType": "container", "ExecCmd": [ "docker", "run", "-it","--rm","--device=/dev/video0","-e","DISPLAY=$DISPLAY ","-v","$XSOCK:$XSOCK","-v","$XAUTH:$XAUTH","-e","XAUTHORITY=$XAUTH","-e"," QT_X11_NO_MITSHM=1","animeshj123/age_det_image:02"]}], "StatusCallbackURI": "http://192.168.43.11:8888/api/v1/services/notification\"}"

[tdrozdovsky]

@Deepak-Enc I successfully reproduced your case with an error and also successfully launched the container using edge-orchestration.
Below is the command that I did to launch successfully

virtual-pc@virtualpc-VirtualBox:~$ curl -X POST "127.0.0.1:56001/api/v1/orchestration/services" -H "accept: application/json" -H "Content-Type: application/json" -d "{ \"ServiceName\": \"age_detection\", \"ServiceInfo\": [{ \"ExecutionType\": \"container\", \"ExecCmd\": [ \"docker\", \"run\", \"-it\",\"--rm\",\"--device=/dev/video0\", \"--net=host\",\"--ipc=host\", \"-e\",\"DISPLAY=$DISPLAY\",\"-v\",\"/tmp/.X11-unix:/tmp/.X11-unix\",\"-e\",\"QT_X11_NO_MITSHM=1\",\"animeshj123/age_det_image:02\"]}], \"StatusCallbackURI\": \"http://127.0.0.1:8888/api/v1/services/notification\"}"
{"Message":"ERROR_NONE","RemoteTargetInfo":{"ExecutionType":"container","Target":"10.0.2.15"},"ServiceName":"age_detection"}

1. My X server settings are slightly different, but you can use your own if it works for you. The main thing to pay attention to quotation marks. In your example, they are not correct (I think this is a typo).
2. The error that you encountered was a consequence of running the container twice. The program that works inside gives an error, because it can’t access the webcam.
   
   Before starting, make sure that such a container is not started before using the command

virtual-pc@virtualpc-VirtualBox:~$ docker ps -a
CONTAINER ID        IMAGE                          COMMAND                  CREATED             STATUS              PORTS               NAMES
0d73793665d9        animeshj123/age_det_image:02   "python detect_age_c…"   16 minutes ago      Up 16 minutes                           crazy_visvesvaraya
f1a1dacb70fd        edge-orchestration:baobab      "sh run.sh"              16 minutes ago      Up 16 minutes                           edge-orchestration
virtual-pc@virtualpc-VirtualBox:~$ 

Stop the running container, and after that you can start using edge-orchestration

Please give feedback what is your result? I hope everything worked out for you. If you have questions, we will be happy to answer!

[Deepak-Enc]

Receiving the same error.

[tdrozdovsky]

You need to make sure that the program in the animeshj123/age_det_image:02 container starts correctly, as the container execution status : 1 line shows that an error has occurred in the program itself.

I suggest doing it. After that, I see that everything works correctly:

virtual-pc@virtualpc-VirtualBox:~$ sudo docker run --rm -ti --device=/dev/video0 --net=host --ipc=host -e DISPLAY=$DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix --env="QT_X11_NO_MITSHM=1" animeshj123/age_det_image:02
[sudo] password for virtual-pc: 
[INFO] loading face detector model...
[INFO] loading age detector model...
[INFO] starting video stream...
^CTraceback (most recent call last):
  File "detect_age_cam.py", line 121, in <module>
    results = detect_and_predict_age(frame, faceNet, ageNet,
  File "detect_age_cam.py", line 29, in detect_and_predict_age
    detections = faceNet.forward()
KeyboardInterrupt
virtual-pc@virtualpc-VirtualBox:~$ docker ps -a
CONTAINER ID        IMAGE                       COMMAND             CREATED             STATUS              PORTS               NAMES
f1a1dacb70fd        edge-orchestration:baobab   "sh run.sh"         17 hours ago        Up 17 hours                             edge-orchestration
virtual-pc@virtualpc-VirtualBox:~$ curl -X POST "127.0.0.1:56001/api/v1/orchestration/services" -H "accept: application/json" -H "Content-Type: application/json" -d "{ \"ServiceName\": \"age_detection\", \"ServiceInfo\": [{ \"ExecutionType\": \"container\", \"ExecCmd\": [ \"docker\", \"run\", \"-it\",\"--rm\",\"--device=/dev/video0\", \"--net=host\",\"--ipc=host\", \"-e\",\"DISPLAY=$DISPLAY\",\"-v\",\"/tmp/.X11-unix:/tmp/.X11-unix\",\"-e\",\"QT_X11_NO_MITSHM=1\",\"animeshj123/age_det_image:02\"]}], \"StatusCallbackURI\": \"http://127.0.0.1:8888/api/v1/services/notification\"}"
{"Message":"ERROR_NONE","RemoteTargetInfo":{"ExecutionType":"container","Target":"10.0.2.15"},"ServiceName":"age_detection"}virtual-pc@virtualpc-VirtualBox:~$ 
virtual-pc@virtualpc-VirtualBox:~$ 

I also see such an error.

it can be fixed if restart docker (service docker restart)

[Deepak-Enc]

Same container running fine on your edge and also on my localhost that means programme have no issue. there may be an issue on my edge-orchestration how to solve this issue.

…

On Mon, 15 Jun 2020, 16:25 tdrozdovsky, ***@***.***> wrote: You need to make sure that the program in the animeshj123/age_det_image:02 container starts correctly, as the container execution status : 1 line shows that an error has occurred in the program itself. [image: изображение] <https://user-images.githubusercontent.com/45031429/84649763-bd171c00-af0f-11ea-9ff6-7ba80aea6662.png> I suggest doing it. After that, I see that everything works correctly: ***@***.***:~$ sudo docker run --rm -ti --device=/dev/video0 --net=host --ipc=host -e DISPLAY=$DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix --env="QT_X11_NO_MITSHM=1" animeshj123/age_det_image:02 [sudo] password for virtual-pc: [INFO] loading face detector model... [INFO] loading age detector model... [INFO] starting video stream... ^CTraceback (most recent call last): File "detect_age_cam.py", line 121, in <module> results = detect_and_predict_age(frame, faceNet, ageNet, File "detect_age_cam.py", line 29, in detect_and_predict_age detections = faceNet.forward() KeyboardInterrupt ***@***.***:~$ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES f1a1dacb70fd edge-orchestration:baobab "sh run.sh" 17 hours ago Up 17 hours edge-orchestration ***@***.***:~$ curl -X POST "127.0.0.1:56001/api/v1/orchestration/services" -H "accept: application/json" -H "Content-Type: application/json" -d "{ \"ServiceName\": \"age_detection\", \"ServiceInfo\": [{ \"ExecutionType\": \"container\", \"ExecCmd\": [ \"docker\", \"run\", \"-it\",\"--rm\",\"--device=/dev/video0\", \"--net=host\",\"--ipc=host\", \"-e\",\"DISPLAY=$DISPLAY\",\"-v\",\"/tmp/.X11-unix:/tmp/.X11-unix\",\"-e\",\"QT_X11_NO_MITSHM=1\",\"animeshj123/age_det_image:02\"]}], \"StatusCallbackURI\": \"http://127.0.0.1:8888/api/v1/services/notification\"}" ***@***.***:~$ ***@***.***:~$ I also see such an error. [image: изображение] <https://user-images.githubusercontent.com/45031429/84649385-1d598e00-af0f-11ea-9b6a-43a7780c7cf5.png> it can be fixed if restart docker (service docker restart) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#91 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANR2JO4EKLU3UYAYP4OSZB3RWX4ZPANCNFSM4N42UCTA> .

[tdrozdovsky]

Below is an example of how it may not work correctly

virtual-pc@virtualpc-VirtualBox:~$ docker run --rm -ti --device=/dev/video0 --net=host --ipc=host -e DISPLAY=$DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix --env="QT_X11_NO_MITSHM=1" animeshj123/age_det_image:02
[sudo] password for virtual-pc: 
[INFO] loading face detector model...
[INFO] loading age detector model...
[INFO] starting video stream...
[ WARN:0] global /io/opencv/modules/videoio/src/cap_v4l.cpp (887) open VIDEOIO(V4L2:/dev/video0): can't open camera by index
Traceback (most recent call last):
  File "detect_age_cam.py", line 117, in <module>
    frame = imutils.resize(frame, width=400)
  File "/usr/local/lib/python3.8/site-packages/imutils/convenience.py", line 69, in resize
    (h, w) = image.shape[:2]
AttributeError: 'NoneType' object has no attribute 'shape'

now look at the error code

virtual-pc@virtualpc-VirtualBox:~$ echo $?
1

the error is the same as the edge-orchestrator returns

I think it is necessary to look for reasons why the program does not work correctly (for example, there is no access to /dev/video0 or others)

[Deepak-Enc]

Kindly clear my doubt. It's working fine on my host machine that means there is no problem in docker container or docker file. There may be an issue with edge-orchestration because I am unable to detect in container or docker file.

[MoonkiHong]

@Deepak-Enc Please describe much more in details about what "an issue with edge-orchestration" is. From my understanding, your targeting container successfully runs on your local host through a docker container / docker file, but it fails when you try to run it on edge-orchestration. We have guided you several recommendation but still it is unclear to get through this issue. Are we correct?
@suresh-lc What is your opinion? It does not seem to be just an configuration issue.

[Deepak-Enc]

2020/06/16 04:27:56 containerexecutor.go:84: [containerexecutor] create container : c22c72a303
2020/06/16 04:28:01 containerexecutor.go:102: [containerexecutor] container execution status : 1
2020/06/16 04:28:01 containerexecutor.go:111: [containerexecutor] Error response from daemon: can not get logs from container which is dead or marked for removal
2020/06/16 04:28:01 orchestration_api.go:340: [orchestrationapi] service status changed [appNames:age_detection][status:]
2020/06/16 04:28:01 containerexecutor.go:122: [containerexecutor] Error response from daemon: removal of container c22c72a303bff50d18b4af5d5b7d644256cdd8fb2c7d2b08ccfa2ea225b3aa7f is already in progress

I am receiving these errors while running age-detection containers(real-time and Video) on edge-orchestration. It captures the first frame of application but then automatically container marked as a dead container.

[suresh-lc]

@Deepak-Enc : We could see from the logs the container is getting marked for removal after reading the first frame due to some error. But exactly why we are also trying to find. Can you share the complete error log file. And also if you could share the video of the steps you did, it would be great help.

[tdrozdovsky]

Below is a video of how a container works under edge-orchestration

Ubuntu_v16_04 [Running] - Oracle VM VirtualBox 6_16_2020 9_30_15 AM.zip

The log that you sent shows that the application in the container did not finish correctly. More data needed

[suresh-lc]

@Deepak-Enc : Can you take Baobab branch and try the scenario.
Looks container whitelisting is being not done for service started from edge orchestration.
err := verifier.GetInstance().ContainerIsInWhiteList(s.ParamStr[paramLen-1]).

Hence can you once try with Baobab branch and let us know the result.

[Deepak-Enc]

It's working on baobab. Thank You.

[MoonkiHong]

Issue Resolved. @suresh-lc @tdrozdovsky @Deepak-Enc Thank you for all your interests and passion!