Launching Latest Containers

[kramerrs]

When I set the aplication.yml to have container images with the :latest tag. ShinyProxy doesn't load the container. I have verified using the Docker Api engine that the images are there, but they don't have a tag eg they are tagged with . If I tag the image say with main, I can load it. However, with Docker/Portainer these images don't get updated automated with the stacks.

I am not sure where the ball is dropped here. Are the images tagged correctly in github, are the images pulled correctly by portainer. I have verified that I can tag the image both with latest and main using the same sha hash, and this gets updated every time I push. The image is pulled in portainer, but not tagged. Is this the correct behavior? However, I can try to create the image with the docker socket using curl, and latest fails, which I assume is the same as how ShinyProxy launches containers. However, Portainer somehow launches the latest image, if I set replicas to 1 in stacks compose.yml.

[LEDfan]

Hi, I'm not sure if I completely understand your question. Note that ShinyProxy does not handle the container images in any way, it just passes them to Docker (or kubernetes). Also, up to ShinyProxy 2.6.1, ShinyProxy would not automatically pull new images. With ShinyProxy 3.0.0, it is now possible to tell ShinyProxy to always pull new images, see: https://shinyproxy.io/documentation/configuration/#docker-image-pull-policy

[kramerrs]

Hi, I've poked around a while and read the docker docs. Sorry, I forgot to update this issue. I think the issue is regarding the pulling of images to docker swarm nodes, for whatever reason docker run doesn't automatically pulling them to nodes which is what I gather Portainer was using to launch the stacks. We'll try ShinyProxy 3.0, I feel like this would be a much better solution, if ShinyProxy pulls the images docker swarm nodes, through the docker proxy, such that we don't need to update and maintain a stack file with each image.

[kramerrs]

The documentation are sparse. However I think you are referring to the flag. proxy.docker.image-pull-policy

However, this is problematic, because if you read closely in the documents, the docker pull policy doesn't really do anything. It will pull an image and use that image if there are no other images that already have that tag. The reason this isn't helpful, is that in my case when a tag for that app already exists, it pulls it, but doesn't update the tag to point to the newest version. This can be done manually, update the tag to point to the newest SHA. I have script that does this very thing using curl, it queries the repository, in this case ghcr.io, then gets the sha tag, takes the sha tag and replaces the image name in the application.yml. However, I have to restart shinyproxy each time, and manually pull the images. Why can't one of shinyproxy, portainer, or docker do this? I have opened in issue in the docker community, because I feel it is philosophical flaw, that something outside the container changes (image tags change sha digest) that then cause the code in the container to break(the sha tags inside the container have to be updated to remain relevant).

[LEDfan]

Hi

However, this is problematic, because if you read closely in the documents, the docker pull policy doesn't really do anything. It will pull an image and use that image if there are no other images that already have that tag.

This is not correct, if you set the pull-policy to Always, Docker will always try to pull a new image, even if an image with this tag already exists.
For example, let's say you pull an image using:

docker pull debian:latest

This will pull the debian image with the latest tag. This image will correspond to a certain digest (hash). If you execute the same command next week, it will check on Docker Hub if a new image is available and pull that new image. That new image will have a different digest (hash). With Docker there is almost never a need to use image digests, you always specify the tag. Also in ShinyProxy, you should specify an image with a tag and not using a digest (in general). When you set the pull policy to Always, you will always use the latest version of the image which is available on the registry.

[kramerrs]

Hi LEDfan, yes it will pull it. However, what happens in a pull when an image exists locally, is that it doesn't tag the new image.

There are several people here who have confirmed that the issue is "by design", counter intuitive I know, since the policy is supposed to give you the most up-to-date version.

https://github.com/moby/moby/issues/37533

I've wrestled with this for a while, verified with curl's and docker socket queries, the image does in fact get pulled but ShinyProxy does not launch it by image name ghcr.io/org/repo:tag (nor can I launch it with a docker socket). As soon as I retag the image it works. It also works, to query the docker cache, find the image by the RepoTag, then use the Digest. Which is what my script did that updated the application.yml to sha digests corresponding to the most recent version of the image in the cache.

In addition, on a swarm backend or being launched from a docker stack as a service, each node has to have the image pulled independently to each node, preferably by connecting to the private repository rather than a local cache.