SHUFFLE_BASE_IMAGE_REPOSITORY : Not used ?

[seb2020]

Describe the bug
For downloading container, we need to use a proxy like JFrog or Nexus. For doing that, we need to use an internal URL instead of docker hub. I have seen in the .env file this variable SHUFFLE_BASE_IMAGE_REPOSITORY but it doesn't seems to be use. Or maybe this variable is for something else ?

To Reproduce
Steps to reproduce the behavior:

1. Set SHUFFLE_BASE_IMAGE_REPOSITORY to docker-remote.jfrog.domain.com
2. Run the docker compose
3. Docker cannot download container because it use https://registry-1.docker.io/v2/ instead of the internal url

Expected behavior
Container should be downloaded trough the SHUFFLE_BASE_IMAGE_REPOSITORY URL

Debug logs

shuffle-backend     | ERROR ABOVE: Trying to pull tags from: frikky/shuffle:email_1.1.0
shuffle-backend     | 2024/04/17 09:25:04 [INFO] Pulling image registry.hub.docker.com/frikky/shuffle:email_1.1.0
shuffle-backend     | 2024/04/17 09:25:19 [ERROR] Failed getting image registry.hub.docker.com/frikky/shuffle:email_1.1.0: Error response from daemon: Get "https://registry.hub.docker.com/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
shuffle-backend     | 2024/04/17 09:25:19 [DEBUG] Failed image build memory. Creating notification with org "": Failed to build / download images frikky/shuffle:email_1.1.0

I think the issue is coming from https://github.com/Shuffle/Shuffle/blob/main/functions/onprem/worker/worker.go#L57 because the registryName is hard coded instead of using the variable SHUFFLE_BASE_IMAGE_REPOSITORY

[frikky]

Thanks @seb2020! It seems like you may be right. It is however for SHUFFLE_BASE_IMAGE_REGISTRY, not not repository, but the problem seems the same. It also looks like it may be fixed for Kubernetes deployments, and not Docker somehow.

@0x0elliot - can you have a look at this when getting into Orborus/Worker optimisations?

[seb2020]

Hi,

So, what's the correct variable to use ? Because variables below are not the same endpoint ?

SHUFFLE_BASE_IMAGE_REGISTRY --> github registry
SHUFFLE_BASE_IMAGE_REPOSITORY --> docker hub

[frikky]

Hi,

So, what's the correct variable to use ? Because variables below are not the same endpoint ?

SHUFFLE_BASE_IMAGE_REGISTRY --> github registry SHUFFLE_BASE_IMAGE_REPOSITORY --> docker hub

Ah, I see the confusion - we shouldn't be using the "REPOSITORY" one. Both dockerhub & github are registries, and should be treated as such. We'll make a fix in the comming week sometime for this, to allow for use of other registries directly.

PS: Could we have a call about this? I'm wondering how you think about proxies and the like, as HTTP_PROXY can be utilised in a lot of cases, and I'd like to document how this problem can be solved.

[seb2020]

I have send to you a DM in the discord server

[seb2020]

Any news on this issue ?

[seb2020]

@frikky and @0x0elliot : Do you have any feedback on my issue ? Thanks