Add support for token login Gitlab (Registry crawler)

[MightyPrefix]

We use a private Gitlab Docker registry which uses a JWT authentication service to issue auth tokens for the registry API. Instead of authenticating with a username and password for the registry crawler, is it possible to support the use of an authentication token?

For example we use https://gitlab.xxx/jwt/auth?service=container_registry&scope=repository:xxx/xxx:pull which gives us the token we need to authenticate with the registry.

Is it possible to either just use that token directly or to make an auth request to our JWT service?

[ErikThoreson]

I think we do something similar for gcr.io but I'm not super happy with they way it's done since it's a one off.

https://github.com/target/portauthority/blob/master/pkg/docker/auth.go#L41-L57

Are you still supplying credentials for the initial login to the registry?

[MightyPrefix]

I'm running it locally on minikube, this is the curl command:

curl -X POST \
  http://192.168.99.100/v1/crawlers/registry  \
  -H "accept: application/json" \
  -H "Content-Type: application/json" \
  -d '{
      "RegCrawler":
      {
        "Registry": "https://registry.io",
        "Repos": ["xxx/xxx"],
        "Tags": ["latest"],
        "MaxThreads": 100,
        "Username": "xxx",
        "Password": "xxx"
      }
    }'

So I just fill in my username in the username field and gitlab access token in the password field.

[ErikThoreson]

Is that how you typically log into your gitlab registry? I took a brief look at their registry docko and it looks like fairly common for a docker v2 registry.

Also with the minikube setup you probably need to add the exposed nodeport to your query. like so:

curl -X POST \
  http://192.168.99.100:31700/v1/crawlers/registry  \
  -H "accept: application/json" \
  -H "Content-Type: application/json" \
  -d '{
      "RegCrawler":
      {
        "Registry": "https://registry.io",
        "Repos": ["xxx/xxx"],
        "Tags": ["latest"],
        "MaxThreads": 100,
        "Username": "xxx",
        "Password": "xxx"
      }
    }'

[MightyPrefix]

I used the nodeport while trying it out, must have deleted it when editing the copy paste.

We first authenticate with our jwt service to issue auth tokens for the registry API, then we use that token for all the other requests.

I think it is indeed similar to the gcr.io in the auth.go code

For example:
TOKEN=$(curl --user user:token "https://gitlab.xxx.io/jwt/auth?service=container_registry&scope=repository:xxx/xxx:pull" | jq -r .token)

And then:
curl -i -H "Authorization: Bearer $TOKEN" "https://registry.xxx.io/v2/xxx/xxx/tags/list"

Which will give me back the list

[MightyPrefix]

Apparently, if using Gitlab, the user provided will need to be an admin user.
Only an admin can use the * wildcard for the scope (For example scope:registry:catalog:*)

So now I ran into an error:
"Crawler":{"ID":10,"Type":"registry","Status":"finished","Messages":{"error":"** Crawl of https://registry.xxx.io produced error: error listing repositories for https://registry.xxx.io: Get /v2/_catalog?last=xxx%2xxx\u0026n=100: unsupported protocol scheme \"\" **"},"Started":"2018-05-16T14:52:25.540035Z","Finished":"2018-05-16T14:52:32.71182Z"}}

The get request goes to a deleted repository which is still found in the _catalog list . (Also no https:// in the GET Request)