Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(server): implement tags/list endpoint #86

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

feat(server): implement tags/list endpoint #86

wants to merge 1 commit into from

Conversation

eonpatapon
Copy link

When a git source is used branch names are exposed in the tags list.

For other source types, only "latest" is exposed.

Fixes: #85

@eonpatapon
Copy link
Author

Using a local clone of nixpkgs:

 skopeo --insecure-policy --debug inspect --tls-verify=false docker://localhost:3000/shell
DEBU[0000] reference rewritten from 'localhost:3000/shell:latest' to 'localhost:3000/shell:latest'
DEBU[0000] Trying to access "localhost:3000/shell:latest"
DEBU[0000] Credentials not found
DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration
DEBU[0000]  No signature storage configuration found for localhost:3000/shell:latest
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/localhost:3000
DEBU[0000] GET https://localhost:3000/v2/
DEBU[0000] Ping https://localhost:3000/v2/ err Get https://localhost:3000/v2/: http: server gave HTTP response to HTTPS client (&url.Error{Op:"Get", URL:"https://localhost:3000/v2/", Err:(*errors.errorString)(0xc000331630)})
DEBU[0000] GET http://localhost:3000/v2/
DEBU[0000] Ping http://localhost:3000/v2/ status 200
DEBU[0000] GET http://localhost:3000/v2/shell/manifests/latest
DEBU[0000] Downloading /v2/shell/blobs/sha256:4cd79f85ab6250c3a111db2fa14c30d3444e00e4941aac99822f2e813113233e
DEBU[0000] GET http://localhost:3000/v2/shell/blobs/sha256:4cd79f85ab6250c3a111db2fa14c30d3444e00e4941aac99822f2e813113233e
DEBU[0000] Credentials not found
DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration
DEBU[0000]  No signature storage configuration found for localhost:3000/shell:latest
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/localhost:3000
DEBU[0000] GET https://localhost:3000/v2/
DEBU[0000] Ping https://localhost:3000/v2/ err Get https://localhost:3000/v2/: http: server gave HTTP response to HTTPS client (&url.Error{Op:"Get", URL:"https://localhost:3000/v2/", Err:(*errors.errorString)(0xc000331af0)})
DEBU[0000] GET http://localhost:3000/v2/
DEBU[0000] Ping http://localhost:3000/v2/ status 200
DEBU[0000] GET http://localhost:3000/v2/shell/tags/list
{
    "Name": "localhost:3000/shell",
    "Digest": "sha256:80dd7c0162c5a7d3b21f39d30b89b7aa0e90d792d8a0c4d18449e9567a1277b9",
    "RepoTags": [
        "latest",
        "fix-waybar",
        "kapow",
        "khronos",
        "nextcloud-paths",
        "nixos-unstable",
        "pipewire",
        "postfix-exporter",
        "release-19.09",
        "safe",
        "terraform-provider-keycloak",
        "terraform-providers",
        "terragrunt",
        "update-gitAndTools.delta",
        "vim-cue",
        "vim-plugins",
        "waybar-0.9.0",
        "wireshark"
    ],
    "Created": "0001-01-01T00:00:00Z",
    "DockerVersion": "",
    "Labels": null,
    "Architecture": "amd64",
    "Os": "linux",
    "Layers": [
        "sha256:4154da7086872108190d44dee0f850201a7c0b934a574af5c18014a018204d92",
        "sha256:60f1476de294ba0a45d8015579da439b5106afb38eec5fc0ee90152457682ab0",
        "sha256:a3a9b38adbc40c72a55bff6b151c475e2af9a2d1fff2ea58963be895cf0896dc",
        "sha256:adfb766e6074c49187e5f855f01cab9f143ecc50d5a1a9a41e7f4b168a5c7a6a",
        "sha256:b56088c8a44a150629bbd7c98d91f235080b59ce9f66dd86727a318d97fe5074",
        "sha256:ae8407af708662d112d57e41ea9088e2e30b13ca86870f6adddd6ca9327ac771",
        "sha256:fa80bfd2dcf2085098b3a967df1b28212cddcfa056a426291d4134e162d65e7b",
        "sha256:19f82ae38f6d463d2bbfe1c4c17c3652964494f4569b5a5f6f920032431dfe3e",
        "sha256:cefc7fc57a1a09633bca23a917f76b6db90bfcaddead61014950a12dd332ef6a",
        "sha256:a38d5cea5f469a68cb3877e81e82a91f541795ef01c87a554992ca66a19dd50c"
    ],
    "Env": null
}

@tazjin
Copy link
Owner

tazjin commented Feb 25, 2020

Hey, thanks for this PR! Seems like a useful feature :-)

Before doing an in-depth review, I have a couple of remarks:

  1. Should both branches and tags be listed?
  2. The plainOpen function from go-git seems to open repositories from local file paths only, however the Nixery git package set can come from remote repositories (e.g. via SSH, HTTP or the git protocol), too.
    Since the fetching currently happens in Nix (via builtins.fetchGit) it might be necessary to turn the fetching into a two-phase operation (caching the fetches for repo settings where cacheKey != "").
  3. Is the new dependency on go-git (and its transitive deps) needed? A git repository's local layout on the filesystem lists branches via the .git/refs/heads folder, and tags via .git/refs/tags. If the repository is present on the filesystem, looking into those to construct the list might be a better way.

@eonpatapon
Copy link
Author

eonpatapon commented Feb 26, 2020 •
edited

Hi, thanks for the feedback

  1. Actually I included only branches names since I wasn't able to build an image using a git tag. I didn't really look into it. I've just supposed it is not supported atm:

When trying to build with some git tag (19.09):

{"cmd":"nixery-prepare-image","eventTime":"2020-02-26T10:02:42.248190084+01:00","image":"shell","message":"[nix] error: cannot update ref 'refs/heads/19.09': trying to write non-commit object 0d99a63fd353198d13b1f6c17eb747a9b28f47c5 to branch 'refs/heads/19.09'","serviceContext":{"service":"nixery","version":"devel"},"severity":"INFO"}

  1. Are you suggesting to fetch the repos (HTTP, git://) before running the nix build ?

  2. I agree, we don't need to pull this extra dependency. I was just a bit too lazy :)

@eonpatapon
feat(server): implement tags/list endpoint
cd155b0 
When a git source is used branch names are exposed in the tags list.

For other source types, only "latest" is exposed.

Fixes: tazjin#85
@eonpatapon
Copy link
Author

Ok I get why building git tags doesn't work.

In the Render function of GitSource:

	if commitRegex.MatchString(tag) {
		args["rev"] = tag
	} else {
		args["ref"] = tag
	}

And I found in the manual for builtins.fetchGit: By default, the ref value is prefixed with refs/heads/. As of Nix 2.3.0 Nix will not prefix refs/heads/ if ref starts with refs/.

So that means that nixery would need to know if the requested docker tag is a git tag or git branch of the current repo before calling fetchGit. Which is kind of tricky since you can have a tag and a branch that have the same name.

@tazjin tazjin mentioned this pull request Feb 28, 2020
Open
@tazjin
Copy link
Owner

tazjin commented Feb 28, 2020

Hm, interesting, thanks! Lets spin the tag issue out into a separate problem for now (#89).

Are you suggesting to fetch the repos (HTTP, git://) before running the nix build?

Yeah, currently this happens as part of the build. In order for the functionality implemented here to work for remote repos the fetching needs to be separate. I believe the call to PlainOpen would currently just fail if Nixery is configured with a remote URL.

As for the dependency on go-git, I would prefer if we could avoid it for now (both branches and tags can be listed just via the filesystem regardless).

@eonpatapon
Copy link
Author

I've removed the dependency to go-git. I will have a look how we can fetch remote git repos before running the build when I have some time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Skopeo inspect does not work
2 participants
@eonpatapon @tazjin