cmd/go: add support for Azure DevOps Git repos to go get

[MrWako]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go1.11 windows/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

GOHOSTARCH=amd64
GOHOSTOS=windows
GOOS=windows

What did you do?

I created a few public Azure DevOps repos to illustrate the problems

• https://dev.azure.com/markawakefield/GoGetTest/_git/app
• https://dev.azure.com/markawakefield/GoGetTest/_git/stringfmt.git
• https://dev.azure.com/markawakefield/GoGetTest/_git/stringutils

Then when trying to get the package with the main function I get:

go get -v dev.azure.com/markawakefield/GoGetTest/_git/app
Fetching https://dev.azure.com/markawakefield/GoGetTest/_git/app?go-get=1
Parsing meta tags from https://dev.azure.com/markawakefield/GoGetTest/_git/app?go-get=1 (status code 200)
package dev.azure.com/markawakefield/GoGetTest/_git/app: unrecognized import path "dev.azure.com/markawakefield/GoGetTest/_git/app" (parse https://dev.azure.com/markawakefield/GoGetTest/_git/app?go-get=1: no go-import meta tags ())

I think because go get does not know its a Git repo, and there are no meta tags supplied to indicate that it is. If I try to just get one of the other packages with a .git extension I get:

go get -v dev.azure.com/markawakefield/GoGetTest/_git/stringfmt.git
# cd .; git ls-remote https://dev.azure.com/markawakefield/GoGetTest/_git/stringfmt
Logon failed, use ctrl+c to cancel basic credential prompt.
fatal: could not read Username for 'https://dev.azure.com': terminal prompts disabled
# cd .; git ls-remote git+ssh://dev.azure.com/markawakefield/GoGetTest/_git/stringfmt
ssh: connect to host dev.azure.com port 22: Connection timed out
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
# cd .; git ls-remote ssh://dev.azure.com/markawakefield/GoGetTest/_git/stringfmt
ssh: connect to host dev.azure.com port 22: Connection timed out
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
package dev.azure.com/markawakefield/GoGetTest/_git/stringfmt.git: cannot download, git://dev.azure.com/markawakefield/GoGetTest/_git/stringfmt uses insecure protocol

I think because the .git extension on the URL is not being passed to the git commands, and therefore it hits an authenticated endpoint.

What did you expect to see?

I expected (hoped) to see the packages add to the GOPATH under the dev.azure.com directory.

What did you see instead?

No files added to the GOPATH

I appreciate that this is not a pure issue with the current Go tool chain, more a miss-match between the conventions of Go and Azure DevOps, and has been noted elsewhere including http://ppanyukov.github.io/2017/02/01/golang-with-vsts-repos.html. Please let me know if a proposal is more appropriate.

For other cloud providers (IBM/Apache/OpenStack) there is logic to accomodate these differences and for Azure DevOps its actually a little worse than just the go get issue as even if this command completed the package ends up under a _git directory and so will be ignored in some instances due to the underscore.

Until now we have used Glide for package management which has allowed us to work around the issue but we would now like to adopt versioned packages so the issue is becoming more pressing.

I'll open a PR with a small revision addressing the issue that can be discussed/reviewed.

Thanks

[gopherbot]

Change https://golang.org/cl/142678 mentions this issue: cmd/go/internal/get: fix go get for Azure DevOps Git repos

[rsc]

@FiloSottile, it would be nice to say what the decision is when you add the NeedsDecision label.

[rsc]

Note that GitHub generates the <meta> tags that the go command needs, so that GitHub-hosted code works with go get regardless of domain (e.g., GitHub Enterprise, although github.com has the tags too). At this point it would be nice if hosting sites like dev.azure.com would put the tags in and therefore own their meaning, instead of making the go project source code the source of truth. (That made sense early on when no one had heard of Go and not many sites cared.)

[MrWako]

thanks @rsc - meta tags on the Azure side is definitely another solution route, although due to the .../_git/... in the URLs, it would be more involved than MS just adding these to the existing address. Azure could, or I could, host vanity URLs with the meta tags pointing to the existing repo URLs but this is starting to get more involved and I'd really appreciate a simpler solution.

[MrWako]

Just adding the links to the issues on the Azure DevOps side:

• https://visualstudio.uservoice.com/forums/330519-visual-studio-team-services/suggestions/34888885-support-repos-for-golang-by-providing-a-url-alias
• https://visualstudio.uservoice.com/forums/330519-azure-devops-formerly-visual-studio-team-services/suggestions/20364613-add-golang-support-to-vsts-git

to indicate that the mismatch between Azure DevOps and Go is a real problem for developers that have no choice but to use Azure DevOps, but want to champion Go.

[FiloSottile]

@FiloSottile, it would be nice to say what the decision is when you add the NeedsDecision label.

Will do. In this instance I'd like an owner of cmd/go to decide if we are still expanding the special case mappings, and if so what transformations should be applied to this one, as it seems to be the only one that magically adds a component to the path.

For the record, I do believe adding <meta> tags at the desired path would serve Azure DevOps best, providing them with more control, flexibility (no issues with adding path components at the right spot) and probably compatibility as there might be tools that extracted the mappings from an existing version of Go (although they should use the go tool instead).

[MrWako]

@bketelsen perhaps you could add your experience, thanks.

[crsdrw]

@rsc Whilst I think it would be a good idea for dev.azure.com to add tags, even if they did, I don't think that would solve the problem. As @MrWako says, because dev.azure.com repository paths have a .../_git/... in them go get results in folders that start with an underscore. One solution to this problem would be to "fix" the go tooling like go list so that it can handle folders that start with an underscore but it seems like an easier solution is to change go get. With the added benefit of getting cleaner import paths for dev.azure.com repositories.

It might be a hard sell to convince MS to host vanity urls for dev.azure.com repositories because of the way go tooling handles folders that start with an underscore, although we are trying.

[FiloSottile]

To be clear I'm not suggesting having import paths with _git in them, but serving meta tags from https://dev.azure.com/markawakefield/GoGetTest/app[/*] pointing to https://dev.azure.com/markawakefield/GoGetTest/_git/app.