x/vgo: avoid github rate limits?

[flibustenet]

What version of Go are you using (go version)?

go version go1.10 linux/amd64 vgo:2018-02-20.1

What did you do?

vgo build with more than 6 dependencies

What did you see instead?

vgo: 403 response from api.github.com

GitHub applies fairly small rate limits to unauthenticated users, and
you appear to be hitting them....

solution

create a token and put it in .netrc

it can be annoying for new user.

[rsc]

Now that the tool is public, I will try to find out from GitHub if there's anything we can do about this. I don't really understand how things like OAuth2 apply to command-line binaries that ship to users and therefore can't hold their own secrets. I agree it's annoying.

[andybons]

Setting an env var with a personal access token will likely be their response: https://help.github.com/articles/creating-a-personal-access-token-for-the-command-line/

[flibustenet]

I believe it should not be mandatory to have a github account at all to use vgo.

[tianon]

For just getting a tag list, implementing git ls-remote --tags ... should be sufficient (and not subject to the GitHub API rate limits). That won't work for arbitrary commits, but those don't necessarily need to be metadata'd every time -- only on fetch when a version isn't already supplied AFAICT (since the .zip archive download works with a raw commit hash directly).

[ericlagergren]

How does homebrew do it? Their entire model is pulling from GitHub, right?

[josharian]

brew gives you instructions for getting and using a github app token when you hit rate limits.

[freman]

I've been using githubs graphql api to avoid ratelimits while checking repo freshness for my caching git proxy...

It's not universal across all hosts, but it works on github :D

[franciscocpg]

I believe it should not be mandatory to have a github account at all to use vgo.

Totally agree. Unless you have private repositories in github, there should be no reason to have a github account to use vgo.

I've been using githubs graphql api to avoid ratelimits

The limits are different. But they are still there.

[freman]

The limits are different. But they are still there.

True enough, but using graphql I can query every single repo we cache in one request for one API call.

Just putting it out there :)

[drewblas]

Many other tools seem to be able to use github-hosted files and repos for dependency fetching without hitting the rate limits. I've personally never experienced a GH rate limit error, so it's surely possible. But now I've hit one with my very first vgo command. So either vgo is doing something very inefficient or its design is such that it requires far too many calls. Hopefully it's fixable!