Migration of big GitHub repo fails

[davidak]

• Gitea version (or commit ref): 1.13.0+rc1
• Git version: 2.20.1
• Operating system: debian stable 10.6
• Database (use [x]):
  • PostgreSQL
  • MySQL
  • MSSQL
  • SQLite
• Can you reproduce the bug at https://try.gitea.io:
  • Yes https://try.gitea.io/nixos/nixpkgs https://try.gitea.io/nixos/nixpkgs/settings/branches
  • No
• Log gist: https://gist.github.com/Codeberg-org/a4e32102e761c857ecf46e3fb6f22237

Description

When i try to migrate nixpkgs to the Codeberg test server, it always fails. We have increased the migration timeouts x100.

When we wait or open https://codeberg-test.org/NixOS/nixpkgs/settings/branches, we get a trace with error in setting_protected_branch.go:34.

It is not a hardware limitation.

Screenshots

[jolheiser]

For anyone looking in to it, this is the line referenced.

gitea/routers/repo/setting_protected_branch.go

Line 34 in ba97c0e

branches := ctx.Data["Branches"].([]string)

[davidak]

@jolheiser are you a bot? 😄 (because of the instant reaction)

Thanks for the information!

It might be possible that not all branches are migrated or the data is broken, so ctx.Data["Branches"] don't exist?

[Codeberg-org]

To add some more color, this log line (also referenced in the gist) indicates that gh throttled API calls, fetched data may or may not be truncated or incomplete:

modules/task/task.go:51:handle() [E] Run task failed: GET https://api.github.com/repos/NixOS/nixpkgs: 403 API rate limit of 5000 still exceeded until 2020-10-21 16:33:28 +0200 CEST, not making remote request. [rate reset in 34m49s]

(fwiw there was no log line reporting API request errors or timeouts).

[jolheiser]

@jolheiser are you a bot?

I really hope not! 🤖

Also, for what it's worth I believe this would benefit from #12244 as well.
Dumping to disk may not help if we are hitting a rate limit anyways.

[davidak]

And nixpkgs is way bigger. It has 100,000 issues and pull requests. Probably the second most on GitHub (1. is vscode).

[Codeberg-org]

Even if API limits are hit, migration should not fail but get throttled, too? Ideally with some estimate of ETA?

[jolheiser]

It seems we do have some code for sleeping if we hit a rate limit.

gitea/modules/migrations/github.go

Lines 113 to 128 in f0fe568

	func (g *GithubDownloaderV3) sleep() {
	for g.rate != nil && g.rate.Remaining <= GithubLimitRateRemaining {
	timer := time.NewTimer(time.Until(g.rate.Reset.Time))
	select {
	case <-g.ctx.Done():
	util.StopTimer(timer)
	return
	case <-timer.C:
	}
	err := g.RefreshRate()
	if err != nil {
	log.Error("g.client.RateLimits: %s", err)
	}
	}
	}

Do you have any logs surrounding that rate limit error that may indicate where in the migration process it was?

[davidak]

Dumping to disk may not help if we are hitting a rate limit anyways.

We got this trace 3 times without seeing the api limit message, so it is not related to this issue. We got the limit after trying too often and waited.

[davidak]

Even if API limits are hit, migration should not fail but get throttled, too? Ideally with some estimate of ETA?

That's a separate issue (#13243).

[6543]

@davidak I dont know how expencive it is, but if one of NixOS maintainers buy a GitHub enterprice account for the periode of migration, it would be faster in genera ... (higher api rate limit)

[davidak]

@6543 do you mean GitHub Pro (https://github.com/account/upgrade) or where can you buy it?

That seem like a useful information that should be part of documentation.

But you should not have to buy your data free in general.

[jolheiser]

@davidak He's referring to https://github.com/enterprise which is basically self-hosted GitHub.

[6543]

@jolheiser no i thought I saw something for github itseve to icrease api request limit ...
... was searching in the docs but could not find it anymore

a other trick we could to: allow multible tokens for github migrator ... to switch token when one got ratelimited?
@davidak what do you think?