Moving Away From GitHub

[Noordfrees]

Three and a half years ago, Widelands moved from Launchpad to GitHub. This was a great move at the time, since we swapped the ancient, outmoded Bazaar for Git, which is superior in literally every way. Later it also gave us the GitHub Actions CI (remember Travis anyone?).

However, GitHub is now going the way Sourceforge once went. Lots of high-profile projects already announced their move away from GitHub when the ongoing Copilot copyright mass infringement debacle begun. Now GitHub is upping its game further by pushing out mandatory Two-Factor Authentication (2FA) to all users who contribute code.

On the one hand, 2FA is great to reduce the risk of threat actors taking over your account, but on the other hand, if you don't want to trust GitHub's authentication apps not to contain any spyware and don't want to tell them your phone number, you now have a big problem. I for one refuse to trust them with that.

My proposal is that it is time to move on and change our code hosting provider again. And there's an alternative already in place, because…

Launchpad now natively supports Git repos!

That's right. We can move back to Launchpad, which several of the developers are already familiar with, and we will not have to give up Git. We will not lose any of our commit and branch history. The only change in your local setup will be to change the remote URLs from git@github.com:widelands/widelands.git to git+ssh://git.launchpad.net/widelands – or even just to lp:widelands after a two-line config change explained in Launchpad's very helpful Git overview page.

The main downside is that we would lose access to GitHub Actions. For this I propose to pull @bunnybot from his retirement (or rather to write a new one, since the old one is Bazaar-specific). He can mirror master and merge request branches to GitHub for us just like in the old days. This also means that the fact that Launchpad's merge request UI is still the same as always is a non-issue, we'll just use the good ol' @bunnybot merge again.

Regarding practicalities, the old repo on LP would be renamed and archived, and a new repo created which would be the same as the one currently hosted here (not a copy, but actually the same commit objects).
For deployment of daily builds, we'll keep using GitHub. The PPA can again be built natively.
So you can simply push your branches to the new origin; no conversion will be necessary (unlike when we migrated from Bazaar to Git – this now is a much easier transition, since we don't change our version control system).
We will need to migrate all issues to Launchpad's superior bug tracker and implement the mergebot.

---

As an example, I pushed a copy of current master to my +git folder. You can get it here:

 git clone git+ssh://nordfriese@git.launchpad.net/~nordfriese/+git/widelands-git-test 

The commands I used in order to make it possible were:

# Adapt gitconfig
echo '[url "git+ssh://nordfriese@git.launchpad.net/"]' >> ~/.gitconfig  # Replace "nordfriese" with your LP user name (no leading tilde)
echo '       insteadof = lp:' >> ~/.gitconfig

# Add my repo
git remote add lptest lp:~nordfriese/+git/widelands-git-test
git fetch lptest

# To push
git push lptest master
# To checkout
git checkout lptest/master

For pushing to an actual project instead of a standalone experiment branch, it would simplify to

# First-time setup
git remote add origin lp:widelands
git fetch origin

# Regular workflow
git push origin branch_name  # Looks familiar ;)

Updated to add: I also moved three of my personal projects which I kept on GitHub to Launchpad today. Took me about half an hour total, though those had no issues to migrate. It all went very smoothly, not a hitch.

---

Opinions?

@widelands/developers Please discuss. This affects everyone.
If we stay on GitHub and they don't reverse their policy (and it currently doesn't look like they will), developers who don't accept GitHub's unreasonable antics will leave (including me, though I really don't want to). Should we go to Launchpad? Or another hosting provider, such as GitLab? What are your thoughts on the matter?

[matthiakl]

if you don't want to trust GitHub's authentication apps not to contain any spyware and don't want to tell them your phone number, you now have a big problem. I for one refuse to trust them with that.

I do not really see a problem here. There are plenty of open source TOTP apps which can be used as an alternative (I for example use FreeOTP v1.5). If I understand that blog post correctly, using SMS or the Github App are just different options, but any TOTP app can do the same job.

Lots of high-profile projects already announced their move away from GitHub when the ongoing Copilot copyright mass infringement debacle begun.

If we follow this path to stay FOSS, we would need to leave Github completely. I understand the predicament here, but personally don't think we should move away from Github soleley for this reason.
Reading through GiveUpGitHub.org gave me some hints on other reasons. However, since launchpad is also proprietary software, maybe another hosting service suche as https://codeberg.org (public instance of gitea) might be interesting.

Conclusion: I vote for staying at Github.

[frankystone]

@matthiakl can you explain how TOTP works in practice? I have tried to find it out myself, but i have found only technical details without an example how it works in practice. From what i read between the lines no phone number is needed?

[matthiakl]

From what i read between the lines no phone number is needed?

Correct. Version 2.x of FreeOTP requires an App password to access the stored keys, so I still use v1.5.
To activate it, you get a secret in form of a QR code. You scan this code from within any TOTP app to add it to your collection.
After logging in with your password on Github, you are asked to enter the 2FA code, so you tap on the Github entry to generate a code which is only valid for some seconds. Enter it and you are logged in. For this mechanism to work, you need to make sure, that your devices have the correct times (TOTP=Timed One Time Password), but (I'm not sure about the technical details on this.
Example from https://freeotp.github.io:

Some alternatives

There exists also oathtool, a cmdline tool for TOTP. Just note the secret during generation somewhere else (click on "enter this secret instead", see below) and feed it into the tool to generate the code (oathtool --base32 --totp $otpkey)
KeepassXC also supports TOTP generation.

[Noordfrees]

I had always thought that this way of authentication worked exclusively via apps, I didn't know it can also be done with a CLI tool – thanks for sharing that.

However I may not have been clear about the rationale of this discussion. I only mentioned Copilot as a reminder that calls from well-known and respected organizations to leave GitHub are already happening and being followed for quite a while, and that this is not the first time GitHub is alienating users. Copilot, for all that I dislike what GitHub is doing there, is not however a reason for me personally against using GitHub. Maybe they are infringing on Widelands's license. Who can tell. But if they are it's definitely not hurting us in any way.

My main reason why I find this new policy intensely inacceptable is that in my opinion GitHub simply has no business ordering everyone to use 2FA. It should be up to every user themselves to how much trouble and inconvenience they want to go to protect their accounts and find their own balance between convenience and security, like on pretty much almost every service I can quickly think of. I can understand the rationale behind requiring extra security for maintainers of important libraries that get millions of downloads every day ("understand", explicitly not "endorse"), or allowing companies with business secrets to protect to demand it from their employees – but everyone who merely wants to use GitHub? It will also serve to discourage new contributors from just quickly submitting an issue report or a patch if, instead of just signing up, they additionally have to install and configure an app. GitHub is absolutely overreaching what's reasonable and acceptable by requiring that. Just no.

[tothxa]

I guess I'd already swallowed the bitter pill when I decided to sign up for gh just over 2 years ago. IIRC by then they had already been well down the current path, and Free Software advocates had already been warning against them. The 2-factor account safety smokescreen tap dance would probably only be a minor inconvenience, as long as some CLI tool works for me as advertised. Or even an improvement over getting OTPs in e-mail when my IP address changes… So I think I would be OK with staying.

As for lp, I understand that Widelands has a long history there, and most of you are familiar with it. But I tried to look at some git projects there, and I couldn't see how its bug tracker is superior, neither how pull requests work there. And canonical isn't all that much better these days than "the big one" behind gh. Just look at their recent move trying to force users to use their snap format for an example.

Codeberg looks much better at first sight, and it looks like they try to copy gh's UI as much as they can, so they clearly try to offer a home for gh refugees. But I understand that this is of little importance for those of you who like lp.

---

Would we also leave transifex for something less scary to sign up to?

[hessenfarmer]

Well, moving back to canonical/launchpad while still using github for the CI sounds like moving from the frying pan into the fire.
Gitlab on gitlab.com) in its licence to attract FOSS has many of the downsides that are reported for github as well (namely being closed source and only offering a restricted opensource version for self installation.
Practically to use bunnybot would require a 2FA as well won't it? And that would need the bunnybot maintainer to maintain this as well, right?
So in my opinion it would be the right way to completely think about the whole topic and evaluate options to move to a new hosting and CI environment that would fulfil our needs while respecting the FOSS mindset. For me a convenient move to Launchpad would not fulfil this.
However personally I don't have much problem with a 2FA. So for me we have some time to evaluate some options.
The more tricky part would be to identify some CI chains that provide the same or at least something near of what github actions provide

[saulkim]

If Widelands moves to a different provider, what will happen to this Github repo?

I bookmarked many merged PRs and I am worried that the links may be dead later when I have time to study the changes.

[andy5995]

If there's a concern about GitHub(Microsoft) introducing spyware or you can't trust them with your data, then the choice to move away from it is pretty easy.

I'll just post some random thoughts I had:

I wonder what the odds are of LaunchPad going the way of SF and GitHub. Is this just an unstoppable trend where developers have to keep switching services every few years? Projects that used to use Transifex have switched to Weblate. How long will it be before Weblate is acquired?

0ad, Blender, and Devuan have either moved (or is planning to move) to self-hosted instances of gitea.

The CI available for codeberg appears to be imminent, but in it's infancy. https://codeberg.org/Codeberg-CI/request-access

[frankystone]

My main reason why I find this new policy intensely inacceptable is that in my opinion GitHub simply has no business ordering everyone to use 2FA. It should be up to every user themselves to how much trouble and inconvenience they want to go to protect their accounts and find their own balance between convenience and security, like on pretty much almost every service I can quickly think of. I can understand the rationale behind requiring extra security for maintainers of important libraries that get millions of downloads every day ("understand", explicitly not "endorse"), or allowing companies with business secrets to protect to demand it from their employees – but everyone who merely wants to use GitHub? It will also serve to discourage new contributors from just quickly submitting an issue report or a patch if, instead of just signing up, they additionally have to install and configure an app. GitHub is absolutely overreaching what's reasonable and acceptable by requiring that. Just no.

I fully understand your point and i feel the same. If it is possible to use 2FA without giving my phone number i am fine with that, although it is additional work. But if i understand correctly 2FA is only needed for logging in. Since i do not delete the github cookies regularly (so i am always logged in) it's not really annoying for me, i will need 2FA probably twice a year or so.
For bug reports we already transfer bugs reported on the website to bugs on GH. The most bugs are anyway reported by developers. For new contributors it's a show stopper, that's probably true, but i guess most new widelands contributors have a GH account anyway.

[hessenfarmer]

but i guess most new widelands contributors have a GH account anyway.

The concern is that more and more people interested in developping FOSS games will leave Github and by this the pool of potential contributors on github is shrinking. I do not know how far this process has got so far, however I think we definitly need to think about a move, but we should consider different options.
maybe we could just try some of the alternatives, before making the final decision. (I am just migrating my fork to codeberg while typing this.)

[hessenfarmer]

Well in our issues we already have the first remark of a bug reporter that he will not post anymore on github. So we really should think of moving as I believe the barrier for reporting bugs and enter contribution is high.
Furthermore I respect and understand the personal discomfort of our chieftain and from my opinion I really like to avoid losing the contributions of @nordfriese

Edit: most bug reports really had been provided on github recently and only a few came in from the website. Regardless where we go would it be possible to provide the bug report template on the website and use the website code to push such a template as a git issue to wherever the repo is hosted?

[klaus-halfmann]

I think 2FA is a good idea, we could allow some service account to push bugs from our homepage to github so the avaverage user would not need an account on github. That KI driven Copilot has a smell I do not really like, but I have no final opinion on that one. I actually attended some conference where they advertised it https://www.thedevopsconference.com/videos (But I can not find that video again). So basically I will go wherever widelands will go :-)

[frankystone]

Just to be clear: I am not against moving away from github. From a political view i would vote for a move.

[luxxyarns]

Hi guys, seems somebody was confused with my name @nordfriese vs @Noordfrees :)

anyhow, I'm always open for repository alternatives

[hessenfarmer]

Hi,
As said I have just cloned / migrated my fork to codeberg for testing it. everytheng went fine ans smooth as they provide a webinterface to just pull everything from github (or others).
the web interface looks familiar. they also have the possibility to define a repo mirror with github. Perhaps this could be used to trigger the Github actions (not sure whether such a mirror would need 2FA) until we may find a new CI provider.

[hessenfarmer]

i just had a look at the cloned repo of @Noordfrees on launchpad and I found one negative point while viewing the diff of a recent commit.
in LP i could not find a possibility to show the surrounding code in the diff. While at codeberg this is the same as here on github.

For doing reviews I always found it helpful to make some lines above and below the actual change available as it is often affected or contains necessary information to understand the change. So not having such posibility is a negative for me.

To see the interface at codeberg you might use my cloned fork https://codeberg.org/hessenfarmer/widelands.

To see what I mean:
click on a commit to see the diff. notice the little arrows on the left to show more lines. try the same in the launchpad repo of @Noordfrees

[hessenfarmer]

ok I tried the context but it still lacks the experience you can have with codeberg as it is restricted to 40 lines. Furthermore changing it takes longer then is less responsive then thje codeberg UI.

However can we have a page where we could list the alternatives and collect pros and cons?
Shall we do such lists separatly for Versioning system and CI providers?

[tothxa]

However can we have a page where we could list the alternatives and collect pros and cons? Shall we do such lists separatly for Versioning system and CI providers?

Wouldn't it be more appropriate to move this discussion to the forums on widelands.org? And the proposed evaluation page could be maintained in the wiki.

[Noordfrees]

I created a wiki page: https://www.widelands.org/wiki/CodeHostingAndVCS/

Please expand with more points

I really don't think we should leave Git for another VCS though

[tothxa]

I really don't think we should leave Git for another VCS though

Fully agree. I think that section can be removed from the wiki page.

[hessenfarmer]

I really don't think we should leave Git for another VCS though

me neither. I think it is very clear that we will use git. Therefore we need to move to a git repo.

[frankystone]

For codeberg it's difficult to find some restrictions about repos. As far i understand this issue there is a limit of 100 repos.

The FAQ says there is no limit in size So maybe the media-repo currently hosted on launchpad due to the big size can also be migrated to codeberg. On the other side their terms of use say:

"3. Forks, migrations and testing repos are considered as inactive when they don't contain unique contributions and are inactive for more than a month. They shouldn't be kept for a prolonged amount of time, and thus might be removed after notifying maintainers and providing a 90 days period to ask for preservation."

Some irrelevancy:
Codeberg is a very young project, founded 2018 with the working tile "TeaHub.io" and finally launched as codeberg.org on the first January 2019. It is a registered association (eingetragener Verein) but they do not (yet) provide any information about their finances. librepay says codeberg gets 228,96 € per week, but this does not include member dues of the codeberg e.v.

I like the codeberg project and may get a member of codeberg e.v.

[SirVer]

Deleting code after 90 days of inactivity sounds scary though.... I think Widelands itself would have been deleted a few times already, not mentioning the website, the meta server, the media repo...

[SirVer]

Jumping in here as original project founder and long time absentee in the development. So I do not think my thoughts should be weighted super strongly.

1. OTP is a great idea. To me, this is like the seatbelt that people didn't want to use when it was first introduced - infringing on rights was mentioned back then - but has been saving lifes ever since. As a CTO of a company where I strongarmed the use of OTPs I have a lot of sympathy for GitHub. Fact is, only one person being sloppy with their password hygiene can ruin a project, so I think enforcing OTP is a good thing, a bit like vaccinating. So to me this is a positive, not a negative that they do that.

2. GitHub has still and will retain the lowest barrier to contribution for a long time. I find it strange to consider moving to a self hosted repo, launchpad or codeberg (which I never heard before) if the argument is that we will loose contributors if we do not do so. I am certain that we will loose a lot of contributors too if we do - just the hassle of setting up accounts, keys, trusting some self hosted service with credentials - my hunch is probably more.

3. As for copyright infringement: Widelands was always about learning for me, I take pride that I learned programming here and I saw so many contributors grow so quickly working on the project. If this can feed some AI that can then teach others I am happy, I do not need to be recognized as the "orginial author" of the code for that.

4. Free & open source is in the vain of Widelands. It is GPLed after all. GitHub cannot take that away from us.

I'd stay on GitHub, but as I said I am hardly contributing lately, so I am not angry if my opinion is not weighted heavily.

[hessenfarmer]

So we have at least some opinions for now.
Some people voted to stay.
Some people including our current chieftain stated they would leave.
Some people including me would go wherever widelands will go and this includes staying here.

So what shall we do?
If we stay we might loose @Noordfrees and probably some other contributors which might lead to a split or fork of the project.
If we leave we might loose some contributors and we might loose visibility as github is still the biggest platform and the opensource community is going to divert to multiple new hosters. So we might attract even less contributors.
As the CI with github actions is still the most inclusive ( in terms of OS supported) and fastest available at no cost, leaving completely would give us some trouble.
Selfhosting and running a repo and CI/CD would be possible but probably not without costs..

taking this into account I evaluated the Codeberg repo a bit more, and I was able to sync my github fork with its codeberg mirror. (all pushes to codeberg are automatically pushed to github as well). Furthermore I was able to trigger appveyor builds from the codeberg repo as well, by using webhooks.
By this it came to my mind whether we might attempt to have a mirrored development environment, to avoid the decision conflict described above.
What is missing currently is:

• synced PR
• synced Issues

[frankystone]

I think selfhosting can be omitted because there aren't enough developers to set it up and, above all things, to maintain it.

[pospec4444]

Can someone estimate requirements for machine running self hosted git + CI?

…

On Fri, Apr 7, 2023, 18:24 kaputtnik ***@***.***> wrote: I think selfhosting can be omitted because there aren't enough developers to set it up and, above all things, to maintain it. — Reply to this email directly, view it on GitHub <#5834 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAS2MQYP6TQH4AU5ZG55ZE3XAA5SDANCNFSM6AAAAAAWNCIQTE> . You are receiving this because you are on a team that was mentioned.Message ID: ***@***.*** com>

[hessenfarmer]

well git should not be much of a problem except storage size, as gitea (which is used by codeberg) is quite a lightweight application.
but the ci would be the problem as we have about 24 build jobs to compile the sources currently. to be finished in reasonable time each job should need 4 cores and about 4 to 8 GB of RAM. and we should at least run 8 of the jobs in parallel resulting in about 6 hours for each commit to each PR and to each commit to master.

[frankystone]

Maintaining would be the most problem i think. See the hassle of maintaining the current CI which needs a lot of time already.
Funnily Gitea itself is hosted on GitHub and has 2k of issues and 247 open PR's. Nothing against using gitea but who wants to take the work and maintain a selfhosted instance of it permanently?

Maybe i am wrong but the hassle of maintaining our django installation (website) shows that it is probably easier said than done. At least on the long run.

[hessenfarmer]

I fully agree.

[tothxa]

Funnily Gitea itself is hosted on GitHub

Just for the record, Codeberg now uses the latest community fork of gitea, called Forgejo, which is hosted on Codeberg.

[Noordfrees]

Although selfhosted would be the best option regarding being able to control the experience for all contributors long-term and never having to move again, I think the effort involved to maintain it and keep it up to date and backed up permanently is not feasable for a project like Widelands. CI integration is also an issue; we'd still need a bunnybot-style automated mirror for integration with external CI providers, and hosting CIs for all three OSs we build on (Ubuntu, MacOS, Windows) is not feasible.

---

I do like the idea of a mirror between GitHub and Codeberg, especially as one Git commits tree can quite easily be hosted in multiple places. This would offer us the combined best of two worlds. If issues and PRs are the only things we need to sync, a simplified two-way mirror can probably be set up with not too much effort using API calls from a bot account with GitHub's API (which is quite developer-friendly) and Codeberg's API (which I can't comment on).

[frankystone]

If i understand correctly 2FA is the main reason for @Noordfrees to leave github. Anyway codeberg (only) recommends 2FA. If we move to codeberg and codeberg makes 2FA mandatory some when, there is nothing won for @Noordfrees ?

[Noordfrees]

Since there seems to be support for both GitHub and Codeberg, the best solution would appear to be to as previously discussed use both. I have written a mirror bot to continuously mirror issues and PRs both ways, so every contributor can choose whether they want to use GitHub or Codeberg or both interchangeably. See https://codeberg.org/wl/wl_addons_server/pulls/77 for an example how this looks in practice.

The bot is currently in beta phase with the add-ons server repo as its test case. I'd appreciate some testing and feedback about this bot, so we can eventually deploy it as the solution for the other repos (Widelands, Website, and if desired Metaserver and IRC Relay) as well.

Also, everyone who would like to use Codeberg please tell me your Codeberg username so I can add you to the https://codeberg.org/wl organization.

[tothxa]

Also, everyone who would like to use Codeberg please tell me your Codeberg username so I can add you to the https://codeberg.org/wl organization.

Just to confirm: Same as here. :)

[klaus-halfmann]

I got myself an Account on codeberg by now. Name is obvious Klaus_Halfmann

[Noordfrees]

I received my 2FA "eviction notice" from GitHub today, from Mid-October onward I will no longer be active on GitHub. I'd been hoping to be able to complete the migration near the end of the year (after the v1.2 release at least), but now the timeline has to be steeped up a little to keep everything smooth.

All the principal bugs in the mirror bot seem to have been ironed out by now. I would like to migrate the widelands-metaserver, widelands-translation-reports, and widelands_irc repos within the next few days to verify that the initial migration runs smoothly (esp. regarding issue #6). And then widelands-website and widelands itself should be migrated mid-September to complete everything and make our Codeberg environment 100% functional.

Are there any objections to this plan?

[tothxa]

I'm about to open 2 enhancement (draft?) PRs. Should I postpone them until you're finished fighting the rate limiting?

BTW are you trying to move the whole github bug and PR archive, including all closed items?

[Noordfrees]

Feel free to open them, the bot should not have any problems with that.

I'm copying the whole bug archive including all closed issues, just like with the launchpad migration, because it's useful to have this available and be able to cross-reference and comment on and reopen old bugs. We can't newly mirror already-closed PRs though because the head branches no longer exist.

[Noordfrees]

Migration is running now, all the major problems seem to be sorted out. I added lots of caching to cut down on API usage, and added some extra slowdowns for bulk requests. It will probably take until tomorrow evening or so several more days.

https://codeberg.org/wl/widelands/

[Noordfrees]

The main migration is now complete. There are still a couple of minor spots of bother specific to this repo that will get fixed over the next days, and the bot will have intermittent downtime until those are all taken care of.

[tothxa]

Do the minor spots include that some bugs were dropped in the migration? gh: 438 open / 4062 closed vs. cb: 433 open / 4022 closed.

Also WIP status of PRs was not mirrored.

Assignment is kind of strange too, though I guess it'll make more sense if dual-site development catches up that assigned to bunnybot means assigned to someone on the other site. I wonder if I made a mistake by assigning myself to one of my PRs on codeberg too…

[tothxa]

gh started to nag me to use 2fa this week. I tried oathtool and it works fine. Can't quite see how it makes it more secure... :P