Transfer community maintained repositories to bazel-contrib

[meteorcloudy]

Since Bazel was open-source in 2016, the Google-owned bazelbuild GitHub organization has hosted a growing number of repositories for the Bazel ecosystem, including Bazel rules, IDE plugins, and remote execution backend. However, many of these repositories are no longer actively maintained by Google teams and rely more on community support.

The Bazel Rules Author SIG was founded in 2021 and governs the bazel-contrib GitHub org. Since then, it has been a home for many Bazel relevant projects, guided by a clear hosting policy. The SIG has also been supported by the community via Open Source Collective and probably by a future open source foundation.

To ensure better maintenance of those projects, we want to migrate the community maintained repositories to bazel-contrib. A repository is a suitable candidate for the move to bazel-contrib if all of the following conditions are met:

• At least one contributor outside of Google actively maintains the repository.
• The source of truth for the repository is not Google’s internal codebase.
• There is no objection from the Google maintainers or stakeholders.

The status of all bazelbuild repositories are summarized in this spreadsheet, with the above standards, we have identified a list of repositories that could be moved to bazel-contrib:

• bazel-buildfarm (@werkt)
• bazel-gazelle (@bazelbuild/go-maintainers)
• bazel-watcher (@achew22)
• rules_dotnet (@purkhusid)
• rules_foreign_cc (@UebelAndre, @jsharpe)
• rules_fuzzing (@stefanbucur, @fmeum)
• rules_go (@bazelbuild/go-maintainers)
• rules_jsonnet (@EdSchouten, @sparkprime, @Globegitter)
• rules_jvm_external (@jin, @shs96c, @cheister)
• rules_kotlin (@ahumesky, @timpeut, @ted-xie, @Bencodes, @restingbull)
• rules_nodejs (@alexeagle)
• rules_perl (@qiwzhang, @skeletonkey)
• rules_proto (@comius, @Yannic)
• rules_scala (@liucijus, @simuons)
• tools_remote (@werkt)
• vscode-bazel (@fweikert, @jfirebaugh, @cameron-martin)

Benefits

• Transferring the repositories sets clear expectations that they are community-supported rather than Google-supported.
  • For bazelbuild repos: Google has full control on the repository and can make any decisions on accepting community contributions. Those projects will always prioritize Google's needs first.
  • For bazel-contrib repos: Community maintainers control this repository and the Bazel Rules SIG or the future foundation support its development. Note that, this doesn't mean Googlers cannot be maintainers of bazel-contrib repos, it's just means the project will prioritize the needs of the open source community.
• The projects will benefit from a well-defined hosting policy that governs their management and operations.
• The Rules SIG will have the opportunity to fund projects that are deemed important by the community, even if they are not prioritized by Google.
• By donating more projects, this transfer helps elevate the Rules Authors SIG towards becoming a Linux Foundation entity, which will be better equipped to support the development of those projects for the community.
• Google CLA is no longer required for contributions to a project under bazel-contrib.

Transfer Process

• Google Approval: To transfer projects from a Google-owned organization to an external one, the Bazel team must undergo an approval process. I will provide an update once this process is complete.
• Rules SIG Approval: In accordance with the Bazel-contrib's Adding Procedure, we can perform a bulk onboarding for all repositories.
• Upon receiving both approvals, we will follow GitHub's guidelines for Transferring a Repository. This ensures that all links to the previous repository location will automatically redirect to the new location.
• Set up the correct permissions for repository maintainers.
• For CI setup, the project may either:
  • Continue using Bazel CI with support from the Bazel team for any necessary updates to CI configurations (benefit from better CI integration with Bazel and the downstream pipeline on breaking changes notification).
  • Switch to GitHub Actions for CI processes (less dependency on Google infra).

Costs

• Project maintainers, the Bazel team, and Rules SIG admins must work together to perform the migration. This will temporarily disrupt normal development work.
• Users must adjust to the new repo location, which will hopefully be seamless with GitHub redirects.
• The migration will result in the loss of the ability to have Kythe index on https://source.bazel.build. Currently, only rules_nodejs is indexed, and it appears to be broken already.

Alternative

• Staying under bazelbuild
  • To ensure clarity for Bazel users, if the current maintainers are unwilling to facilitate the transfer, it would be recommended to include a disclaimer in the README.md file. This disclaimer should state that the project is maintained by the community and is not officially supported by Google.

This discussion is to gather feedback from the community. Please let us know if you think any repository should or should not be moved to bazel-contrib!

[meteorcloudy]

/cc @meisterT @Wyverald @jhfield

[jsharpe]

/cc @jheaff1 who is also a rules_foreign_cc maintainer

[jfirebaugh]

What are the functional differences, if any, between being in the bazelbuild versus bazel-contrib organizations? I'm thinking of things like access to Bazel CI and what repository configuration options are available to maintainers.

[alexeagle]

Currently most (all?) repositories under bazel-contrib use GitHub Actions for CI rather than the Buildkite-based Bazel CI. I don't think it's been an impediment, but it does mean those rulesets are not considered when Bazel core is rolling breaking changes to their "downstream pipeline". So using Bazel CI might make sense for more bazel-contrib rules. OTOH it's nice to be independent of Google, not worry about how many Mac machines are available in a Google datacenter, or the 5am-European-time brownout when they run their nightly jobs.

[meteorcloudy]

All projects using Bazel CI can continue to do so, nothing would change, I added a "Transfer Process" section to the post.

[sluongng]

It's a bit weird to see rules_go, gazelle, and rules_proto are being moved out, but rules_python and buildtools are staying in.

It's also unclear to me what problem are we solving for these projects. And whether the problems outweigh the churn when the entire community has to switch from 1 namespace to another.

Recently we had a similar transfer for toolchains-llvm and IIRC, the SHA256 of some past releases was changed for several archives. Could we make sure that won't happen for the impacted projects?

[sluongng]

Transferring the repositories sets clear expectations that they are community-supported rather than Google-supported.

I read this as: Google no longer supporting these projects. Which I think is fair.

However, I did highlight that some projects such as rules_go and gazelle are core dependencies of some Google OSS projects. It's a bit backward to see these being dropped to community-supported status instead of the reverse: being elevated and staffed with Google engineers.

We have seen this done with rules_rust, rules_python recently, which is why I don't see them on the list here?

The projects will benefit from a well-defined hosting policy that governs their management and operations.

Who will help with the lift and shift?

Are we taxing the existing community maintainers to make this happen?

Or should Bazel team/Rules SIG help out with the transition?

The Rules SIG will have the opportunity to fund projects that are deemed important by the community, even if they are not prioritized by Google.

Realistically, we have had Rules SIG putting bounties for work done within Bazel.git.
So I don't think by s/bazelbuild/bazel-contrib/ things will magically change here.

I would love to hear more regarding how maintainers and users of each impacted project would benefit from these "funding opportunities" as a result of the namespace change though.

By donating more projects, this transfer helps elevate the Rules Authors SIG towards becoming a Linux Foundation entity

This sounds like the main motivation here.
However, it's a motivation for Rules Authors SIG, not for the project maintainers and users.

How does this translate?

---

@meteorcloudy I don't think this is a strong reason/benefit to motivate the move.

I was expecting something more like: Org X is looking to sponsor rules_go with Y minutes of CI, but was unable to do so if the project is under bazelbuild. Moving to bazel-contrib would allow these to happen. But it does not seem to be the case.

I wonder if we are aiming to solve the right problems here.

[meteorcloudy]

I read this as: Google no longer supporting these projects. Which I think is fair.

However, I did highlight that some projects such as rules_go and gazelle are core dependencies of some Google OSS projects. It's a bit backward to see these being dropped to community-supported status instead of the reverse: being elevated and staffed with Google engineers.

I think here are the more clear expectations, I'll add them to the main post:

• For bazelbuild repos: Google has full control on the repository and can make any decisions on accepting community contributions. Those projects will always prioritize Google's needs first.
• For bazel-contrib repos: Community maintainers control this repository and the Bazel Rules SIG or the future foundation support its development. Note that, this doesn't mean Googlers cannot be maintainers of bazel-contrib repos, it's just means the project will prioritize the needs of the open source community.

I do hope relevant Google teams can elevate their support on open source Go rules and will reach out to them, but even that's true, that doesn't mean they cannot support the project under bazel-contrib. Google OSS projects will always have third party dependencies we don't fully control, and that's fine.

Who will help with the lift and shift?

Are we taxing the existing community maintainers to make this happen?

Or should Bazel team/Rules SIG help out with the transition?

I don't think the transition will be super disruptive and the Bazel team and Rules SIG can definitely help. It sounds like you might think this is a very costly process for go maintainers? Do you have any specific concerns?

Realistically, we have had Rules SIG putting bounties for work done within Bazel.git.
So I don't think by s/bazelbuild/bazel-contrib/ things will magically change here.

Fair point, but if they project is located under bazel-contrib, it'll make it easier to justify the funding and less worry about Google not responding to the contributions.

However, it's a motivation for Rules Authors SIG, not for the project maintainers and users.

But the goal of the Rules Authors SIG and the future foundation is to better support the project maintainers and provide better experience to Bazel users, right?

[linzhp]

The major difference is that core maintainers for go rules are all non-googlers
Does this mean if I join Google, rules_go will be kept in bazelbuild?

The underlying technology of Go and Proto are owned and used by Google. These rule sets often need to be updated before or soon after a new version of the underlying technologies is available, so Google controls the direction of these rule sets in some ways.

I hope Bazel team can overcome the bureaucratic barriers some day and have Go and Proto teams support these rule sets. Moving rules_go and rules_proto to bazel-contrib sends a message to the community: Google supports Bazel, Go, Proto individually, but doesn't support their integration and has no plan to support it in the future either.

Is Google is still planning to replace internal Blaze with open source Bazel in the future? If so, those rule sets can be synchronized to Google's internal codebase too.

I think the problem is expectations.

The expectation can be set with a big bold sentence at the top of the README of these projects, which requires less change than moving them to a different org.

[alexeagle]

sends a message to the community: Google supports Bazel, Go, Proto individually, but doesn't support their integration and has no plan to support it in the future either

I think it's intentional to send that message, because it's accurate. I can't speak for the Go team's plans, but I did discuss with protobuf team at length about it.

[meteorcloudy]

The underlying technology of Go and Proto are owned and used by Google. These rule sets often need to be updated before or soon after a new version of the underlying technologies is available, so Google controls the direction of these rule sets in some ways.

I believe it's totally fine that the toolchain and the Bazel rules are maintained by different parties. This is actually true for most of rulesets.

Is Google is still planning to replace internal Blaze with open source Bazel in the future? If so, those rule sets can be synchronized to Google's internal codebase too.

I don't think that would happen in the near future. Even if it does, Google will probably keep using the internal version of Go rules instead of migrating to the open source one.

And +1 for sending the correct message that matches the reality.

[purkhusid]

From rules_dotnet perspective the only thing I'm a bit worried about is switching from Bazel CI to Github Actions.

What I like about Bazel CI is:

• The downstream checks have caught some potential breakage
• It's very fast compared to Github Actions using the free executors
• It's very easy to test using RBE. Moving from Bazel CI would require that we switch to a RBE provider that is free for OSS projects.
• Supports Linux/Mac/Windows on both x64 and arm64

I think that it could be considerable effort to migrate form Bazel CI to Github Actions to maintain the same coverage.

[alexeagle]

@meteorcloudy I think we could quickly setup the GitHub permissions and demonstrate a project under bazel-contrib that uses Bazel CI, to alleviate this concern. I know Bazel CI doesn't want to accept new workloads [1], but it should be an easy case to make that existing workloads are allow-listed.

[1] I couldn't find a reference for this, bazelbuild/continuous-integration#1135 seems like it should mention what new projects you currently accept

[jsharpe]

Likewise for rules_foreign_cc - there is very little maintainer time available for this so sucking that up with a migration of CI to github actions will be detrimental to the maintainability of that ruleset.
I note that buildkite have just made available private preview of hosted runners. Is this something that Google would be open to funding to increase capacity of the Bazel CI system at peak times?

[meteorcloudy]

I added a "Transfer Process" section to clarify this. Projects currently using Bazel CI can continue to do so, the Bazel team will help update CI configs.

[vogelsgesang]

How would this migration work process-wise? Would each of the migrated repositories go through the "Adding Procedure" on its own? Or would there be a bulk onboarding for all repositories?

[meteorcloudy]

Good question, added the "Transfer process" section to explain this.

[vogelsgesang]

Do you already have a rough timeline for this move? Are you planning to execute on this plan already within the next weeks? Still this month? Next month?

[meteorcloudy]

I'm still not sure how long the internal approval process would take, the estimate is at least several weeks and I'll be out of office in most of May, which could slow things down a bit. So my best guess is late May or early June.

[werkt]

What about https://github.com/bazelbuild/tools_remote?
I'd also like to request a name change in the process for "bazel-buildfarm" to just "buildfarm"

[meteorcloudy]

Thanks, I might have missed that one, @bergsieker any objection?

I'd also like to request a name change in the process for "bazel-buildfarm" to just "buildfarm"

I think that's definitely possible. We can discuss the details when we actually start the transfer.

[cgrindel]

FYI. That can cause some headaches for the clients of the repository. When toolchains_llvm moved to bazel-contrib, the name was changed. This caused the SHA values for existing releases to change. The existing clients had to scramble.

That being said, if a name change was desirable, perhaps we could take a different tact. The repo could be forked to bazel-contrib and then renamed. The original repo could tell visitors to go to the new location and then be archived.

[werkt]

I'm aware of the disruption toolchains_llvm caused (and was impacted by it). Still it's the preferred move, and I'd prefer to have an actual redirect as I expect we'll receive from bazelbuild/bazel-buildfarm -> bazel-contrib/buildfarm than a manual jump from a dead repo location. Willing to take the hit on impact for clients.

[bergsieker]

No objections to moving tools_remote to bazel-contrib.

I also agree that changing bazel-buildfarm to buildfarm makes. Buildfarm is essentially independent of Bazel, is expected to work with RE clients beyond Bazel, and has been owned and maintained by werkt essentially from inception. I have no opinion on the technical issues that move might create.

[meteorcloudy]

Thanks, I updated to post to include tools_remote as well!

[jin]

Thanks for the "benefits" section @meteorcloudy. Could you further elaborate on the tradeoffs for making such a move, which includes the known costs (or what will the projects be giving up - perhaps Kythe indexing on source.bazel.build?)? And perhaps a section on alternatives, including what will happen should we remain in the status quo.

I think there's another point to be made about no longer requiring Google CLA in bazel-contrib.

[jin]

tagging another RJE maintainers @shs96c @cheister

[meteorcloudy]

Done, thanks for the suggestions!