update-repos: Should it stay or should it go?

[cgrindel]

Overview

With the introduction of Bazel modules (bzlmod), executing update-repos for Go and some other languages is no longer necessary. However, some language implementations do have a use for update-repos (e.g., rules_swift_package_manager) even when bzlmod is enabled. The question to the community is what should become of update-repos in Gazelle?

Developer Experience

This section briefly introduces the developer experience/ergonomics of using a bzlmod-enabled repository with and without update-repos. It is not a critique of the implementations. It is meant to describe the developer experience and how update-repos plays into that experience.

Go Implementation (without update-repos)

The current Go implementation moved much of the logic in update-repos into the go_deps extension. The result is that a client needs to do the following to add Go-Gazelle support (see this doc for more details):

Initial Setup

Assumption: The go.mod file already exists.

1. Add bazel_dep declarations for rules_go and gazelle to their MODULE.bazel file.
2. Add gazelle(name = "update_build_files") and a gazelle:prefix directive to the BUILD.bazel at the workspace's root.
3. Run mod tidy using bazel run @rules_go//go -- mod tidy -v.
4. Generate build files using bazel run //:update_build_files.
5. Run a build (e.g., bazel build //...). Build fails.

Assuming that external dependencies are listed in go.mod, there should be a build error and a warning about missing declarations in the MODULE.bazel file. The warning looks like the following:

WARNING: /Users/chuck/code/cgrindel/bazel-starlib/fix_workspace/MODULE.bazel:31:24: The module extension go_deps defined in @bazel_gazelle//:extensions.bzl reported incorrect imports of repositories via use_repo():

Not imported, but reported as direct dependencies by the extension (may cause the build to fail):
    com_github_creasty_defaults, com_github_ekalinin_github_markdown_toc_go, com_github_stretchr_testify, in_gopkg_alecthomas_kingpin_v2, in_gopkg_yaml_v3

 ** You can use the following buildozer command(s) to fix these issues:

buildozer 'use_repo_add @bazel_gazelle//:extensions.bzl go_deps com_github_creasty_defaults com_github_ekalinin_github_markdown_toc_go com_github_stretchr_testify in_gopkg_alecthomas_kingpin_v2 in_gopkg_yaml_v3' //MODULE.bazel:all

6. Run the buildozer command.
7. Run a build: bazel build //.... Build succeeds.

NOTE: An update-repos command was not defined or executed against the repository.

Add an External Dependency

To add an external dependency, you do the following:

1. Get the Go dependency: bazel run @rules_go//go get golang.org/x/text@v0.3.2.
2. Import the Go package in the source code.
3. Run mod tidy using bazel run @rules_go//go -- mod tidy -v.
4. Generate build files using bazel run //:update_build_files.
5. Run a build (e.g., bazel build //...). It will fail and provide a buildozer command.
6. Execute the buildozer command.
7. Execute the build. Build succeeds.

rules_swift_package_manager Implementation (with update-repos)

The rules_swift_package_manager implementation splits the logic for external dependency management between the update-repos implementation and the swift_deps extension.

For details on the use of rules_swift_package_manager, please see this doc.

Initial Setup

Assumption: The Package.swift file already exists.

1. Add bazel_dep declarations for rules_swift, rules_swift_package_manager and gazelle to their MODULE.bazel file.
2. Add gazelle(name = "update_build_files") and gazelle(name = "swift_update_pkgs", command = "update-repos") to the BUILD.bazel at the workspace's root. (In actuality, rules_swift_package_manager provides a macro that defines two flavors of update-repos targets.)
3. Run the update-repos command: bazel run //:swift_update_pkgs.
4. Generate build files: bazel run //:update_build_files.
5. Execute the build: bazel build //....

At this point, the developer's build should be green. The reason that this can happen is that the update-repos implementation performs a number of steps behind the scenes:

1. Resolves the transitive dependencies.
2. Collects information about the Swift packages, their modules, and the Bazel targets that correspond with the modules.
3. Writes the information from the previous step to a JSON file called swift_deps_index.json.
4. Updates the MODULE.bazel file with the appropriate extension code (e.g. use_extension, use_repo).

Add an External Dependency

1. Update the Package.swift file with the dependency.
2. Run update-repos: bazel run //:swift_update_pkgs.
3. Import the Swift module(s) in their code.
4. Generate build files: bazel run //:update_build_files.
5. Execute the build: bazel build //....

Comparison of with/without update-repos

Both types of implementation achieve success for the developer!

Without update-repos

• Pro: All external dependency code lives in Starlark. The only logic outside Starlark is the build file generation logic.
• Con: Requires an unsuccessful build to be told about the missing extension information.
• Con: Requires running an additional command to make the build green (e.g., buildozer).

With update-repos

• Pro: Better developer ergonomics. Does not require an unsuccessful build to complete the process.
• Pro: Faster to green. Fewer commands to run.
  • Without update-repos: build, bulldozer, build
  • With update-repos: update-repos, build
• Neutral: Can process external dependencies outside of build and cache the result. Example: swift_deps_index.json
• Con: Some duplication of dependency logic between Gazelle code and Starlark code.

Summary

While some languages have moved away from using update-repos relying on Bazel's suggested buildozer command, I believe there is real value in languages that provide the extra code generation in update-repos. I propose the following:

1. At a minimum, keep update-repos in Gazelle.
2. Possibly incorporate facilities in Gazelle to make updating code in the MODULE.bazel file easier.

What are your thoughts?

[fmeum]

Thanks for compiling this great summary!

I have a few points to add:

Without update-repos

• Pro (specific to rules_go): Since Gazelle depends on the go_deps extension, running bazel run //:update_build_files should already trigger the buildozer warnings for this extension - step 5 thus shouldn't be needed.

With update-repos

• Con: If the update-repos implementation for a particular language foo depends on dependencies fetched by the foo_deps extension, it may fail to build if use_repos refer to non-existent repos - before it gets a chance to fix them. This mostly applies to Go, but may present bootstrapping challenges for other languages.
• Con: In addition to parsing dependency files, update-repos may need to parse the MODULE.bazel (e.g. for go_deps.module). This isn't too difficult now, but may become more difficult in the future when loads are introduced, which makes the difference between parsing (update-repos) and evaluating (Bazel) more noticeable.

But even considering them, at least for languages other than Go, the update-repos approach currently has the better developer experience (and worse "maintainer experience").

I think that we should aim to improve the situation by reducing the friction of the "without update-repos" approach, which I believe can be achieved rather easily:

1. Introduce a new fix subcommand of bazel mod that evaluates (a subset of) all module extensions and automatically applies all their buildozer fixes. This makes the experience when adding a dep equivalent to that of update-repos by removing the additional command: Just replace update-repos with bazel mod fix. It also helps with the initial setup by solving a weak point of the use_repo warnings: They aren't emitted when the extension is never evaluated, which can easily happen before the user has added the first use_repo for it.
2. As a follow-up to 1., think about running bazel mod fix automatically after a dependency has been added. For Go, this could mean running bazel mod fix gazelle (or whatever we believe is good syntax for evaluating gazelle module extensions only) as part of bazel run @rules_go//go mod ... and bazel run @rules_go//go get .... With this, we could get down to zero additional commands: Add the deps, regenerate BUILD files if necessary and the next bazel build //... will just work.

I would be very interested in hearing your thoughts on this. My hope is that this approach allows us to achieve the best outcome for both target demographics: Very low friction for developers that is consistent across rulesets and low maintenance overhead for ruleset maintainers by having everything in Starlark without the need to write a Gazelle plugin for dependency management (BUILD file generation is of course a different story).

For the meantime, update-repos is definitely a good strategy with certain upsides and I don't think we should deprecate it. When the Bazel-provided tooling for use_repo management has matured, it may not be needed anymore, but even then I could still see applications (e.g. generating use_extensions during initial setup, updating hashes, ...).

CC @linzhp @tyler-french

[cgrindel]

Con: If the update-repos implementation for a particular language foo depends on dependencies fetched by the foo_deps extension, it may fail to build if use_repos refer to non-existent repos - before it gets a chance to fix them. This mostly applies to Go, but may present bootstrapping challenges for other languages.

Two thoughts:

• I agree that this is a Go-only issue.
• From a bootstrap perspective, the update-repo developer must ensure the dependencies exist. Otherwise, the whole thing does not get off the ground. So, to be nit-picky, this is more of a Neutral in my mind.

update-repos may need to parse the MODULE.bazel

Of course, this depends on the source of truth for the language. Go and Swift have external files that provide this information. Since bzlmod is not meant to replace these mechanisms, can we simplify the problem to "replace the current use_repo declarations with the current list"?

In terms of the complexity when loads are allowed, this is always tricky. However, we can simplify this problem by supporting certain patterns, just like the legacy WORKSPACE support in update-repos. With legacy loading, gazelle would either update the WORKSPACE or create/update a macro in a Starlark file.

Introduce a new fix subcommand of bazel mod...Just replace update-repos with bazel mod fix

I am very much in favor of moving these capabilities closer to Bazel itself.

I presume that extensions would need to have two hooks or be told which mode it was in. The code for "fixing" can be different from the code for loading.

[fmeum]

From a bootstrap perspective, the update-repo developer must ensure the dependencies exist. Otherwise, the whole thing does not get off the ground. So, to be nit-picky, this is more of a Neutral in my mind.

It's unfortunately more difficult than that: Consider a situation where the end user adds use_repo(go_deps, "foobar") to their MODULE.bazel file and then runs bazel run //:gazelle -- update-repos. Since go_deps doesn't provide the foobar repo, the evaluation of the go_deps extension fails and with it the build of Gazelle itself - in particular, the build never gets to a point where Gazelle could run and remove that use_repo in the MODULE.bazel file. This problem can't be avoided by the Gazelle/update-repo developer.

I presume that extensions would need to have two hooks or be told which mode it was in. The code for "fixing" can be different from the code for loading.

Could you explain which two modes you would expect here? How it works right now is that an extension can optionally return a list of root_module_direct_deps (repos it believes the root module should use_repo on a use_extension with dev_dependency = False) and root_module_direct_dev_deps (repos it believes the root module should use_repo on a use_extension with dev_dependency = True). There are no different modes, this metadata is returned whenever the extension runs.

Currently, it translates into warnings with buildozer fixup commands, but in the future bazel mod fix could automate this.

[cgrindel]

Consider a situation where the end user adds use_repo(go_deps, "foobar") to their MODULE.bazel file

The real issue here is that the developer added the dependency incorrectly. If a procedure is provided to add a dependency and the developer does not follow it, there will be trouble. 🤷‍♂️

Could you explain which two modes you would expect here?

I was thinking about how rules_swift_package_manager pre-computes information about the external dependencies. In this scenario, the extension would need to know whether it was performing the pre-compute or just loading the information.

[cgrindel]

BTW, I started a separate discussion about pre-computing external dependency information.

[fmeum]

The real issue here is that the developer added the dependency incorrectly. If a procedure is provided to add a dependency and the developer does not follow it, there will be trouble. man_shrugging

This can happen naturally though:

1. Dev makes changes to a Go file that result in the last import for a certain third-party dep being dropped.
2. Dev runs bazel run @rules_go//go mod tidy to update go.mod.
3. From now on, builds (including of Gazelle itself) will fail as the existing use_repo for the third-party dep is still there, but the backing repo isn't when the extension reruns due to the go.mod change.

[linzhp]

Since Gazelle depends on the go_deps extension, running bazel run //:update_build_files should already trigger the buildozer warnings - step 5 thus shouldn't be needed.

Does this trigger the missing use_repo warning for only Gazelle's dependencies or all dependencies?

[fmeum]

Only for go_deps, so this point is really only valid for Go or other languages that are deps of your Gazelle binary. It comes at a price though as issues with the tags and use_repos for these extensions can prevent Gazelle from running in the first place.

[aherrmann]

I'm not sure I fully understand the implications of the options. So, I'll just present the current update-repos use-case in rules_haskell and ask what an alternative approach without update-repos could be.

For rules_haskell we have a Gazelle extension called gazelle_cabal (Cabal is a Haskell package manager). It goes through a project that may contain multiple Cabal files (Cabal package descriptions). For each Cabal file, it generates Bazel targets corresponding to the components of this package (a package may have multiple library, binary, test components). It resolves dependencies between Cabal packages within the project, and assumes that third-party packages are provided in an external workspace called @stackage (configurable).

The update-repos component goes through all Haskell targets, collects all references to targets under @stackage, and then generates the WORKSPACE import for these third-party packages. I.e. the information flow is Cabal files --> BUILD files --> WORKSPACE.

What would an alternative approach without update-repos look like?

We currently don't have a direct line between the Cabal files and the WORKSPACE/MODULE file. So, unless we change that, we would require some intermediate tool that collects all third party dependencies and writes them into WORKSPACE/MODULE.

[fmeum]

How many Cabal files are there in a typical large project?

If there aren't too many, then the user could pass their labels to a module extension that translates the Cabal files into the set of all deps it needs to materialize under @stackage (or in separate repos for better strict deps support).

If that isn't feasible, a tool seems like the best bet, but it could also just produce a JSON file read by a module extension instead of directly emitting repo rule instantiations.

[aherrmann]

Sorry for the late reply.

How many Cabal files are there in a typical large project?

Depends on the project, but in most cases probably between one to some tens or so. Cabal packages tend to be relatively large and coarse grained.

then the user could pass their labels to a module extension [...]

That does sound feasible.

a tool seems like the best bet, but it could also just produce a JSON file read by a module extension instead of directly emitting repo rule instantiations

That is also a nice idea. In fact, Cabal did introduce a "project file" a while ago that could take that role. However, it has a bespoke syntax and supports globs, so it still needs a tool to parse it.