feat(bazel): make googleapis a Bzlmod module #855

tyler-french · 2023-09-27T13:51:37Z

This PR adds the googleapis repo to the Bazel central registry as a Bazel Module.

tyler-french · 2023-11-07T15:26:39Z

@fmeum Could you take a look? It's a bit difficult because this package doesn't have versioned releases

fmeum · 2023-11-08T15:53:55Z

MODULE.bazel

@@ -0,0 +1,22 @@
+module(
+    name = "com_google_googleapis",
+    version = "0.0.1",


You can leave this at 0.0.0 and rely on the Publish to the BCR plugin to patch in the real version number when a tag is created. Well, if no tags are created we will need to release this manually, e.g. using Go pseudo-versions.

MODULE.bazel

extensions.bzl

fmeum · 2023-11-08T15:55:56Z

MODULE.bazel

@@ -0,0 +1,22 @@
+module(


We should enable a CI run with Bzlmod (only the default in the not yet released Bazel 7).

I am not sure CI is running in this repo. I will try to set up a BCR presubmit also.

…oogleapis/googleapis/pull/855

silicon-ninja · 2023-12-05T05:58:24Z

Hey everyone,

I'm interested in helping with the transition of googleapis to bzlmod support. While reviewing the dependency graph, I noticed a key decision point: should we opt for pure bzlmod support, or consider a partial approach using use_repo?

Adopting a purely bzlmod-based approach means that all gapic dependencies would also need to support bzlmod. This, in turn, hinges on grpc being a bzlmod project. However, it appears that grpc requires a substantial patch file to support bzlmod, which could complicate matters.

What are the team's thoughts on this? Are there any plans already in place for tackling these dependencies?

alexander-fenster · 2023-12-13T20:42:27Z

Hi @tyler-french, @fmeum, and @silicon-ninja,

Thank you for your suggestions! We indeed cannot immediately switch to bzlmod: gRPC dependency makes it somewhat complicated, but that's not the only issue. For now, we will keep using Bazel 6 (via Bazelisk) and WORKSPACE for some more time, while we think internally how we can switch. We definitely should figure out how to switch soon because of Bazel's plans to deprecate / remove WORKSPACE support.

Please note that we are unable to merge pull requests here, because all Bazel configuration (and protos) in this repository is auto-synced from the internal code storage. I will leave this PR open for discussions, but the code change will come from the regular sync process, so this PR will not get merged.

Thank you!

arnehormann · 2023-12-14T13:43:22Z

Hi @alexander-fenster, isn't it still possible to support both in a soft migration?
See https://bazel.build/external/migration#workspace.bzlmod - a WORKSPACE.bzlmod file that makes Bazel with bzlmod ignore the WORKSPACE file while Bazel without bzlmod will ignore WORKSPACE.bzlmod and MODULE.bazel

SanjayVas · 2023-12-14T18:21:18Z

See bazelbuild/bazel-central-registry#1113

Note that the migration of core ecosystem infrastructure such as this blocks downstream migration. I appreciate in this case that it's in fact upstream dependencies that may be blocking this one, e.g. protobuf and grpc.

My project just uses googleapis for proto definitions in the google.type and google.api packages, so I was able to patch in just enough support in my own Bazel registry.

SanjayVas · 2023-12-14T20:56:11Z

Perhaps the global common component types should be split out from googleapis.

alexander-fenster · 2023-12-15T00:38:00Z

@SanjayVas We actually have https://github.com/googleapis/api-common-protos for this specific purpose, I'll see if we can actually update that one first and see if it helps.

In general, folks, please give us some time and we'll figure this out - I just want to set some expectations that it likely won't happen until the new year. There are a lot of moving parts involved (much more than I would like to have :) ), but we are on it.

SanjayVas · 2023-12-15T02:00:36Z

@SanjayVas We actually have https://github.com/googleapis/api-common-protos for this specific purpose, I'll see if we can actually update that one first and see if it helps.

I am aware, but that often lags way behind the main googleapis repo. It's not automatically kept up-to-date. I imagine that's why the AIPs point to googleapis as the canonical location and why lots of core Bazel infra (e.g. rules_go) depends on it.

ah-edg · 2024-03-04T12:45:00Z

@SanjayVas We actually have https://github.com/googleapis/api-common-protos for this specific purpose, I'll see if we can actually update that one first and see if it helps.

In general, folks, please give us some time and we'll figure this out - I just want to set some expectations that it likely won't happen until the new year. There are a lot of moving parts involved (much more than I would like to have :) ), but we are on it.

Hi @alexander-fenster, I hope it's okay to ping you after 2.5 months.
As it's the new year now, I'd love to get an update if you could please provide one. Even if it's just a rough plan. This issue (here and in grpc / grpc-java) blocks migrations to Bazel 7 and locks us into legacy software and into some maintenance heavy workarounds. I'd really love to get rid of them without additional manual patches.

keith · 2024-03-26T20:12:34Z

MODULE.bazel

@@ -0,0 +1,16 @@
+module(
+    name = "com_google_googleapis",


Suggested change

name = "com_google_googleapis",

name = "googleapis",

this is what most modules are doing, folks can use the fully qualified name if they need to

keith

t doesn't have a name, so I assume you meant module here

keith · 2024-03-26T20:47:41Z

extensions.bzl

+
+        for t in module.tags.use_languages:
+            if is_tag_set:
+                fail("Multiple use_language tags are set in the root module: '{}' and '{}'. Only one is allowed.".format(set_tag_name, t.name))


Suggested change

fail("Multiple use_language tags are set in the root module: '{}' and '{}'. Only one is allowed.".format(set_tag_name, t.name))

fail("Multiple use_language tags are set in the root module: '{}' and '{}'. Only one is allowed.".format(set_tag_name, module.name))

keith · 2024-03-26T20:47:46Z

extensions.bzl

+                fail("Multiple use_language tags are set in the root module: '{}' and '{}'. Only one is allowed.".format(set_tag_name, t.name))
+
+            is_tag_set = True
+            set_tag_name = t.name


Suggested change

set_tag_name = t.name

set_tag_name = module.name

keith · 2024-03-26T20:49:23Z

MODULE.bazel

+    cc = True,
+    go = True,
+    java = True,
+    python = True,


Suggested change

python = True,

python = True,

grpc = True,

It looks to me if you add this some other deps are missing

maybe blocked on bazelbuild/bazel-central-registry#353

SeeYaaG · 2024-03-26T23:05:32Z

Explain. Ihave no clue but definitely interested

…

On Tue, Mar 26, 2024, 3:48 PM Keith Smiley ***@***.***> wrote: ***@***.**** commented on this pull request. t doesn't have a name, so I assume you meant module here ------------------------------ In extensions.bzl <#855 (comment)> : > + "ruby": attr.bool(default = False), + }, +) + +def _switched_rules_impl(ctx): + attrs = {} + for module in ctx.modules: + if not module.is_root: + continue + + is_tag_set = False + set_tag_name = "" + + for t in module.tags.use_languages: + if is_tag_set: + fail("Multiple use_language tags are set in the root module: '{}' and '{}'. Only one is allowed.".format(set_tag_name, t.name)) ⬇️ Suggested change - fail("Multiple use_language tags are set in the root module: '{}' and '{}'. Only one is allowed.".format(set_tag_name, t.name)) + fail("Multiple use_language tags are set in the root module: '{}' and '{}'. Only one is allowed.".format(set_tag_name, module.name)) ------------------------------ In extensions.bzl <#855 (comment)> : > + +def _switched_rules_impl(ctx): + attrs = {} + for module in ctx.modules: + if not module.is_root: + continue + + is_tag_set = False + set_tag_name = "" + + for t in module.tags.use_languages: + if is_tag_set: + fail("Multiple use_language tags are set in the root module: '{}' and '{}'. Only one is allowed.".format(set_tag_name, t.name)) + + is_tag_set = True + set_tag_name = t.name ⬇️ Suggested change - set_tag_name = t.name + set_tag_name = module.name — Reply to this email directly, view it on GitHub <#855 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A7YUO6N7YVTI6C72DDHXLELY2HNJTAVCNFSM6AAAAAA5JM2EPWVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMYTSNRRG4YDANJQGE> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

keith · 2024-03-27T21:37:38Z

i stacked some commits on this over here #892

This appears to be a fairly recent development, and isn't yet 100% officially supported in the googleapis/googleapis repo, but it works: - googleapis/googleapis#855 - bazelbuild/bazel-central-registry#1699

* Enable bzlmod, update README, fix Swift build Rather than mess with the WORKSPACE rules, the shortest path to fixing `blaze build //swift:tests` appeared to be introducing MODULE.bazel. MODULE.bazel, a.k.a. bzlmod, appears to be the new hotness, so I'll also try updating the other builds to use it as well. - https://bazel.build/external/migration - https://bazel.build/external/overview#workspace-shortcomings - bazelbuild/bazel#18958 * Convert `bazel build //python/...` to MODULE.bazel * Convert `bazel build //csharp/...` to MODULE.bazel * Convert //go/..., gazelle to MODULE.bazel * Migrate protobuf, rules_proto to MODULE.bazel Also bumps to: - protobuf: 23.1 - rules_proto: 6.0.0-rc2 * Add rules_java, rules_jvm_external to MODULE.bazel rules_java was an implicit dependency before. Bumps: - rules_java: 7.4.0 => 7.5.0. - rules_jvm_external: 4.4.2 => 6.0 The rules_jvm_external docs describe using `maven.install` in MODULE.bazel as a replacement for `maven_install` in WORKSPACE: - https://github.com/bazelbuild/rules_jvm_external/blob/master/docs/bzlmod.md However, our current `maven_install` depends on the definition of IO_GRPC_GRPC_JAVA_ARTIFACTS from @io_grpc_grpc_java. I'll attempt that migration next. * MODULE.bazel: skylib, rules_proto_grpc, protobuf `bazel build //java/...` and `bazel test //java/...` both work with these changes. * Move grpc-java to MODULE.bazel, bump to 1.62.2 grpc-java only got MODULE.bazel support as of this most recent version: - grpc/grpc-java#11046 - bazelbuild/bazel-central-registry#1699 This grpc-java version bump exposed two issues that are fixed in this commit: 1. The //java/com/engflow/notificationqueue:client target dependency on @maven//:io_netty_netty_handler broke. The original WORKSPACE import of io_grpc_grpc_java imported this dependency directly by passing IO_GRPC_GRPC_JAVA_ARTIFACTS directly to `maven_install`. The `maven.install` call from grpc/grpc-java's MODULE.bazel sets `strict_visibility = True`. Somehow the other dependencies registered by grpc-java's MODULE.bazel are accessible to notificationqueue:client, but netty-handler isn't. The solution was to add the `io.netty:netty-handler:4.1.100.Final` artifact to the `maven.install` call in this project's MODULE.bazel. It doesn't seem an optimal solution, but it works for now. 2. grpc/grpc-java removed `io.grpc.stub.MetadataUtils.attachHeaders()` in grpc/grpc-java#10443. This caused notificationqueue:client to fail to compile, but that PR revealed the replacement for the deprecated `attachHeaders` call. This commit applies that replacement. * Move googleapis to MODULE.bazel This appears to be a fairly recent development, and isn't yet 100% officially supported in the googleapis/googleapis repo, but it works: - googleapis/googleapis#855 - bazelbuild/bazel-central-registry#1699 * Move rules_kotlin to MODULE.bazel, bump to v1.9.5 * Move rules_perl to MODULE.bazel, bump to 0.2.0 There's actually a 0.2.1 release, but it hasn't been pushed to https://registry.bazel.build/ yet. * Add rules_scala GitHub issue links rules_scala hasn't migrated to bzlmod yet, but discussion is underway. These links will help track its progress. * Migrate aspect_rules_ts to MODULE.bazel * Replace deps.bzl with go_deps This enables `bazel {build,test} //infra/...` to succeed using MODULE.bazel. See: - https://github.com/bazelbuild/rules_go/blob/master/docs/go/core/bzlmod.md#specifying-external-dependencies * Set test sizes to "small" as appropriate This eliminates warnings that these tests are sized too big, since the default is "medium". * Move http_{file,archive} calls to MODULE.bazel This moves the following http_file calls: - emacs - ubuntu_20.04_1.3GB and the following http_archive calls: - com_engflow_engflowapis - io_abseil_py * Update //platform rules, remove dotnet constraints The //dotnet/toolchain constraint was causing a Bazel error saying that the @@rules_dotnet//dontent/toolchain package didn't exist. Removing this constraint allowed remote execution to succeed anyway. * Update README, explain swift incompatibility Most of these changes are cosmetic, with the notable exception of the explantion behind the inability to build //swift remotely. Also added a `git` command to ignore python/requirements_lock.txt per: - https://stackoverflow.com/a/73720550

tyler-french force-pushed the bzlmod branch from 94b1def to 7be3f97 Compare September 27, 2023 13:54

tyler-french changed the title ~~make googleapis a Bzlmod module~~ feat(bazel): make googleapis a Bzlmod module Sep 27, 2023

tyler-french marked this pull request as ready for review November 7, 2023 15:26

fmeum reviewed Nov 8, 2023

View reviewed changes

nickbreen added a commit to nickbreen/bes that referenced this pull request Nov 11, 2023

brow beat googleapis into a module so it's sane to work with based on g…

a0a8fc4

…oogleapis/googleapis/pull/855

ah-edg mentioned this pull request Dec 12, 2023

Add a MODULE.bazel in order to support for bzlmod grpc/grpc-java#9559

Closed

MERIEMZ4 approved these changes Dec 20, 2023

View reviewed changes

tyler-french force-pushed the bzlmod branch from 7be3f97 to c0c4280 Compare December 30, 2023 21:14

feat(bazel): make googleapis a Bzlmod module

942c6c7

tyler-french force-pushed the bzlmod branch from c0c4280 to 942c6c7 Compare December 30, 2023 21:16

mmorel-35 mentioned this pull request Mar 25, 2024

cel-spec@0.14.0 bazelbuild/bazel-central-registry#1641

Merged

keith reviewed Mar 26, 2024

View reviewed changes

keith mentioned this pull request Mar 27, 2024

feat(bazel): make googleapis a Bzlmod module #892

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(bazel): make googleapis a Bzlmod module #855

feat(bazel): make googleapis a Bzlmod module #855

tyler-french commented Sep 27, 2023 •

edited

tyler-french commented Nov 7, 2023

fmeum Nov 8, 2023

fmeum Nov 8, 2023

tyler-french Dec 30, 2023

silicon-ninja commented Dec 5, 2023

alexander-fenster commented Dec 13, 2023

arnehormann commented Dec 14, 2023

SanjayVas commented Dec 14, 2023 •

edited

SanjayVas commented Dec 14, 2023

alexander-fenster commented Dec 15, 2023

SanjayVas commented Dec 15, 2023

ah-edg commented Mar 4, 2024

keith Mar 26, 2024

keith left a comment

keith Mar 26, 2024

keith Mar 26, 2024

keith Mar 26, 2024

keith Mar 26, 2024

SeeYaaG commented Mar 26, 2024 via email

keith commented Mar 27, 2024

	fail("Multiple use_language tags are set in the root module: '{}' and '{}'. Only one is allowed.".format(set_tag_name, t.name))
	fail("Multiple use_language tags are set in the root module: '{}' and '{}'. Only one is allowed.".format(set_tag_name, module.name))

feat(bazel): make googleapis a Bzlmod module #855

Are you sure you want to change the base?

feat(bazel): make googleapis a Bzlmod module #855

Conversation

tyler-french commented Sep 27, 2023 • edited

tyler-french commented Nov 7, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

silicon-ninja commented Dec 5, 2023

alexander-fenster commented Dec 13, 2023

arnehormann commented Dec 14, 2023

SanjayVas commented Dec 14, 2023 • edited

SanjayVas commented Dec 14, 2023

alexander-fenster commented Dec 15, 2023

SanjayVas commented Dec 15, 2023

ah-edg commented Mar 4, 2024

Choose a reason for hiding this comment

keith left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SeeYaaG commented Mar 26, 2024 via email

keith commented Mar 27, 2024

tyler-french commented Sep 27, 2023 •

edited

SanjayVas commented Dec 14, 2023 •

edited