Failure in 6.1 due to generated http_file with no urls set

[BalestraPatrick]

Hey folks! It's my first week using rules_jvm_external, so pardon my possible misunderstanding.

After upgrading to 6.1 from 6.0, I noticed this failure in our workspace:

ERROR: An error occurred during the fetch of repository 'com_google_guava_listenablefuture_jar_sources_9999_0_empty_to_avoid_conflict_with_guava':
   Traceback (most recent call last):
	File "/private/var/tmp/_bazel_patrick/1b6b63f4770b1664b25ee56aae5b009b/external/bazel_tools/tools/build_defs/repo/http.bzl", line 172, column 29, in _http_file_impl
		all_urls = _get_all_urls(ctx)
	File "/private/var/tmp/_bazel_patrick/1b6b63f4770b1664b25ee56aae5b009b/external/bazel_tools/tools/build_defs/repo/http.bzl", line 77, column 13, in _get_all_urls
		fail("At least one of url and urls must be provided")
Error in fail: At least one of url and urls must be provided
ERROR: /Users/patrick/src/WORKSPACE:22:25: fetching http_file rule //external:com_google_guava_listenablefuture_jar_sources_9999_0_empty_to_avoid_conflict_with_guava: Traceback (most recent call last):
	File "/private/var/tmp/_bazel_patrick/1b6b63f4770b1664b25ee56aae5b009b/external/bazel_tools/tools/build_defs/repo/http.bzl", line 172, column 29, in _http_file_impl
		all_urls = _get_all_urls(ctx)
	File "/private/var/tmp/_bazel_patrick/1b6b63f4770b1664b25ee56aae5b009b/external/bazel_tools/tools/build_defs/repo/http.bzl", line 77, column 13, in _get_all_urls
		fail("At least one of url and urls must be provided")
Error in fail: At least one of url and urls must be provided
ERROR: no such package '@@com_google_guava_listenablefuture_jar_sources_9999_0_empty_to_avoid_conflict_with_guava//file': At least one of url and urls must be provided

When bisecting, the offending commit is the following: 3ec0bbb

When I compare the contents of .../external/rules_jvm_external_deps/defs.bzl between the above commit and the parent, the following caught my eye.
Before:

http_file(
    name = "com_google_guava_listenablefuture_9999_0_empty_to_avoid_conflict_with_guava",
    sha256 = "b372a037d4230aa57fbeffdef30fd6123f9c0c2db85d0aced00c91b974f33f99",
    netrc = "../rules_jvm_external_deps/netrc",
    urls = ["https://repo1.maven.org/maven2/com/google/guava/listenablefuture/9999.0-empty-to-avoid-conflict-with-guava/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar"],
    downloaded_file_path = "v1/com/google/guava/listenablefuture/9999.0-empty-to-avoid-conflict-with-guava/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar",
)

After:

http_file(
    name = "com_google_guava_listenablefuture_9999_0_empty_to_avoid_conflict_with_guava",
    sha256 = "b372a037d4230aa57fbeffdef30fd6123f9c0c2db85d0aced00c91b974f33f99",
    netrc = "../rules_jvm_external_deps/netrc",
    urls = ["https://repo1.maven.org/maven2/com/google/guava/listenablefuture/9999.0-empty-to-avoid-conflict-with-guava/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar"],
    downloaded_file_path = "v1/com/google/guava/listenablefuture/9999.0-empty-to-avoid-conflict-with-guava/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar",
)
http_file(
    name = "com_google_guava_listenablefuture_jar_sources_9999_0_empty_to_avoid_conflict_with_guava",
    sha256 = "None",
    netrc = "../rules_jvm_external_deps/netrc",
    urls = [],
    downloaded_file_path = "None",
)

The failure is coming from the newly generated http_file named com_google_guava_listenablefuture_jar_sources_9999_0_empty_to_avoid_conflict_with_guava which has no urls set.

Am I doing something wrong, or do you have any idea what could the problem be?

[shs96c]

Do you have a reproducible test case for this? I use guava in my own projects, and haven't seen this problem. Can you also please confirm which resolver you're using? Is the default one or the new maven-based one?

[BalestraPatrick]

I'm using the default resolver.

I debugged this a bit more, and here is my main observation: at this line, raw_artifacts contains the following:

"com.google.guava:listenablefuture": {"shasums": {"jar": "b372a037d4230aa57fbeffdef30fd6123f9c0c2db85d0aced00c91b974f33f99", "sources": None}, "version": "9999.0-empty-to-avoid-conflict-with-guava"}

While if I comment out a bunch of our dependencies, the contents are:

"com.google.guava:listenablefuture": {"shasums": {"jar": "b372a037d4230aa57fbeffdef30fd6123f9c0c2db85d0aced00c91b974f33f99", "sources": "2bd5a29b9607c6b6fdaf4385b8b2f8fda27f183d72fac797f318fa59d2cf91f2"}, "version": "9999.0-empty-to-avoid-conflict-with-guava"}

I'm not sure how this can happen exactly. I tested the bump (and that linked commit specifically) in another project that we have and I didn't run into that error, so I assume it's some set of dependencies that's triggering the bug. We have thousands of dependencies so I haven't been able to track it down into a reproducible test case yet (and it's possible it'll have internal dependencies, so hard to share). Let me know if this helps a bit more at least!

[honnix]

I think it was this line resulted in generating a wrong http_file target.

https://repo1.maven.org/maven2/com/google/guava/listenablefuture/9999.0-empty-to-avoid-conflict-with-guava/ is by design not to have source jar, so ideally rules_jvm_external should better handle this type of cases than generating a wrong target.

[cheister]

I was seeing this error when calling bazel sync on our repo.

The reason this started showing up with 3ec0bbb is because fetch_sources = True was added to the rules_jvm_external_deps maven_install and that has 9999.0-empty-to-avoid-conflict-with-guava as a dependency.

#1148 will stop creating http_file declarations when the resolver was unable to download a file.