docs(examples): Add example of using a `src` dir and separate `tests` dir with gazelle #1842

dougthor42 · 2024-04-11T18:52:56Z

Add an example of using a src dir and separate tests dir and
having gazelle generate targets correctly so that tests can be run.

Fixes #1775.

dougthor42 · 2024-04-16T16:56:05Z

I can't seem to figure out why builds are failing. Any tips?

If I run bazel test //... from the new example directory, things run just fine.

$ cd examples/bzlmod_python_src_dir_with_separate_tests_dir/
$ bazel test //...
INFO: Analyzed 10 targets (4 packages loaded, 24 targets configured).
INFO: Found 7 targets and 3 test targets...
INFO: Elapsed time: 20.303s, Critical Path: 18.58s
INFO: 13 processes: 7 internal, 2 local, 4 processwrapper-sandbox.
INFO: Build completed successfully, 13 total actions
//:gazelle_python_manifest.test                                          PASSED in 0.2s
//:requirements_test                                                     PASSED in 16.2s
//tests:test_my_python_module                                            PASSED in 0.6s

Executed 3 out of 3 tests: 3 tests pass.
There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are.

If I run from the git repo root, I get the same failures seen in CI (unknown repo 'pypi').

$ cd -
/c/dev/rules_python
$ bazel test //...
ERROR: Skipping '//...': error loading package under directory '': error loading package 'examples/bzlmod_python_src_dir_with_separate_tests_dir': Unable to find package for @@[unknown repo 'pypi' requested from @@]//:requirements.bzl: The repository '@@[unknown repo 'pypi' requested from @@]' could not be resolved: No repository visible as '@pypi' from main repository.
ERROR: error loading package under directory '': error loading package 'examples/bzlmod_python_src_dir_with_separate_tests_dir': Unable to find package for @@[unknown repo 'pypi' requested from @@]//:requirements.bzl: The repository '@@[unknown repo 'pypi' requested from @@]' could not be resolved: No repository visible as '@pypi' from main repository.
INFO: Elapsed time: 25.549s
INFO: 0 processes.
ERROR: Build did NOT complete successfully
ERROR: Couldn't start the build. Unable to run tests

From what I can tell, the only differences between this example and bzlmod or bzlmod_build_file_generation is that this example:

Doesn't have a WORKSPACE file
Uses pypi for the hub name instead of pip.

But I tried making both of those edits locally and still got the same error.

Note: I'm ignoring the MacOS and Windows CI failures for now.

aignas · 2024-04-17T14:17:52Z

Please run the pre-commit hooks that will update .bazelignore, which should fix the errors seen in the tests running from the root.

Edit: or the .bazelrc, can't remember which one in this case.

dougthor42 · 2024-04-17T17:52:35Z

Thanks. Turns out it's both .bazelrc and .bazelignore.

I also noticed that the pre-commit hook update-deleted-packages doesn't appear to work correctly, so I've opened #1858 with details.

aignas

Sorry, I thought I had submitted these review comments, but it seems that I had not done this. Happy to move this forward.

I think we may want to add a bazel-in-bazel integration test somewhere here.

aignas · 2024-04-18T01:02:20Z

examples/bzlmod_python_src_dir_with_separate_tests_dir/MODULE.bazel

@@ -0,0 +1,69 @@
+# Define metadata about this repository/project.


Having an empty WORKSPACE file may fix some of the issues you saw with the pre-commit hook, I think.

Hmm... still no dice. Same issue.

Given that CI is passing, I'm not too concerned about the pre-commit check. My assumption is that it's something specific to my computer/setup rather than specific to this branch, as pre-commit run --all-files also fails on the latest main commit 3730803.

aignas · 2024-04-18T01:05:05Z

examples/bzlmod_python_src_dir_with_separate_tests_dir/MODULE.bazel

+# In the old WORKSPACE file, this would be 4 items:
+#   1.  `load` the http_archive rule
+#   2.  run the http_archive rule, grapping rules_python from github
+#   3.  load the py_repositories target from rules_python
+#   4.  execute py_respositories()


We could just link to the relevant example (e.g. pip_parse or pip_parse_vendored).

Relative file references added, but I kept the original list because I find it easier to follow if I don't have to jump around to various other examples. LMK if you feel strongly about removing the list.

aignas · 2024-04-18T01:06:04Z

examples/bzlmod_python_src_dir_with_separate_tests_dir/MODULE.bazel

+)
+
+# Install rules_python, which allows us to define how bazel should work with python files.
+# See https://github.com/bazelbuild/rules_python/blob/main/examples/bzlmod/MODULE.bazel


Why not use relative file references? At least in vim users could use gf to go to the file instantly and I hope other editors might support something similar.

Suggested change

# See https://github.com/bazelbuild/rules_python/blob/main/examples/bzlmod/MODULE.bazel

# See ../bzlmod/MODULE.bazel

+1, updated here and for gazelle/README.md below.

aignas · 2024-04-18T01:06:44Z

examples/bzlmod_python_src_dir_with_separate_tests_dir/MODULE.bazel

+    # Use the bazel downloader for pulling pypi packages.
+    experimental_index_url = "https://pypi.org/simple",


aignas · 2024-04-18T01:08:09Z

examples/bzlmod_python_src_dir_with_separate_tests_dir/MODULE.bazel

+    requirements_lock = "//:requirements.lock",
+)
+
+# Same as WORKSPACE install_deps() - actually install the python deps.


This is not technically correct. Here we just expose the repo to be used by the module and the install happens lazily by bazel. I do think the intention is good - to draw parallels between WORKSPACE and MODULE.bazel, but this example might be something that people may read without any WORKSPACE knowledge.

Ah, good to know! Updated, PTAL and check for correctness.

dougthor42

I thought I had submitted these review comments

I do the same things all the time. I really wish GitHub had some "you have an unsubmitted review pending" notification or something.

But anyway, thanks for the comments! And don't worry about the delay! You've already been super responsive elsewhere, and adding a new example is a low priority.

I think we may want to add a bazel-in-bazel integration test somewhere here.

Can you elaborate on this?

dougthor42 · 2024-05-18T20:53:39Z

examples/bzlmod_python_src_dir_with_separate_tests_dir/MODULE.bazel

@@ -0,0 +1,69 @@
+# Define metadata about this repository/project.


Hmm... still no dice. Same issue.

Given that CI is passing, I'm not too concerned about the pre-commit check. My assumption is that it's something specific to my computer/setup rather than specific to this branch, as pre-commit run --all-files also fails on the latest main commit 3730803.

dougthor42 · 2024-05-18T20:54:05Z

examples/bzlmod_python_src_dir_with_separate_tests_dir/MODULE.bazel

+)
+
+# Install rules_python, which allows us to define how bazel should work with python files.
+# See https://github.com/bazelbuild/rules_python/blob/main/examples/bzlmod/MODULE.bazel


+1, updated here and for gazelle/README.md below.

dougthor42 · 2024-05-18T20:57:59Z

examples/bzlmod_python_src_dir_with_separate_tests_dir/MODULE.bazel

+# In the old WORKSPACE file, this would be 4 items:
+#   1.  `load` the http_archive rule
+#   2.  run the http_archive rule, grapping rules_python from github
+#   3.  load the py_repositories target from rules_python
+#   4.  execute py_respositories()


Relative file references added, but I kept the original list because I find it easier to follow if I don't have to jump around to various other examples. LMK if you feel strongly about removing the list.

dougthor42 · 2024-05-18T21:07:19Z

examples/bzlmod_python_src_dir_with_separate_tests_dir/MODULE.bazel

+    requirements_lock = "//:requirements.lock",
+)
+
+# Same as WORKSPACE install_deps() - actually install the python deps.


Ah, good to know! Updated, PTAL and check for correctness.

aignas · 2024-05-30T02:35:02Z

I've been thinking about this PR and how do we organise it together with other gazelle docs that we have. We are running short on CI executors so just adding this example into the CI is not so straight forward. For running bazel-in-bazel integration tests we use https://github.com/bazel-contrib/rules_bazel_integration_test. However, this makes our tests slow to execute as they become yet another thing that needs to be executed when we do bazel test //....

What is more, if we merge this PR, does it trump the other example on build_file_generation? What is the expected example people are to follow? Given that the Python community is recommending this structure, does it mean that we should follow their recommendations as well? bazel has different limitations and benefits and I am wondering about the tradeoffs here and in other places.

Given that this example may be more complex than the existing gazelle example I would be +1 on replacing the existing example with this or merging them in some way.

Sorry that this is dragging so long, but I am a bit lost with how we teach our users to use gazelle and it seems that more and more people are starting to use it, so your PR is a great addition to the knowledge base, but I am concerned that if we are not running CI on it, it will become stale and will cause new issues about "stale example".

Am I overthinking it here? What are your thoughts? We can have a call to chat this because I do think that having an async messaging is slowing us down. Shall we get in touch on the bazel slack?

dougthor42 · 2024-05-30T05:02:20Z

I don't have a slack account, sorry. I could do Google Meet if that works for you.

No, you're not overthinking this at all - you have great concerns and know a bunch more about the project than I do. For example, I didn't know about bazel-in-bazel (though I should have suspected, haha) or about how there are a limited number of CI executors.

does it trump the other example? What is the expected example people are to follow?

I would say that there should be a single "main" example to follow, ideally with a limited feature set consisting of the most common use cases (how many projects need whl_mods from the start?)

IMO the feature set for the main example should be:

python toolchain (hermetic python)
pip.parse, obviously
gazelle python manifest
gazelle and some common directives

We don't need the requirement locking (compile_pip_requirements). At least IME, large projects already have that via tools like poetry or uv.

should Bazel follow the python community recommendation for project structure?

I'd say that depends on the goals of rules_python.

Do people want rules_python to be used in more and more open-source projects (eg: get numpy to switch from meson to bazel; get flask to use it; have it be the default for greenfield projects)? If so, then the "main" example should follow community recommendation for project structure, and modifying as necessary to address the Basel-specific directory structure needs.

Or is rules_python targeting more closed-source systems, like Google and Uber and Dropbox, etc.? If so, then the "main" example should probably follow the a structure that more closely reflects what is done in corporate environs (which would be harder to figure out).

I would be +1 on replacing the existing example with this or merging them in some way

Same. When we make a decision I can work on that.

I am concerned that if we are not running CI on it, it will become stale

+100 on that. Incorrect docs can be worse than no docs, haha.

dougthor42 added 3 commits April 11, 2024 11:50

Add python directory structure and code. No bazel yet.

53f8a26

Add bazel and gazelle config

472850e

bazel run: requiremens.update, manifest, gazelle

a5dad2e

dougthor42 marked this pull request as ready for review April 11, 2024 18:53

dougthor42 requested a review from rickeylev as a code owner April 11, 2024 18:53

dougthor42 added 2 commits April 11, 2024 14:06

Better comments in README.md

60a6482

Update changelog

df44f4b

dougthor42 added 2 commits April 17, 2024 09:18

run ./tools/update_deleted_pacakges.sh

fb5a495

Add new example to .bazelignore

0dc8f34

dougthor42 mentioned this pull request Apr 17, 2024

pre-commit hook for updating deleted packages removes items added by the script that it calls #1858

Open

aignas reviewed May 13, 2024

View reviewed changes

dougthor42 added 4 commits May 18, 2024 13:56

Use relative file references

4215fb7

Add links to other WORKSPACE examples

ec41cfe

Update comment about 'use_repo'

7303a70

Spelling, grammar, and capitalization of comments

4fac351

dougthor42 commented May 18, 2024

View reviewed changes

aignas self-assigned this May 30, 2024

aignas added the gazelle Gazelle plugin related issues label Jun 4, 2024

aignas mentioned this pull request Jun 4, 2024

Gazelle example that mixes internal and third party deps #1386

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs(examples): Add example of using a `src` dir and separate `tests` dir with gazelle #1842

docs(examples): Add example of using a `src` dir and separate `tests` dir with gazelle #1842

dougthor42 commented Apr 11, 2024

dougthor42 commented Apr 16, 2024

aignas commented Apr 17, 2024 •

edited

dougthor42 commented Apr 17, 2024

aignas left a comment

aignas Apr 18, 2024

dougthor42 May 18, 2024

aignas Apr 18, 2024

dougthor42 May 18, 2024

aignas Apr 18, 2024

dougthor42 May 18, 2024

aignas Apr 18, 2024

aignas Apr 18, 2024

dougthor42 May 18, 2024

dougthor42 left a comment

dougthor42 May 18, 2024

dougthor42 May 18, 2024

dougthor42 May 18, 2024

dougthor42 May 18, 2024

aignas commented May 30, 2024

dougthor42 commented May 30, 2024

		@@ -0,0 +1,69 @@
		# Define metadata about this repository/project.

	# See https://github.com/bazelbuild/rules_python/blob/main/examples/bzlmod/MODULE.bazel
	# See ../bzlmod/MODULE.bazel

		# Use the bazel downloader for pulling pypi packages.
		experimental_index_url = "https://pypi.org/simple",

docs(examples): Add example of using a src dir and separate tests dir with gazelle #1842

Are you sure you want to change the base?

docs(examples): Add example of using a src dir and separate tests dir with gazelle #1842

Conversation

dougthor42 commented Apr 11, 2024

dougthor42 commented Apr 16, 2024

aignas commented Apr 17, 2024 • edited

dougthor42 commented Apr 17, 2024

aignas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dougthor42 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aignas commented May 30, 2024

dougthor42 commented May 30, 2024

docs(examples): Add example of using a `src` dir and separate `tests` dir with gazelle #1842

docs(examples): Add example of using a `src` dir and separate `tests` dir with gazelle #1842

aignas commented Apr 17, 2024 •

edited