feat: compile source files at build time

[rickeylev]

This implements precompiling: performing Python source to byte code compilation at build time. This allows improved program startup time by allowing the byte code compilation step to be skipped at runtime.

Precompiling is disabled by default, for now. A subsequent release will enable it by default. This allows the necessary flags and attributes to become available so users can opt-out prior to it being enabled by default. Similarly, //python:features.bzl is introduced to allow feature detection.

This implementation is made to serve a variety of use cases, so there are several attributes and flags to control behavior. The main use cases being served are:

• Large mono-repos that need to incrementally enable/disable precompiling.
• Remote execution builds, where persistent workers aren't easily available.
• Environments where toolchains are custom defined instead of using the ones created by rules_python.

To that end, there are several attributes and flags to control behavior, and the toolchains allow customizing the tools used.

Fixes #1761

[rickeylev]

My apologies for the very large PR. The core logic in common_bazel.bzl#maybe_precompile isn't too big, but there's a lot of supporting code to define a toolchain, flags, attributes, precompiler tools, and etc.

I'll work on the CI failures. It looks like I'll need to add extra logic to gate these behind bazel 7 and pystar.

[aignas]

The overall structure makes sense. I gave it a once-over and I could not see anything blocking. I like the enum, features.bzl plumbing that I might be able to reuse them in pip.

[aignas]

Since this is a substantial amount of code, what do you think about adding a ruff pre-commit hook and ensuring that this passes it? I wonder what are your thoughts on having type hints everywhere? I guess we have to make them compatible with 3.8 upwards, so we might need the old style annotations, but ruff could be configured to ensure that we do 3.8-compatible type hints.

It's fine to introduce ruff as a separate PR though since it would be scope creep.

[rickeylev]

I like all the ideas :). That said:

• Yes, I think adding ruff is out of scope. +1 on adding linting as a check though
• Yes, I like type annotations. I added them. But, because the imports are deferred (deferring them makes non-worker executions twice as fast), I don't think it's going to play well with type checking. If/when we can move the worker impl to another file, then we can probably get a type checker running over it. But I don't want to block this PR on figuring that out. The type annotations are a bit anemic, too, because most of the objects being interacted with are "dynamicly defined" (argparse Namespace, json request/response object), or rely on side effects (e.g. compile writes to the input file paths). But it's a start.
• Compatibility with older Python versions is a good point. The non-worker and serial worker impls will probably work fine going back pretty far. I'm not sure about the asyncio based one, though. In any case, there are toggles that can help work around this situation, and it's initially disabled.

Depending on how much trouble CI gives me, I'll see if I have time to add some basic tests of the precompiler (a little can go a long way).

[dougthor42]

PR message and changelog say it's disabled by default?

[rickeylev]

Oops, thanks for catching that. I've sent a PR to fix the docs here. It is disabled by default for now. After the next release, we can consider enabling it by default