Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Option to avoid download python exe from https://github.com/indygreg/python-build-standalone #154

Open
da-liii opened this issue Apr 5, 2023 · 4 comments
Labels
feature request Denotes a request for a new scie-pants feature.

Comments

@da-liii
Copy link

da-liii commented Apr 5, 2023

For example, in Github Action, we'd prefer to install the python needed by Pants itself by actions/setup-python.

@jsirois
Copy link
Contributor

jsirois commented Apr 13, 2023

@darcy-shen that's a no-go. The whole point of scie-pants is that it ships with a Python interpreter you, as the consumer, are unaware of. It is absolutely not intended to work with an external interpreter. Do you actually care about the interpreter it uses, or just that it doesn't download the interpreter dynamically? If its the latter, that can be accomodated. Right now we just ship a "thin" scie that does not include the Python interpreter - to save space - but downloads it as needed instead. We can also ship "fat" scies that include the interpreter. That said, downloads will still happen (of wheels) when a new Pants version is installed.

I guess what I really need here is more explanation of why you want to avoid https://github.com/indygreg/python-build-standalone - due to the technology, due to the runtime use of the network, or some other reason?

@jsirois jsirois self-assigned this Apr 13, 2023
@da-liii
Copy link
Author

da-liii commented Apr 15, 2023

due to the technology

I have found a better way to install Python, see pantsbuild/pants#17962

due to the runtime use of the network

Yes, it can be unstable for me. The network is not guaranteed to be stable.

@da-liii
Copy link
Author

da-liii commented Apr 15, 2023

We can also ship "fat" scies that include the interpreter.

That's a great idea.

@jsirois jsirois added the feature request Denotes a request for a new scie-pants feature. label Apr 26, 2023
jsirois added a commit to jsirois/scie-pants that referenced this issue May 26, 2023
Instead of using raw `scie-jump` and a hand-crafted JSON lift manifest,
use `science` and a TOML lift mabifest to build `scie-pants`.

The `science` tool and its more friendly interface are the present and
future of end-user scie construction. We convert here and gain a few
things:
+ A manifest format we can comment.
+ Abstraction over platforms and PBS details.
+ Support for embedded build provenanace information for more
  comprehensive black box debugability.
+ A clean path to publishing both "thin" and "fat" binaries and solving
  pantsbuild#154.
jsirois added a commit that referenced this issue May 26, 2023
Instead of using raw `scie-jump` and a hand-crafted JSON lift manifest,
use `science` and a TOML lift manifest to build `scie-pants`.

The `science` tool and its more friendly interface are the present and
future of end-user scie construction. We convert here and gain a few
things:
+ A manifest format we can comment.
+ Abstraction over platforms and PBS details.
+ Support for embedded build provenanace information for more
  comprehensive black box debugability.
+ A clean path to publishing both "thin" and "fat" binaries, solving
  #154.
@jsirois jsirois added in progress Denotes an assigned issue is being actively worked on. and removed in progress Denotes an assigned issue is being actively worked on. labels May 27, 2023
@jsirois
Copy link
Contributor

jsirois commented May 27, 2023

With scie-pants now converted to science for building its scie, creating fat scies for release is as easy as adding --invert-lazy cpython39 --app-name scie-pants-fat to the science command here:

execute(
Command::new(&skinny_scie_tools.science)
.args([
"lift",
"--include-provenance",
"--file",
&format!(
"scie-pants.bin={scie_pants_exe}",
scie_pants_exe = path_as_str(scie_pants_exe)?
),
"--file",
&format!(
"tools.pex={tools_pex}",
tools_pex = path_as_str(tools_pex_file)?
),
"build",
"--dest-dir",
path_as_str(&scie_pants_package_dir)?,
"--use-platform-suffix",
"--hash",
"sha256",
path_as_str(&scie_pants_manifest)?,
])
.current_dir(&build_context.workspace_root),
)?;

The question that needs to be considered is whether to make the fat scie fully fat (Also include --invert-lazy cpython38). Although this question is probably easy to answer "no" to since 3.8 is only needed for Pants < 2.5 which is quite old at this point, it gets harder to answer when Pants switches to using CPython 3.11. At that point there will be very recent versions of Pants that need 3.9; so the fat variant including both cpython39 and cpython311 distributions (but still probably not cpython38) may make more sense.

I'll leave all this for others to kvetch over and implement.

@jsirois jsirois removed their assignment May 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request Denotes a request for a new scie-pants feature.
Projects
None yet
Development

No branches or pull requests

2 participants