Facilitating Python Usage: Proposal for s5cmd Wheels

[jcfr]

Firstly, huge thanks for your work on the s5cmd tool and for contributing to the community with such a valuable tool. 🙏

Goal

We're keen on enhancing the accessibility of s5cmd within Python projects. Our aim is to streamline its integration by enabling installation via a simple pip command:

pip install s5cmd

Upon installing the wheel package, we envision an automatic creation of a convenient executable script¹, allowing seamless invocation of the underlying s5cmd executable.

Proposed Next Steps

To foster cohesion and prevent fragmentation, we propose the creation of a dedicated project repository, tentatively named peak/s5cmd-python-distributions. This draws inspiration from analogous projects like scikit-build/cmake-python-distributions² and scikit-build/ninja-python-distributions³.

We're committed to aiding in the setup and maintenance of this infrastructure. With this in place, we can align our project, ImagingDataCommons/idc-index, to become a pure Python wheel, contingent upon the s5cmd project.

The implementation approach mirrors our current practices within our package:

• idc_index/s5cmd/init.py: A Python module with a function used as entry point for the launcher script generation.
• CMakeLists.txt: CMake configuration for downloading the go executable using URL and SHA specified in s5cmdUrls.cmake.

Furthermore, we propose the implementation of a GitHub workflow, automatically triggered upon each new release of the corresponding go executable. We plan to adapt the bump function from cmake-python-distributions⁴ for this purpose.

Alternative Project Repository

We are also considering ImagingDataCommons/s5cmd-python-distributions as an alternative home for this project.

Background Details

• @fedorov is leading the https://github.com/ImagingDataCommons initiative

• I (@jcfr) along with @henryiii, @thewtex and many others are developing and maintaining the cmake², ninja³ and scikit-build-core⁵ Python projects.

Footnotes

1. https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#creating-executable-scripts ↩

2. scikit-build/cmake-python-distributions ↩ ↩²

3. scikit-build/ninja-python-distributions ↩ ↩²

4. https://github.com/scikit-build/cmake-python-distributions/blob/30a96f66e3aeea2be72d517d415f2fb19542485f/noxfile.py#L138-L149 ↩

5. https://scikit-build-core.readthedocs.io/ ↩

[jcfr]

After running pip install s5cmd, the following will be possible from Python without having to manually download/extract/install:

import shutil
import subprocess


s5cmdPath = shutil.which("s5cmd")

if s5cmdPath is None:
    raise FileNotFoundError(
        "s5cmd executable not found. Please install s5cmd from https://github.com/peak/s5cmd#installation"
    )

print(f"Found s5cmd executable: {s5cmdPath}")

subprocess.check_call([s5cmdPath, "--help"])

[jcfr]

To follow-up on this, a dedicated project has been created at https://github.com/jcfr/s5cmd-python-distributions

Wheels have been published at https://pypi.org/project/s5cmd/

The following platforms are supported by the binary wheels:

OS	Arch
Windows	64-bit 32-bit ARM64
Linux Intel	manylinux 64-bit musllinux 64-bit manylinux 32-bit musllinux 32-bit
Linux ARM	manylinux AArch64 musllinux AArch64
Linux PowerPC	manylinux ppc64le musllinux ppc64le
macOS 10.10+	Intel
macOS 11+	Apple Silicon