Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

convert/python: parse requirements.txt, skip if package exist on APKINDEX, rework error printing #704

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

Dentrax
Copy link
Contributor

@Dentrax Dentrax commented Sep 20, 2023

As we discussed at #621, this PR is initial attempt to support parsing requirements.txt that given by argument as a either a FILE or URL.

Motivation

One of the limitations of the convert python subcommand is that we can not provide arbitrary requirements.txt. So this makes melange tightly coupled to pypi to search and get dependencies of given package.

By this limitation, it makes impossilbe to generate melange manifests if package is not hosted on pypi.

This commands also does not respect already-created packages that hosted on APKINDEX: https://packages.wolfi.dev. Since it do recursive calls to find all of the dependencies, it makes the process even longer and eventually we have duplicated (already-crafted) packages.

Implementation

To tackle with this problems to enrich UX even further;

  • a new EXPERIMENTAL flag is introduced: --use-existing! It basically gathers all the Python packages from Wolfi APK to not re-generate again to speed up the process
  • convert python now supports FILE and URL:
    1. go run . convert python requirements.txt --use-existing
    1. go run . convert python https://raw.githubusercontent.com/kubeflow/pipelines/master/backend/src/apiserver/visualization/requirements.txt --use-existing - took 25m on my machine - follow up proposal: convert {apkbuild, gem, python}: make concurrent generation pipeline  #705
  • BEHAVIORAL CHANGE: To not get interrupted during the package generation, once the generate process is completely finished, aggregated errors will be printed out in batch
  • BEHAVIORAL CHANGE: YAML ident set to 2 (from 4) to match with the current manifests

Signed-off-by: Furkan Türkal <furkan.turkal@chainguard.dev>
Co-authored-by: Batuhan <batuhan.apaydin@chainguard.dev>
@Dentrax Dentrax marked this pull request as ready for review September 20, 2023 21:20
@Dentrax Dentrax marked this pull request as draft September 21, 2023 07:27
@Dentrax Dentrax marked this pull request as ready for review September 26, 2023 18:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant