Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: enable multiprocessing on compile object level #1114

Draft
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

MatteoVoges
Copy link
Contributor

Proposed Changes

  • new flag --multiprocess-objects
  • splits compile_obj from targets and run them in parallel, to balance out heavy targets

Performance Measurements

  • for inventories with targets with equally distributed target to compile_obj ratio --> 1:1
  • for inventories with unbalanced ratios: performance gain up to 400 %

Problems with tests

The tests are failing because of the removal target. The order of execution matters in this target. I think, we should change this example because compile objects should not depend on each other neither different targets should.

@MatteoVoges MatteoVoges added enhancement enhancement to an existing feature performance issue regarding performance loss labels Jan 11, 2024
@MatteoVoges MatteoVoges self-assigned this Jan 11, 2024
@MatteoVoges
Copy link
Contributor Author

Came up in workgroup meeting: Compile objects can depend on each other. So we need a functionality to specify some dependencies among compile objects. First iteration --> boolean contains_dependencies in parameters.kapitan.vars, so that target will not split its compile objects.

@MatteoVoges MatteoVoges marked this pull request as draft January 26, 2024 19:29
@github-actions github-actions bot added the Stale label Mar 27, 2024
@github-actions github-actions bot removed the Stale label May 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement enhancement to an existing feature performance issue regarding performance loss
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant