Simplify input type Helm #977 #979

gburiola · 2023-03-22T18:44:06Z

Fixes issue #977

test

this target file supports both formats of input_type: helm:

with a local chart
with URL and without input_path

As we can see in the output below, the helm binary is called in two different ways for each mode.

launching helm with arguments: ['template', '--include-crds', '--skip-tests', '--output-dir', '/tmp/tmp6318ts8t', '--generate-name', '/home/lburiola/git/github/kapitan/tests/test_resources/charts/acs-engine-autoscaler']

launching helm with arguments: ['template', '--include-crds', '--skip-tests', '--repo', 'https://charts.jetstack.io', '--version', '1.1.0', '--output-dir', '/tmp/tmp0iqle40w', '--generate-name', 'cert-manager']

FULL OUTPUT

$ ../../bin/kapitan compile -t acs-engine-autoscaler -v
2023-03-22 18:37:30,374 git.cmd      DEBUG    Popen(['git', 'version'], cwd=/home/lburiola/git/github/kapitan/tests/test_resources, universal_newlines=False, shell=None, istream=None)
2023-03-22 18:37:30,384 git.cmd      DEBUG    Popen(['git', 'version'], cwd=/home/lburiola/git/github/kapitan/tests/test_resources, universal_newlines=False, shell=None, istream=None)
2023-03-22 18:37:30,452 kapitan.cli  DEBUG    Running with args: Namespace(cache=False, cache_paths=[], compose_node_name=False, embed_refs=False, fetch=False, force=False, force_fetch=False, func=<function trigger_compile at 0x7feda5336b00>, ignore_version_check=False, indent=2, inventory_path='./inventory', jinja2_filters='lib/jinja2_filters.py', labels=[], name='compile', output_path='.', parallelism=4, prune=False, quiet=False, refs_path='./refs', reveal=False, schemas_path='./schemas', search_paths=['.', 'lib'], subparser_name='compile', targets=['acs-engine-autoscaler'], use_go_jsonnet=False, validate=False, verbose=True, yaml_dump_null_as_empty=False, yaml_multiline_string_style='double-quotes')
2023-03-22 18:37:30,543 kapitan.resources DEBUG    Using reclass inventory config defaults
2023-03-22 18:37:30,578 kapitan.utils DEBUG    hashable_lru_cache: True not serialiseable, using generic lru_cache instead
2023-03-22 18:37:30,614 kapitan.targets DEBUG    validating target name matches the name of yml file acs-engine-autoscaler
2023-03-22 18:37:30,614 kapitan.targets DEBUG    load_target_inventory: found valid kapitan target acs-engine-autoscaler
2023-03-22 18:37:30,614 kapitan.targets INFO     Rendered inventory (0.07s)
2023-03-22 18:37:30,873 git.cmd      DEBUG    Popen(['git', 'version'], cwd=/home/lburiola/git/github/kapitan/tests/test_resources, universal_newlines=False, shell=None, istream=None)
2023-03-22 18:37:30,879 git.cmd      DEBUG    Popen(['git', 'version'], cwd=/home/lburiola/git/github/kapitan/tests/test_resources, universal_newlines=False, shell=None, istream=None)
2023-03-22 18:37:31,271 kapitan.inputs.base DEBUG    Compiling /home/lburiola/git/github/kapitan/tests/test_resources/charts/acs-engine-autoscaler
2023-03-22 18:37:31,272 kapitan.helm_cli DEBUG    launching helm with arguments: ['template', '--include-crds', '--skip-tests', '--output-dir', '/tmp/tmp6318ts8t', '--generate-name', '/home/lburiola/git/github/kapitan/tests/test_resources/charts/acs-engine-autoscaler']
2023-03-22 18:37:31,363 kapitan.inputs.base DEBUG    Wrote /tmp/tmpom1otnvu.kapitan/compiled/acs-engine-autoscaler/./acs-engine-autoscaler/templates/secrets.yaml
2023-03-22 18:37:31,364 kapitan.inputs.helm DEBUG    Wrote file /tmp/tmp6318ts8t/acs-engine-autoscaler/templates/secrets.yaml to /tmp/tmpom1otnvu.kapitan/compiled/acs-engine-autoscaler/./acs-engine-autoscaler/templates/secrets.yaml
2023-03-22 18:37:31,439 kapitan.inputs.base DEBUG    Wrote /tmp/tmpom1otnvu.kapitan/compiled/acs-engine-autoscaler/./acs-engine-autoscaler/templates/deployment.yaml
2023-03-22 18:37:31,440 kapitan.inputs.helm DEBUG    Wrote file /tmp/tmp6318ts8t/acs-engine-autoscaler/templates/deployment.yaml to /tmp/tmpom1otnvu.kapitan/compiled/acs-engine-autoscaler/./acs-engine-autoscaler/templates/deployment.yaml
2023-03-22 18:37:31,440 kapitan.inputs.base DEBUG    Compiling 
2023-03-22 18:37:31,440 kapitan.helm_cli DEBUG    launching helm with arguments: ['template', '--include-crds', '--skip-tests', '--repo', 'https://charts.jetstack.io', '--version', '1.1.0', '--output-dir', '/tmp/tmp0iqle40w', '--generate-name', 'cert-manager']
2023-03-22 18:37:33,127 kapitan.inputs.base DEBUG    Wrote /tmp/tmpom1otnvu.kapitan/compiled/acs-engine-autoscaler/./cert-manager/templates/cainjector-serviceaccount.yaml
2023-03-22 18:37:33,127 kapitan.inputs.helm DEBUG    Wrote file /tmp/tmp0iqle40w/cert-manager/templates/cainjector-serviceaccount.yaml to /tmp/tmpom1otnvu.kapitan/compiled/acs-engine-autoscaler/./cert-manager/templates/cainjector-serviceaccount.yaml
2023-03-22 18:37:33,130 kapitan.inputs.base DEBUG    Wrote /tmp/tmpom1otnvu.kapitan/compiled/acs-engine-autoscaler/./cert-manager/templates/serviceaccount.yaml
2023-03-22 18:37:33,130 kapitan.inputs.helm DEBUG    Wrote file /tmp/tmp0iqle40w/cert-manager/templates/serviceaccount.yaml to /tmp/tmpom1otnvu.kapitan/compiled/acs-engine-autoscaler/./cert-manager/templates/serviceaccount.yaml
2023-03-22 18:37:33,133 kapitan.inputs.base DEBUG    Wrote /tmp/tmpom1otnvu.kapitan/compiled/acs-engine-autoscaler/./cert-manager/templates/webhook-serviceaccount.yaml
2023-03-22 18:37:33,133 kapitan.inputs.helm DEBUG    Wrote file /tmp/tmp0iqle40w/cert-manager/templates/webhook-serviceaccount.yaml to /tmp/tmpom1otnvu.kapitan/compiled/acs-engine-autoscaler/./cert-manager/templates/webhook-serviceaccount.yaml
2023-03-22 18:37:33,152 kapitan.inputs.base DEBUG    Wrote /tmp/tmpom1otnvu.kapitan/compiled/acs-engine-autoscaler/./cert-manager/templates/cainjector-rbac.yaml
2023-03-22 18:37:33,152 kapitan.inputs.helm DEBUG    Wrote file /tmp/tmp0iqle40w/cert-manager/templates/cainjector-rbac.yaml to /tmp/tmpom1otnvu.kapitan/compiled/acs-engine-autoscaler/./cert-manager/templates/cainjector-rbac.yaml
2023-03-22 18:37:33,227 kapitan.inputs.base DEBUG    Wrote /tmp/tmpom1otnvu.kapitan/compiled/acs-engine-autoscaler/./cert-manager/templates/rbac.yaml
2023-03-22 18:37:33,227 kapitan.inputs.helm DEBUG    Wrote file /tmp/tmp0iqle40w/cert-manager/templates/rbac.yaml to /tmp/tmpom1otnvu.kapitan/compiled/acs-engine-autoscaler/./cert-manager/templates/rbac.yaml
2023-03-22 18:37:33,235 kapitan.inputs.base DEBUG    Wrote /tmp/tmpom1otnvu.kapitan/compiled/acs-engine-autoscaler/./cert-manager/templates/webhook-rbac.yaml
2023-03-22 18:37:33,235 kapitan.inputs.helm DEBUG    Wrote file /tmp/tmp0iqle40w/cert-manager/templates/webhook-rbac.yaml to /tmp/tmpom1otnvu.kapitan/compiled/acs-engine-autoscaler/./cert-manager/templates/webhook-rbac.yaml
2023-03-22 18:37:33,239 kapitan.inputs.base DEBUG    Wrote /tmp/tmpom1otnvu.kapitan/compiled/acs-engine-autoscaler/./cert-manager/templates/service.yaml
2023-03-22 18:37:33,239 kapitan.inputs.helm DEBUG    Wrote file /tmp/tmp0iqle40w/cert-manager/templates/service.yaml to /tmp/tmpom1otnvu.kapitan/compiled/acs-engine-autoscaler/./cert-manager/templates/service.yaml
2023-03-22 18:37:33,243 kapitan.inputs.base DEBUG    Wrote /tmp/tmpom1otnvu.kapitan/compiled/acs-engine-autoscaler/./cert-manager/templates/webhook-service.yaml
2023-03-22 18:37:33,243 kapitan.inputs.helm DEBUG    Wrote file /tmp/tmp0iqle40w/cert-manager/templates/webhook-service.yaml to /tmp/tmpom1otnvu.kapitan/compiled/acs-engine-autoscaler/./cert-manager/templates/webhook-service.yaml
2023-03-22 18:37:33,250 kapitan.inputs.base DEBUG    Wrote /tmp/tmpom1otnvu.kapitan/compiled/acs-engine-autoscaler/./cert-manager/templates/cainjector-deployment.yaml
2023-03-22 18:37:33,250 kapitan.inputs.helm DEBUG    Wrote file /tmp/tmp0iqle40w/cert-manager/templates/cainjector-deployment.yaml to /tmp/tmpom1otnvu.kapitan/compiled/acs-engine-autoscaler/./cert-manager/templates/cainjector-deployment.yaml
2023-03-22 18:37:33,259 kapitan.inputs.base DEBUG    Wrote /tmp/tmpom1otnvu.kapitan/compiled/acs-engine-autoscaler/./cert-manager/templates/deployment.yaml
2023-03-22 18:37:33,259 kapitan.inputs.helm DEBUG    Wrote file /tmp/tmp0iqle40w/cert-manager/templates/deployment.yaml to /tmp/tmpom1otnvu.kapitan/compiled/acs-engine-autoscaler/./cert-manager/templates/deployment.yaml
2023-03-22 18:37:33,270 kapitan.inputs.base DEBUG    Wrote /tmp/tmpom1otnvu.kapitan/compiled/acs-engine-autoscaler/./cert-manager/templates/webhook-deployment.yaml
2023-03-22 18:37:33,270 kapitan.inputs.helm DEBUG    Wrote file /tmp/tmp0iqle40w/cert-manager/templates/webhook-deployment.yaml to /tmp/tmpom1otnvu.kapitan/compiled/acs-engine-autoscaler/./cert-manager/templates/webhook-deployment.yaml
2023-03-22 18:37:33,276 kapitan.inputs.base DEBUG    Wrote /tmp/tmpom1otnvu.kapitan/compiled/acs-engine-autoscaler/./cert-manager/templates/webhook-mutating-webhook.yaml
2023-03-22 18:37:33,276 kapitan.inputs.helm DEBUG    Wrote file /tmp/tmp0iqle40w/cert-manager/templates/webhook-mutating-webhook.yaml to /tmp/tmpom1otnvu.kapitan/compiled/acs-engine-autoscaler/./cert-manager/templates/webhook-mutating-webhook.yaml
2023-03-22 18:37:33,283 kapitan.inputs.base DEBUG    Wrote /tmp/tmpom1otnvu.kapitan/compiled/acs-engine-autoscaler/./cert-manager/templates/webhook-validating-webhook.yaml
2023-03-22 18:37:33,283 kapitan.inputs.helm DEBUG    Wrote file /tmp/tmp0iqle40w/cert-manager/templates/webhook-validating-webhook.yaml to /tmp/tmpom1otnvu.kapitan/compiled/acs-engine-autoscaler/./cert-manager/templates/webhook-validating-webhook.yaml
2023-03-22 18:37:33,283 kapitan.targets INFO     Compiled acs-engine-autoscaler (2.01s)
2023-03-22 18:37:33,291 kapitan.targets DEBUG    Copied /tmp/tmpom1otnvu.kapitan/compiled/acs-engine-autoscaler into ./compiled/acs-engine-autoscaler
2023-03-22 18:37:33,581 kapitan.targets DEBUG    Removed /tmp/tmpom1otnvu.kapitan

Moep90 · 2023-03-22T19:21:13Z

Add Tests
Update Website, if needed

gburiola · 2023-03-22T18:48:39Z

docs/pages/input_types/helm.md

@@ -55,7 +74,7 @@ Special flags:

 See the [helm doc](https://helm.sh/docs/helm/helm_template/) for further detail.

-#### Example
+## Example


fix precommit error

markdownlint-fix.........................................................Failed - hook id: markdownlint-fix - exit code: 1 docs/pages/input_types/helm.md:77 MD001/heading-increment/header-increment Heading levels should only increment by one level at a time [Expected: h2; Actual: h4]

kapitan/inputs/base.py

gburiola · 2023-03-22T20:04:47Z

Add Tests

added on https://github.com/kapicorp/kapitan/blob/a56d551/tests/test_helm_input.py#L48-L57

gburiola · 2023-03-23T08:15:08Z

Update Website, if needed

@Moep90 I don't understand this. The PR already updates the documentation. Do you think it's not sufficient or are you talking about updating something else?

uberspot

Looks safe and a nice simplification for using helm 👍

kapitan/inputs/helm.py

kapitan/inputs/base.py

kapitan/targets.py

Moep90 · 2023-03-23T09:41:30Z

Update Website, if needed

@Moep90 I don't understand this. The PR already updates the documentation. Do you think it's not sufficient or are you talking about updating something else?

NVM, my bad.

ramaro · 2023-03-24T09:58:30Z

Hey @gburiola this is awesome! One question - how does this deal with caching? E.g. will subsequent executions of kapitan compile use a cached helm repo on disk or will it download the repo everytime?

ramaro

Generally looks good to me, just requesting a small comment

gburiola · 2023-03-25T22:41:19Z

Hey @gburiola this is awesome! One question - how does this deal with caching? E.g. will subsequent executions of kapitan compile use a cached helm repo on disk or will it download the repo everytime?

It doesn't cache anything. It fetches the chart on the fly using helm cli with parameter --repo <CHART_URL>.

Bear in mind that the the current way using dependencies also makes an HTTP call to the depency URL every time kapitan compile is called.

MatteoVoges · 2023-03-26T13:06:12Z

Bear in mind that the the current way using dependencies also makes an HTTP call to the depency URL every time kapitan compile is called.

Are you sure about that? I thought that we have the flags --fetch / --force-fetch and --cache to specify which behavior we want for our dependencies, so that they initially get fetched, get fetched everytime or even get cached...
Note that we also have a force_fetch property in the inventory as well, that enables fetching on user-selected resources.
Could you check that this behavior integrates with your changes?

gburiola · 2023-03-26T21:22:20Z

Are you sure about that?

Yes. --fetch always dowloads external file, unpacks and doesn't replace local copy on disk if it already exists.
--force-fetch does the same as above, but overrites existing copy on disk.

--cache only refers to kapitan dependencies management.

I'm not using dependency mgmt in this case. I'm using the helm cli --repos URL directly.

I've updated the docs on 614807e

- input_type: helm and input_paths or - input_type: helm and helm_params.chart_name

ramaro · 2023-03-31T09:30:54Z

Bear in mind that the the current way using dependencies also makes an HTTP call to the depency URL every time kapitan compile is called.

OK, worth making it very obvious in the documentation of the behaviour of this alternative way if we still think it's worth it! Is there no way helm will do some sort of caching at all? If not, this feels suboptimal to me.

ademariag · 2023-03-31T09:49:36Z

@gburiola @ramaro helm does seem to support caching? https://helm.sh/docs/helm/helm/

We also need to understand what happens if multiple compile of the same helm compete to download the charts at the same time.

gburiola requested review from uberspot, ramaro and ademariag March 22, 2023 18:44

gburiola force-pushed the issue_977 branch from 5e29b29 to cadc9ef Compare March 22, 2023 18:47

gburiola commented Mar 22, 2023

View reviewed changes

gburiola force-pushed the issue_977 branch from cadc9ef to a56d551 Compare March 22, 2023 20:02

uberspot approved these changes Mar 23, 2023

View reviewed changes

MatteoVoges suggested changes Mar 23, 2023

View reviewed changes

kapitan/inputs/helm.py Show resolved Hide resolved

kapitan/inputs/base.py Show resolved Hide resolved

kapitan/targets.py Outdated Show resolved Hide resolved

MatteoVoges approved these changes Mar 23, 2023

View reviewed changes

gburiola mentioned this pull request Mar 23, 2023

Simplify kapitan/inputs/base.py #980

Open

ademariag added the enhancement enhancement to an existing feature label Mar 24, 2023

ramaro requested changes Mar 24, 2023

View reviewed changes

gburiola requested a review from ramaro March 25, 2023 22:48

gburiola added 4 commits March 27, 2023 10:57

Simplify input type Helm #977

508ecb1

Update jsonschema to deal with all variations of PR977

723f042

- input_type: helm and input_paths or - input_type: helm and helm_params.chart_name

Add comment

6bdaf91

Update doc on helm input type

ce4e399

gburiola force-pushed the issue_977 branch from 614807e to ce4e399 Compare March 27, 2023 09:57

Merge branch 'master' into issue_977

6a060e2

github-actions bot added the Stale label Nov 9, 2023

github-actions bot removed the Stale label May 11, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Simplify input type Helm #977 #979

Simplify input type Helm #977 #979

gburiola commented Mar 22, 2023 •

edited

Moep90 commented Mar 22, 2023 •

edited

gburiola Mar 22, 2023

gburiola commented Mar 22, 2023

gburiola commented Mar 23, 2023

uberspot left a comment

Moep90 commented Mar 23, 2023

ramaro commented Mar 24, 2023

ramaro left a comment

gburiola commented Mar 25, 2023

MatteoVoges commented Mar 26, 2023

gburiola commented Mar 26, 2023

ramaro commented Mar 31, 2023

ademariag commented Mar 31, 2023 •

edited

Simplify input type Helm #977 #979

Are you sure you want to change the base?

Simplify input type Helm #977 #979

Conversation

gburiola commented Mar 22, 2023 • edited

test

Moep90 commented Mar 22, 2023 • edited

gburiola Mar 22, 2023

Choose a reason for hiding this comment

gburiola commented Mar 22, 2023

gburiola commented Mar 23, 2023

uberspot left a comment

Choose a reason for hiding this comment

Moep90 commented Mar 23, 2023

ramaro commented Mar 24, 2023

ramaro left a comment

Choose a reason for hiding this comment

gburiola commented Mar 25, 2023

MatteoVoges commented Mar 26, 2023

gburiola commented Mar 26, 2023

ramaro commented Mar 31, 2023

ademariag commented Mar 31, 2023 • edited

gburiola commented Mar 22, 2023 •

edited

Moep90 commented Mar 22, 2023 •

edited

ademariag commented Mar 31, 2023 •

edited