Declarative test setup #9283

npolshakova · 2024-03-28T14:53:22Z

The declarative environment builder tool can setup Kubernetes test cluster(s) with IstioOperators, Helm Charts, and any Applications required for testing. This is based on the declarative setup tool used in GME.

How do I get started?

A make target is provided to run the setup. A config file needs to be provided via CONFIG variable.

To test the simple gloo gateway setup:

CONFIG=test/setup/example_configs/gloo-gateway-setup.yaml make setup-declarative-env

This can also be run directly with:

go run ./test/setup setup -c test/setup/example_configs/gloo-gateway-setup.yaml

Make sure the images and helm chart are present before running the go command directly.

What happens if I already have a (partial) setup?

Running the declarative setup will not automatically fail if you have different resources already setup. This is slightly different from the regression tests which require you to set SKIP_TEARDOWN or SKIP_INSTALL.

For example:

If you have a kind cluster already running, the declarative setup will use the existing kind cluster
If you already have a namespace, the declarative setup will use the existing namespace, add any labels defined in the config file and continue.
If you have gloo already installed, the declarative setup will run a helm upgrade -i and upgrade the existing installation.

How do I write my own config?

Configuration is typically provided via yaml file. Usage is usually coupled with make targets
since a lot of default environment variables are defined there.

The configuration file is broken into a list of clusters, each with a distinct set of capabilities:

clusters:
    - name: management-cluster
      management: true
      kindConfig:
        ...
      istioOperators:
        ...
      charts:
        ...
      namespaces:
        ...
      apps:
        ...
      images:
        ...

kindConfig: The configuration for the KinD cluster
istioOperators: Installation mechanism for Istio uses istio operators. This is a list of istio operator configs to install into the cluster. Uses the istioctl set by the ISTIO_VERSION environment variable, or the installed istioctl binary if not set for installation.
charts: helm charts to install into the cluster.
- name: name of the chart.
- namespace: namespace to install the chart into
- path: if true, the chart is installed from the local filesystem. Otherwise, it is installed from the configured helm repo.
- values: values to pass to the chart.
namespaces: namespaces to create in the cluster. Labels can be specified on the namespace.
apps: applications to install into the cluster. Follows the k8s resources apis and includes deployments, serviceaccounts, services, configmaps, etc.
images: images to load into the cluster. If an image override is defined on the helm chart, the setup command will also load that into the kind cluster so images only need to be defined in one place.

github-actions · 2024-03-28T15:14:15Z

Visit the preview URL for this PR (updated for commit f784298):

https://gloo-edge--pr9283-npolshak-declarative-7t317rco.web.app

_{(expires Tue, 16 Apr 2024 16:29:47 GMT)}

_{🔥 via Firebase Hosting GitHub Action 🌎}

_{Sign: 77c2b86e287749579b7ff9cadb81e099042ef677}

sam-heilbron · 2024-04-02T19:01:46Z

Makefile

@@ -921,6 +921,18 @@ build-test-chart: ## Build the Helm chart and place it in the _test directory
 	helm package --destination $(TEST_ASSET_DIR) $(HELM_DIR)
 	helm repo index $(TEST_ASSET_DIR)

+# Sets up environment for running e2e tests
+.PHONY: setup-declarative-env


I had imagined that the benefit for a declarative environment, is that the config we supply/declare is what defines what should be run. What if instead of having env variables that gate the steps, we just say the processor of the config says "if docker is to true, build the images", "if chart needs to be built, build the chart"...etc? That way we isolate complexity to this declarative setup?

Yeah, I the separate docker/helm/declarative-setup is how GME handles it. I think it's possible to have the declarative setup also build the images/package the helm chart, but it's a little tricky because I think we'd want it to be identical to what we run in CI so we'd want to somehow reuse the make targets. I don't think it makes sense to hard code all the images/helm chart paths in the declarative setup code to do this since then that could drift.

sam-heilbron · 2024-04-02T19:03:09Z

test/setup/cmd/setup.go

+	}
+
+	var istioctlBinary string
+	if os.Getenv(defaults.IstioctlVersionEnv) != "" {


How do we separate what is defined in the config and what is defined in the command line? I would have imagined that we could instead have the config support a definition of the istio version and we could parse it from there. Is there a reason we wouldn't want to do that?

Personally, I think having Istio set in the env makes it easier to switch between versions with just export ISTIOCTL_VERSION=1.18.2. But maybe it doesn't make as much sense on the edge side if we're not switching between Istio versions that often?

I like having often changed values (cluster names, Istio versions, image versions, etc.) set as env values, and then values that are not changed as often (test images, etc.) hard coded.

sam-heilbron · 2024-04-02T19:04:27Z

test/setup/cmd/setup.go

+				kubeConfig string
+			)
+
+			if cluster.KindConfig != nil {


More and more I kind of think we should split environment setup into 2 separate pieces of config:

My cluster(s) definition

My infrastructure setup (helm charts, additional services)

I think this will make it easier to support non-kind environments, and also allow devs to easily re-use cluster that may exist.

The declarative setup does already re-use clusters that exists. If the KindConfig is not set, it'll just reuse the cluster with the same name from your kubeconfig. Eventually I think we'd want to add support loading images built from source to remote registries (AWS, GCP, etc ...), but if you have your kubeconfig pointing to a real cluster, this setup should still work (if you've manually uploaded the images)!

Having separate config makes it a little harder to read config for multi cluster setups (so that's why we have it in one file in GME: https://github.com/solo-io/gloo-mesh-enterprise/blob/main/test/gloo-core/gloo-core-multi-cluster.yaml#L8). Maybe on the edge side it makes more sense to separate the files, but I think that would still make it tricky to test multi cluster fed setups.

sam-heilbron · 2024-04-02T19:05:48Z

test/setup/helpers/helpers.go

+}
+
+// Merge maps, merge priority is left to right (i.e. right maps will overwrite left maps)
+func MergeValueMaps(maps ...map[string]interface{}) map[string]interface{} {


I think these utilities could be defined outside of our test package, and in a top level utils package, what do you think?

Yep agreed! The merge functionality is pretty useful!

sam-heilbron · 2024-04-02T19:06:56Z

test/setup/helpers/timer.go

+var components []string
+var lock sync.Mutex
+
+func ListTimers() {


In our codegen library, we have https://github.com/solo-io/solo-kit/blob/main/pkg/code-generator/metrics/metrics.go. Perhaps we could re-use it?

Or we can move these to the metrics util? Do you have an example that uses the solo-kit metrics? All of the examples seem to initialize the metrics aggregator in init() and then call it only once per operation like here? Versus we want to time it per cluster:

timerFn := helpers.TimerFunc((fmt.Sprintf("[%s] %s helm installation", cluster.GetName(), name))) defer timerFn()

test/setup/defaults/defaults.go

test/setup/config/config.go

sam-heilbron · 2024-04-02T19:10:17Z

test/setup/helm/helm.go

+	Logger     *zap.SugaredLogger
+}
+
+func (o Opts) InstallChart(ctx context.Context, chart *types.Chart, cluster *kubernetes.Cluster) error {


A lot of this mirrors our CLI installer (which uses helm under the hood): https://github.com/solo-io/gloo/blob/main/projects/gloo/cli/pkg/cmd/install/installer.go. I think we should try to consolidate these as much as possible, and ideally the logic for the installer should live in the CLI, and we can just invoke a top-level go function from here.

I don't think so, this is installing any chart defined in the config file. It should be more flexible that the cli installer. If I wanted to install GME or Otel along side Edge, this helm installer would work for that.

The user facing glooctl CLI tool depends on all of these imports:

"github.com/solo-io/gloo/install/helm/gloo/generate" "github.com/solo-io/gloo/pkg/cliutil" "github.com/solo-io/gloo/pkg/cliutil/helm" "github.com/solo-io/gloo/pkg/version" "github.com/solo-io/gloo/projects/gloo/cli/pkg/cmd/options" "github.com/solo-io/gloo/projects/gloo/cli/pkg/constants" "github.com/solo-io/gloo/projects/gloo/cli/pkg/helpers"

This file is agnostic to any gloo dependencies or constants and is only used for test env setup. It's not meant to be a released cli tool.

sam-heilbron · 2024-04-02T19:11:17Z

test/setup/istio/istioctl.go

+	"github.com/solo-io/go-utils/contextutils"
+)
+
+func Install(


Should this be defined in the Istio commands of our CLI: https://github.com/solo-io/gloo/blob/main/projects/gloo/cli/pkg/cmd/istio/root.go. Again, we can just call the function we need, instead of relying on the CLI artifact, but it might allow us to easily re-use code

I'd say no, mostly because the istioctl cli tool should be responsible for installing Istio. Even on the gloo-mesh side, meshctl doesn't install Istio. I think it would give glooctl too much responsibility if we owned the Istio installation and it would be be more work to maintain.

Having this as a non user facing testing tool gives us flexibility, but doesn't require we maintain this in a real released artifact (so we don't have to test across istio versions, etc.)

…setup

npolshakova · 2024-05-16T17:26:51Z

Closing in favor of new framework

npolshakova added 2 commits March 28, 2024 10:11

only add declarative setup

7f1ea61

dedup helm values from declarative setup

4aec0e2

npolshakova requested a review from a team as a code owner March 28, 2024 14:53

github-actions bot added the keep pr updated signals bulldozer to keep pr up to date with base branch label Mar 28, 2024

changelog

6770aff

This was referenced Mar 28, 2024

[DNM] Snapshot e2e tests for gateway and classic api #9284

Closed

[DNM] Gloo gateway e2e test framework #9238

Closed

npolshakova and others added 7 commits March 28, 2024 12:43

revert mergo version

1ffaf1c

move changelog

754b868

regen

c1ead87

Merge refs/heads/main into npolshak/declarative-setup

8c1f0d7

Merge refs/heads/main into npolshak/declarative-setup

f17b956

Adding changelog file to new location

0793f9d

Deleting changelog file from old location

a557d94

sam-heilbron reviewed Apr 2, 2024

View reviewed changes

npolshakova and others added 14 commits April 2, 2024 16:11

Merge remote-tracking branch 'origin/main' into npolshak/declarative-…

30589af

…setup

Merge refs/heads/main into npolshak/declarative-setup

590c640

pr feedback

ef49794

Merge refs/heads/main into npolshak/declarative-setup

88ce86c

Merge refs/heads/main into npolshak/declarative-setup

174e83e

Adding changelog file to new location

b2b123c

Deleting changelog file from old location

f8959bc

Merge refs/heads/main into npolshak/declarative-setup

1b996a3

Merge refs/heads/main into npolshak/declarative-setup

92fa0ae

Merge refs/heads/main into npolshak/declarative-setup

6dfabef

Merge refs/heads/main into npolshak/declarative-setup

0b62590

Merge refs/heads/main into npolshak/declarative-setup

97fa5bb

Adding changelog file to new location

a860944

Deleting changelog file from old location

9ba2cfb

soloio-bulldozer bot and others added 12 commits April 9, 2024 14:02

Merge refs/heads/main into npolshak/declarative-setup

d0ce0c1

Merge refs/heads/main into npolshak/declarative-setup

f784298

Adding changelog file to new location

af0885e

Deleting changelog file from old location

41acb20

Adding changelog file to new location

7343ecc

Deleting changelog file from old location

61270a7

Adding changelog file to new location

286f7a3

Deleting changelog file from old location

0737b0d

Adding changelog file to new location

ee0db27

Deleting changelog file from old location

c9c540b

Adding changelog file to new location

8b43782

Deleting changelog file from old location

a38e60e

sam-heilbron added work in progress signals bulldozer to keep pr open (don't auto-merge) and removed keep pr updated signals bulldozer to keep pr up to date with base branch labels Apr 19, 2024

changelog-bot added 10 commits April 25, 2024 15:59

Adding changelog file to new location

8b5fee2

Deleting changelog file from old location

bd8645c

Adding changelog file to new location

11cc186

Deleting changelog file from old location

ef3e849

Adding changelog file to new location

8538828

Deleting changelog file from old location

2d8eac7

Adding changelog file to new location

db8e393

Deleting changelog file from old location

a466f3e

Adding changelog file to new location

f561fda

Deleting changelog file from old location

cdc2c95

npolshakova closed this May 16, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Declarative test setup #9283

Declarative test setup #9283

npolshakova commented Mar 28, 2024 •

edited

github-actions bot commented Mar 28, 2024 •

edited

sam-heilbron Apr 2, 2024

npolshakova Apr 2, 2024

sam-heilbron Apr 2, 2024

npolshakova Apr 2, 2024

sam-heilbron Apr 2, 2024

npolshakova Apr 2, 2024

npolshakova Apr 2, 2024

sam-heilbron Apr 2, 2024

npolshakova Apr 2, 2024

sam-heilbron Apr 2, 2024

npolshakova Apr 2, 2024

sam-heilbron Apr 2, 2024

npolshakova Apr 2, 2024

sam-heilbron Apr 2, 2024

npolshakova Apr 2, 2024

npolshakova commented May 16, 2024

Declarative test setup #9283

Declarative test setup #9283

Conversation

npolshakova commented Mar 28, 2024 • edited

How do I get started?

What happens if I already have a (partial) setup?

How do I write my own config?

github-actions bot commented Mar 28, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

npolshakova commented May 16, 2024

npolshakova commented Mar 28, 2024 •

edited

github-actions bot commented Mar 28, 2024 •

edited