Initial e2e test migration from in-tree to gcp cloud provider #683

seans3 · 2024-05-06T01:08:55Z

Initial migration of in-tree gce e2e tests to out-of-tree cloud-provider-gcp repository.
Creates e2e test infrastructure, validating e2e tests by running them from the command line or in prow CI.
Currently only migrated two network tests
- 1. A simple firewall test [sig-network] Firewall rule control plane should not expose well-known ports
- 1. A load-balancer test [sig-network] LoadBalancers should be able to change the type and ports of a UDP service [Slow]
More tests will be migrated in future PR's.
Migrate some gce provider code for the e2e tests.
Includes script tools/run-e2e-test.sh to run the e2e tests from the command line or in prow CI.
- Builds the e2e.test binary by running go test -c within the test/e2e directory.
- Runs the e2e test using the gce provider and the ginkgo tester.

Current Status

Sucessfully runs the e2e tests using the ginkgo tester with the following output:

Running Suite: Cloud Provider GCP End-to-End Tests - /home/sean/go/src/k8s.io/cloud-provider-gcp/_rundir/sean-test
==================================================================================================================
Random Seed: 1715403389 - will randomize all specs

Will run 2 of 2 specs
Running in parallel across 30 processes
••

Ran 2 of 2 Specs in 186.584 seconds
SUCCESS! -- 2 Passed | 0 Failed | 0 Pending | 0 Skipped

seans3 · 2024-05-06T01:09:39Z

/assign @BenTheElder
/assign @aojea

BenTheElder · 2024-05-07T21:26:49Z

First pass: this is generally looking good, we'll want to stand up a job to actually run this in CI (we can make that manually run at first) and sort out what to do re: Bazel.

tools/run-e2e-test.sh

aojea · 2024-05-13T10:32:13Z

First pass: this is generally looking good, we'll want to stand up a job to actually run this in CI (we can make that manually run at first) and sort out what to do re: Bazel.

yeah kubernetes/test-infra#32603

aojea · 2024-05-13T11:36:54Z

/test pull-cloud-provider-gcp-e2e

aojea · 2024-05-13T15:21:58Z

/home/prow/go/src/cloud-provider-gcp
cp: cannot stat '/root/go/bin/ginkgo': No such file or directory
+ EXIT_VALUE=1
+ set +o xtrace
Cleaning up after docker in docker.

seans3 · 2024-05-18T03:31:09Z

/test pull-cloud-provider-gcp-e2e

seans3 · 2024-05-18T05:00:14Z

/test pull-cloud-provider-gcp-e2e

seans3 · 2024-05-18T06:33:37Z

/test pull-cloud-provider-gcp-e2e

seans3 · 2024-05-18T20:21:24Z

/test pull-cloud-provider-gcp-e2e

seans3 · 2024-05-21T04:28:32Z

/test pull-cloud-provider-gcp-e2e

seans3 · 2024-05-21T05:34:24Z

/test pull-cloud-provider-gcp-e2e

aojea · 2024-05-21T15:18:37Z

it went through 👏

aojea · 2024-05-21T15:23:23Z

tools/run-e2e-test.sh

+if [[ ! -z "${PROJECT}" ]]; then
+    echo "Cleaning up boskos..."
+    cleanup_boskos
+    echo
+fi
+
+echo "Cleaning up run dir: ${RUN_DIR}"
+rm -rf ${RUN_DIR}


todo

trap cleanup INT TERM EXIT cleanup() { if [[ ! -z "${PROJECT}" ]]; then echo "Cleaning up boskos..." cleanup_boskos echo fi echo "Cleaning up run dir: ${RUN_DIR}" rm -rf ${RUN_DIR} }

seans3 · 2024-05-21T17:13:57Z

/test pull-cloud-provider-gcp-e2e

BenTheElder · 2024-05-21T20:18:03Z

tools/run-e2e-test.sh

+
+# Install bazel
+#
+BAZEL_VERSION="5.4.0"


TODO: setup .bazel-version in this repo (and/or remove bazel) and respect that here (it's like GOTOOLCHAIN)

BenTheElder · 2024-05-21T20:25:29Z

https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/cloud-provider-gcp/683/pull-cloud-provider-gcp-e2e/1792967087560331264

sweet!

BenTheElder

let's merge and iterate, I don't have any major blocking comments and we want to start letting people collaborate on expanding the migrated tests soon

/lgtm
/approve
/hold
[we'll either do some quick nits or /hold cancel shortly]

BenTheElder · 2024-05-21T20:24:51Z

test/e2e/suite_test.go

+
+func TestE2E(t *testing.T) {
+	gomega.RegisterFailHandler(framework.Fail)
+	repoDir := framework.TestContext.ReportDir


[non-blocking nit, could be a follow-up]

Suggested change

repoDir := framework.TestContext.ReportDir

reportDir := framework.TestContext.ReportDir

there are a lot of other places in testing where we get something like KUBE_ROOT or REPO_DIR that is the root of the source directory, so scanning this is a little confusing.

BenTheElder · 2024-05-21T20:27:15Z

cluster/gce/util.sh

@@ -253,6 +253,7 @@ function copy-to-staging() {
    fi
  fi

+  rm -f "${tar}.sha512"


we shouldn't need to do this? > below should already overwrite, >> would be append? what am I missing?

I ran into an issue when running the e2e tests locally. If the file already exists from a previous run, it is sometimes created without write permissions. So the following command fails: echo "${hash}" > "${tar}.sha512" . I think it had to do with my umask. I can remove this if you want.

ooh, we should defend against umask issues when manipulating tarballs etc.

I'm not sure what the best answer is here. in some other places where we're containerized we've explicitly set the permission bits.

TODO: follow-up on umask fun and these scripts

BenTheElder · 2024-05-21T20:27:52Z

test/boskos.sh

@@ -1,5 +1,24 @@
 #!/usr/bin/env bash

+# Copyright 2024 The Kubernetes Authors.


we should be able to leave this whole thing to the kubetest2 deployer? but let's leave that to a follow-up

I had problems leaving it up to kubetest2 to create the boskos project, because I needed to pass the project name on to the ginkgo-tester test args.

BenTheElder · 2024-05-21T20:28:34Z

tools/run-e2e-test.sh

+#  - Packages e2e test binary with "ginkgo" and "kubectl" binaries.
+#  - Starts a cluster with kubetest2, using the "gce" deployer.
+#    - Cluster started in either passed cluster (GCP_CLUSTER env var),
+#      or "boskos" cluster in prow CI.


TODO: revisit this versus letting kubetest2 boskos client handle this from go?

I don't think that works. We need to pass the GCE_PROJECT into the ginkgo-tester test args. But if we let kubetest2 create the boskos project, there doesn't seem to be a way to get the project name and then pass it into the test args. Please let me know if there is a way to do this.

I don't think that works. We need to pass the GCE_PROJECT into the ginkgo-tester test args. But if we let kubetest2 create the boskos project, there doesn't seem to be a way to get the project name and then pass it into the test args. Please let me know if there is a way to do this.

IMHO the deployer should either set it as the current project (in gcloud?) and we depend on that in the test as the default or we should modify kubetest2 to support passing through information like this.

We could maybe smuggle it in the kubeconfig.

something TODO later, this is super unweildy and we should find a better answer.

k8s-ci-robot · 2024-05-21T20:29:33Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: BenTheElder, seans3

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [BenTheElder]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

seans3 · 2024-05-21T21:09:36Z

/test pull-cloud-provider-gcp-e2e

seans3 · 2024-05-21T22:11:16Z

/triage accepted

seans3 · 2024-05-21T22:11:35Z

/hold cancel

BenTheElder · 2024-05-21T22:27:10Z

/lgtm

k8s-ci-robot requested review from aojea and cheftako May 6, 2024 01:09

k8s-ci-robot assigned aojea and BenTheElder May 6, 2024

seans3 force-pushed the e2e-test-migration branch from ee9bdaf to 93266ab Compare May 6, 2024 01:27

BenTheElder mentioned this pull request May 7, 2024

Add DevContainer Definitons For One-Click Dev Env Setup #659

Open

seans3 changed the title ~~[WIP] Initial e2e test migration from in-tree to gcp cloud provider~~ Initial e2e test migration from in-tree to gcp cloud provider May 11, 2024

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 11, 2024

aojea mentioned this pull request May 13, 2024

Remove gcp in-tree cloud provider and credential providers kubernetes/kubernetes#124519

Merged

aojea reviewed May 13, 2024

View reviewed changes

tools/run-e2e-test.sh Outdated Show resolved Hide resolved

aojea mentioned this pull request May 13, 2024

run e2e tests as presubmit for cloud-provider-gcp kubernetes/test-infra#32603

Merged

aojea reviewed May 21, 2024

View reviewed changes

seans3 force-pushed the e2e-test-migration branch from d46c6da to 55e3a96 Compare May 21, 2024 17:12

BenTheElder reviewed May 21, 2024

View reviewed changes

k8s-ci-robot added do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. lgtm "Looks good to me", indicates that a PR is ready to be merged. labels May 21, 2024

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 21, 2024

Initial e2e test migration from in-tree to gcp cloud provider

76d177f

seans3 force-pushed the e2e-test-migration branch from 55e3a96 to 76d177f Compare May 21, 2024 21:08

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 21, 2024

k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels May 21, 2024

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label May 21, 2024

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 21, 2024

k8s-ci-robot merged commit ab93615 into kubernetes:master May 21, 2024
8 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Initial e2e test migration from in-tree to gcp cloud provider #683

Initial e2e test migration from in-tree to gcp cloud provider #683

seans3 commented May 6, 2024 •

edited

seans3 commented May 6, 2024

BenTheElder commented May 7, 2024

aojea commented May 13, 2024

aojea commented May 13, 2024

aojea commented May 13, 2024

seans3 commented May 18, 2024

seans3 commented May 18, 2024

seans3 commented May 18, 2024

seans3 commented May 18, 2024

seans3 commented May 21, 2024

seans3 commented May 21, 2024

aojea commented May 21, 2024

aojea May 21, 2024

seans3 May 21, 2024

seans3 commented May 21, 2024

BenTheElder May 21, 2024

seans3 May 21, 2024

BenTheElder commented May 21, 2024

BenTheElder left a comment

BenTheElder May 21, 2024

seans3 May 21, 2024

BenTheElder May 21, 2024

seans3 May 21, 2024

BenTheElder May 21, 2024

BenTheElder May 21, 2024

seans3 May 21, 2024

BenTheElder May 21, 2024

seans3 May 21, 2024 •

edited

BenTheElder May 21, 2024

k8s-ci-robot commented May 21, 2024

seans3 commented May 21, 2024

seans3 commented May 21, 2024

seans3 commented May 21, 2024

BenTheElder commented May 21, 2024

	repoDir := framework.TestContext.ReportDir
	reportDir := framework.TestContext.ReportDir

		@@ -1,5 +1,24 @@
		#!/usr/bin/env bash

		# Copyright 2024 The Kubernetes Authors.

Initial e2e test migration from in-tree to gcp cloud provider #683

Initial e2e test migration from in-tree to gcp cloud provider #683

Conversation

seans3 commented May 6, 2024 • edited

Current Status

seans3 commented May 6, 2024

BenTheElder commented May 7, 2024

aojea commented May 13, 2024

aojea commented May 13, 2024

aojea commented May 13, 2024

seans3 commented May 18, 2024

seans3 commented May 18, 2024

seans3 commented May 18, 2024

seans3 commented May 18, 2024

seans3 commented May 21, 2024

seans3 commented May 21, 2024

aojea commented May 21, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

seans3 commented May 21, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BenTheElder commented May 21, 2024

BenTheElder left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

seans3 May 21, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

k8s-ci-robot commented May 21, 2024

seans3 commented May 21, 2024

seans3 commented May 21, 2024

seans3 commented May 21, 2024

BenTheElder commented May 21, 2024

seans3 commented May 6, 2024 •

edited

seans3 May 21, 2024 •

edited