WIP: Automatic docker builds for cluster mode and basic tests layout,…

… local/remote configuration (#60)

* bump go to 1.21.3, update flake.lock

* remove vendor, update CI

* temp disable bench, move lint/test/e2e to nix

* automate docker build, wip

* wrap all in one go test

* update README.md for tests layout

.dockerignore

@@ -0,0 +1 @@
+.git/

Dockerfile

@@ -1,14 +1,17 @@
 # Example Dockerfile for k8s run
-FROM golang:1.20 as build
-ARG BUILD_ROOT
+# Builds all the tests in some directory that must have go.mod
+# All tests are built as separate binaries with name "module.test"
+FROM golang:1.21 as build
+ARG TESTS_ROOT
 
 WORKDIR /go/src
-COPY . .
+COPY . /tests
 
-RUN CGO_ENABLED=0 cd ${BUILD_ROOT} && go test -c -o wasp_test
+RUN CGO_ENABLED=0 cd /tests && go test -c ./...
 
 FROM debian
-ARG BUILD_ROOT
+ARG TESTS_ROOT
 
-COPY --from=build ${BUILD_ROOT} /
+COPY --from=build /tests .
+RUN apt-get update && apt-get install -y ca-certificates
 ENTRYPOINT /bin/bash

TUTORIAL.md → HOW_IT_WORKS.md

@@ -1,18 +1,5 @@
-# Tutorial
-## Setup
-Let's create our first load test
-```bash
-make start
-```
-Insert `GRAFANA_TOKEN` created in previous command
-```bash
-export LOKI_URL=http://localhost:3030/loki/api/v1/push
-export GRAFANA_URL=http://localhost:3000
-export GRAFANA_TOKEN=...
-export DATA_SOURCE_NAME=Loki
-export DASHBOARD_FOLDER=LoadTests
-make dashboard
-```
+# How it works
+
 ## Overview
 General idea is to be able to compose load tests programmatically by combining different `Generators`
 
@@ -126,48 +113,49 @@ sequenceDiagram
     Grafana->>Devs: Notify devs (Dashboard URL/Alert groups)
     end
 ```
-## Examples
 
-## RPS test
-- [test](https://github.com/smartcontractkit/wasp/blob/master/examples/simple_rps/main.go#L9)
-- [gun](https://github.com/smartcontractkit/wasp/blob/master/examples/simple_rps/gun.go#L23)
-```
-cd examples/simple_rps
-go run .
-```
-Open [dashboard](http://localhost:3000/d/wasp/wasp-load-generator?orgId=1&refresh=5s&var-go_test_name=generator_healthcheck&var-gen_name=generator_healthcheck&var-branch=generator_healthcheck&var-commit=generator_healthcheck&from=now-5m&to=now)
-
-`Gun` must implement this [interface](https://github.com/smartcontractkit/wasp/blob/master/wasp.go#L39)
-
-## VUs test
-- [test](https://github.com/smartcontractkit/wasp/blob/master/examples/simple_vu/main.go#L10)
-- [vu](https://github.com/smartcontractkit/wasp/blob/master/examples/simple_vu/vu.go#L19)
-```
-cd examples/simple_vu
-go run .
-```
-Open [dashboard](http://localhost:3000/d/wasp/wasp-load-generator?orgId=1&refresh=5s&var-go_test_name=generator_healthcheck&var-gen_name=generator_healthcheck&var-branch=generator_healthcheck&var-commit=generator_healthcheck&from=now-5m&to=now)
+Example cluster component diagram:
+```mermaid
+---
+title: Workload execution. P - Profile, G - Generator, VU - VirtualUser
+---
+flowchart TB
+    ClusterProfile-- generate k8s manifests/deploy/await jobs completion -->P1
+    ClusterProfile-->PN
+    ClusterProfile-- check NFRs -->Grafana
+    subgraph Pod1
+    P1-->P1-G1
+    P1-->P1-GN
+    P1-G1-->P1-G1-VU1
+    P1-G1-->P1-G1-VUN
+    P1-GN-->P1-GN-VU1
+    P1-GN--->P1-GN-VUN
+
+    P1-G1-VU1-->P1-Batch
+    P1-G1-VUN-->P1-Batch
+    P1-GN-VU1-->P1-Batch
+    P1-GN-VUN-->P1-Batch
+    end
+    subgraph PodN
+    PN-->PN-G1
+    PN-->PN-GN
+    PN-G1-->PN-G1-VU1
+    PN-G1-->PN-G1-VUN
+    PN-GN-->PN-GN-VU1
+    PN-GN--->PN-GN-VUN
+
+    PN-G1-VU1-->PN-Batch
+    PN-G1-VUN-->PN-Batch
+    PN-GN-VU1-->PN-Batch
+    PN-GN-VUN-->PN-Batch
 
-`VirtualUser` must implement this [interface](https://github.com/smartcontractkit/wasp/blob/master/wasp.go#L47)
+    end
+    P1-Batch-->Loki
+    PN-Batch-->Loki
 
-## Profile test (group multiple generators in parallel)
-- [test](https://github.com/smartcontractkit/wasp/blob/master/examples/profiles/main_test.go#L11)
-- [gun](https://github.com/smartcontractkit/wasp/blob/master/examples/profiles/gun.go#L23)
-- [vu](https://github.com/smartcontractkit/wasp/blob/master/examples/profiles/vu.go#L19)
-```
-cd examples/profiles
-go test -v -count 1 .
-```
-Open [dashboard](http://localhost:3000/d/wasp/wasp-load-generator?orgId=1&refresh=5s&var-go_test_name=TestProfile&var-gen_name=second%20API&var-gen_name=third%20API&var-gen_name=first%20API&var-branch=generator_healthcheck&var-commit=generator_healthcheck&from=now-5m&to=now)
+    Loki-->Grafana
 
-## Scenario with simulating users behaviour
-- [test](https://github.com/smartcontractkit/wasp/blob/master/examples/go_test/main_test.go#L15)
-- [gun](https://github.com/smartcontractkit/wasp/blob/master/examples/go_test/gun.go#L23)
-```
-cd examples/scenario
-go test -v -count 1 .
 ```
-Open [dashboard](http://localhost:3000/d/wasp/wasp-load-generator?orgId=1&refresh=5s&var-go_test_name=TestScenario&var-gen_name=Two%20sequential%20calls%20scenario&var-branch=generator_healthcheck&var-commit=generator_healthcheck&from=now-5m&to=now)
 
 ## Defining NFRs and checking alerts
 You can define different non-functional requirements groups
@@ -201,7 +189,7 @@ Check [dashboard](http://localhost:3000/d/wasp/wasp-load-generator?orgId=1&refre
 ## Cluster test with k8s
 `Warning`: we don't have Loki + Grafana k8s setup yet, if you have them in your `k8s` set up you can run this test
 
-Cluster mode [overview](CLUSTER.md)
+Cluster mode [overview](examples/CLUSTER.md)
 
 You may also need to set your `LOKI_TOKEN` env var, depends on your authorization
 

README.md

@@ -22,6 +22,7 @@ Protocol-agnostic load testing library for `Go`
 - Easy to reuse any custom client `Go` code
 - Easy to grasp
 - Have slim codebase (500-1k loc)
+- No test harness or CLI, easy to integrate and run with plain `go test`
 - Have predictable performance footprint when tested with protocol mocks
 - Easy to create synthetic or user-based scenarios
 - Scalable in `k8s` without complicated configuration or vendored UI interfaces
@@ -33,6 +34,7 @@ We are using `nix` for deps, see [installation](https://nixos.org/manual/nix/sta
 nix develop
 ```
 
+
 ## Run example tests with Grafana + Loki
 ```bash
 make start
@@ -61,8 +63,11 @@ Remove environment:
 make stop
 ```
 
-## Tutorial
-Check [tutorial](./TUTORIAL.md) for more examples and project overview
+## Test Layout and examples
+Check [examples](examples/README.md) to understand what is the easiest way to structure your tests, run them both locally and remotely, at scale, inside `k8s`
+
+## How it works
+Check this [doc](./HOW_IT_WORKS.md) for more examples and project overview
 
 ## Run pyroscope test
 ```

build_test_image.sh

@@ -0,0 +1,36 @@
+#!/bin/bash
+
+set -o pipefail
+set +e
+
+# Check if required parameters are provided
+if [ "$#" -ne 5 ]; then
+    echo "Usage: $0 DOCKERFILE_PATH TESTS_ROOT_PATH IMAGE_TAG ECR_REGISTRY_NAME ECR_REGISTRY_REPO_NAME"
+    exit 1
+fi
+
+DOCKERFILE_PATH="$1"
+TESTS_ROOT_PATH="$2"
+IMAGE_TAG="$3"
+ECR_REGISTRY_NAME="$4"
+ECR_REGISTRY_REPO_NAME="$5"
+
+# Build Docker image
+docker build --platform linux/amd64 -t "$IMAGE_TAG" -f "$DOCKERFILE_PATH" --build-arg TESTS_ROOT="$TESTS_ROOT_PATH" "$TESTS_ROOT_PATH"
+
+# Authenticate Docker with ECR
+aws ecr get-login-password --region us-west-2 | docker login --username AWS --password-stdin "$ECR_REGISTRY_NAME"
+
+# Tag the Docker image with ECR registry name
+docker tag "$IMAGE_TAG" "$ECR_REGISTRY_NAME/$ECR_REGISTRY_REPO_NAME:$IMAGE_TAG"
+
+# Push Docker image to ECR
+docker push "$ECR_REGISTRY_NAME/$ECR_REGISTRY_REPO_NAME:$IMAGE_TAG"
+
+# Verify push success
+if [ $? -eq 0 ]; then
+    echo "Image successfully pushed to ECR."
+else
+    echo "Failed to push image to ECR."
+    exit 1
+fi

charts/wasp/Chart.yaml

@@ -2,5 +2,5 @@ apiVersion: v2
 name: wasp
 description: Wasp cluster test
 type: application
-version: 0.1.7
-appVersion: "0.1.7"
+version: 0.1.8
+appVersion: "0.1.8"

charts/wasp/templates/job.yaml

@@ -25,7 +25,7 @@ spec:
         - name: wasp
           image: {{ $.Values.image }}
           command:
-            - ./wasp_test
+            - ./{{ $.Values.test.binaryName }}
             - -test.v
             - -test.run
             - {{ $.Values.test.name }}

charts/wasp/values.yaml

@@ -7,6 +7,7 @@ sync:
 test:
   name: ""
   timeout: "24h"
+  binaryName: ""
 
 # image + tag string - ${IMAGE}:${TAG} format
 image: public.ecr.aws/chainlink/wasp-test:latest