ci: add npm run conformance (#374)

Adds command to run conformance tests: - `npm run conformance` Very similar to the Go setup, with a few minor modifications. Conformance `package-lock.json` is ignored, because it's just pointing to `file:../../`. <details><summary>FULL LOGS</summary> <p> ``` npm run conformance > @google-cloud/functions-framework@1.10.1 conformance > ./run_conformance_tests.sh Defaulting to latest client. Use './run_conformance_tests vX.X.X' to specify a specific release version. up to date, audited 3 packages in 866ms found 0 vulnerabilities ========== INSTALLING CLIENT@latest ========== Note: only works with Go 1.16+ by default, see run_conformance_tests.sh for more information. Done installing client@latest ========== HTTP CONFORMANCE TESTS ========== 2021/10/26 16:45:24 Validating for http... 2021/10/26 16:45:24 Framework server started. 2021/10/26 16:45:29 HTTP validation started... 2021/10/26 16:45:29 HTTP validation passed! 2021/10/26 16:45:29 Framework server shut down. Wrote logs to /var/folders/27/9q8h1w651z91hycvdf4qf0h0009rgz/T/serverlog_stdout.txt and /var/folders/27/9q8h1w651z91hycvdf4qf0h0009rgz/T/serverlog_stderr.txt. 2021/10/26 16:45:29 All validation passed! ========== BACKGROUND EVENT CONFORMANCE TESTS ========== 2021/10/26 16:45:29 Validating for legacyevent... 2021/10/26 16:45:29 Framework server started. 2021/10/26 16:45:34 Legacy event validation with legacy event requests... 2021/10/26 16:45:34 Events tried: - firebase-auth (PASSED) - firebase-db1 (PASSED) - firebase-db2 (PASSED) - firebase-db3 (PASSED) - firebase-db4 (PASSED) - firebase-db5 (PASSED) - firebase-db6 (PASSED) - firebase-db7 (PASSED) - firebase-db8 (PASSED) - firebase-dbdelete1 (PASSED) - firebase-dbdelete2 (PASSED) - firestore_complex (PASSED) - firestore_simple (PASSED) - legacy_pubsub (PASSED) - pubsub_binary (PASSED) - pubsub_text (PASSED) - storage (PASSED) 2021/10/26 16:45:34 Legacy event validation with CloudEvent requests... 2021/10/26 16:45:34 Events tried: - firebase-auth (PASSED) - firebase-db1 (PASSED) - firebase-db2 (PASSED) - firebase-db3 (PASSED) - firebase-db4 (PASSED) - firebase-db5 (PASSED) - firebase-db6 (PASSED) - firebase-db7 (PASSED) - firebase-db8 (PASSED) - firebase-dbdelete1 (PASSED) - firebase-dbdelete2 (PASSED) - firestore_complex (PASSED) - firestore_simple (PASSED) - legacy_pubsub (PASSED) - pubsub_binary (PASSED) - pubsub_text (PASSED) - storage (PASSED) 2021/10/26 16:45:34 Legacy event validation passed! 2021/10/26 16:45:34 Framework server shut down. Wrote logs to /var/folders/27/9q8h1w651z91hycvdf4qf0h0009rgz/T/serverlog_stdout.txt and /var/folders/27/9q8h1w651z91hycvdf4qf0h0009rgz/T/serverlog_stderr.txt. 2021/10/26 16:45:34 All validation passed! ========== CLOUDEVENT CONFORMANCE TESTS ========== 2021/10/26 16:45:34 Validating for cloudevent... 2021/10/26 16:45:34 Framework server started. 2021/10/26 16:45:39 CloudEvent validation with CloudEvent requests... 2021/10/26 16:45:39 Events tried: - firebase-auth (PASSED) - firebase-db1 (PASSED) - firebase-db2 (PASSED) - firebase-db3 (PASSED) - firebase-db4 (PASSED) - firebase-db5 (PASSED) - firebase-db6 (PASSED) - firebase-db7 (PASSED) - firebase-db8 (PASSED) - firebase-dbdelete1 (PASSED) - firebase-dbdelete2 (PASSED) - firestore_complex (PASSED) - firestore_simple (PASSED) - legacy_pubsub (PASSED) - pubsub_binary (PASSED) - pubsub_text (PASSED) - storage (PASSED) 2021/10/26 16:45:39 CloudEvent validation with legacy event requests... 2021/10/26 16:45:39 Events tried: - firebase-auth (PASSED) - firebase-db1 (PASSED) - firebase-db2 (PASSED) - firebase-db3 (PASSED) - firebase-db4 (PASSED) - firebase-db5 (PASSED) - firebase-db6 (PASSED) - firebase-db7 (PASSED) - firebase-db8 (PASSED) - firebase-dbdelete1 (PASSED) - firebase-dbdelete2 (PASSED) - firestore_complex (PASSED) - firestore_simple (PASSED) - legacy_pubsub (PASSED) - pubsub_binary (PASSED) - pubsub_text (PASSED) - storage (PASSED) 2021/10/26 16:45:39 CloudEvent validation passed! 2021/10/26 16:45:39 Framework server shut down. Wrote logs to /var/folders/27/9q8h1w651z91hycvdf4qf0h0009rgz/T/serverlog_stdout.txt and /var/folders/27/9q8h1w651z91hycvdf4qf0h0009rgz/T/serverlog_stderr.txt. 2021/10/26 16:45:39 All validation passed! ========== DECLARATIVE HTTP CONFORMANCE TESTS ========== 2021/10/26 16:45:39 Validating for http... 2021/10/26 16:45:39 Framework server started. 2021/10/26 16:45:44 HTTP validation started... 2021/10/26 16:45:44 HTTP validation passed! 2021/10/26 16:45:44 Framework server shut down. Wrote logs to /var/folders/27/9q8h1w651z91hycvdf4qf0h0009rgz/T/serverlog_stdout.txt and /var/folders/27/9q8h1w651z91hycvdf4qf0h0009rgz/T/serverlog_stderr.txt. 2021/10/26 16:45:44 All validation passed! ========== DECLARATIVE CLOUDEVENT CONFORMANCE TESTS ========== 2021/10/26 16:45:44 Validating for cloudevent... 2021/10/26 16:45:44 Framework server started. 2021/10/26 16:45:49 CloudEvent validation with CloudEvent requests... 2021/10/26 16:45:49 Events tried: - firebase-auth (PASSED) - firebase-db1 (PASSED) - firebase-db2 (PASSED) - firebase-db3 (PASSED) - firebase-db4 (PASSED) - firebase-db5 (PASSED) - firebase-db6 (PASSED) - firebase-db7 (PASSED) - firebase-db8 (PASSED) - firebase-dbdelete1 (PASSED) - firebase-dbdelete2 (PASSED) - firestore_complex (PASSED) - firestore_simple (PASSED) - legacy_pubsub (PASSED) - pubsub_binary (PASSED) - pubsub_text (PASSED) - storage (PASSED) 2021/10/26 16:45:49 CloudEvent validation with legacy event requests... 2021/10/26 16:45:49 Events tried: - firebase-auth (PASSED) - firebase-db1 (PASSED) - firebase-db2 (PASSED) - firebase-db3 (PASSED) - firebase-db4 (PASSED) - firebase-db5 (PASSED) - firebase-db6 (PASSED) - firebase-db7 (PASSED) - firebase-db8 (PASSED) - firebase-dbdelete1 (PASSED) - firebase-dbdelete2 (PASSED) - firestore_complex (PASSED) - firestore_simple (PASSED) - legacy_pubsub (PASSED) - pubsub_binary (PASSED) - pubsub_text (PASSED) - storage (PASSED) 2021/10/26 16:45:49 CloudEvent validation passed! 2021/10/26 16:45:49 Framework server shut down. Wrote logs to /var/folders/27/9q8h1w651z91hycvdf4qf0h0009rgz/T/serverlog_stdout.txt and /var/folders/27/9q8h1w651z91hycvdf4qf0h0009rgz/T/serverlog_stderr.txt. 2021/10/26 16:45:49 All validation passed! ``` </p> </details>
GoogleCloudPlatform · Oct 27, 2021 · 8110d3a · 8110d3a
1 parent 78ba5c1
commit 8110d3a
Show file tree

Hide file tree

Showing 4 changed files with 69 additions and 1 deletion.
diff --git a/.gitignore b/.gitignore
@@ -13,4 +13,5 @@ yarn-error.log
 function_output.json
 serverlog_stderr.txt
 serverlog_stdout.txt
-temp
+temp
+test/conformance/package-lock.json
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -34,6 +34,14 @@ To run an individual test, you can run a command such as the following:
 npm run test -- -g 'loading function'
 ```
 
+### Conformance Tests
+
+To run the conformance tests, first install Go 1.16+, then run the tests:
+
+```
+npm run conformance
+```
+
 ### Manual Testing
 
 When developing a feature locally, you can install a local version of the Functions Framework

diff --git a/package.json b/package.json
@@ -18,6 +18,7 @@
   },
   "scripts": {
     "test": "mocha build/test --recursive",
+    "conformance": "./run_conformance_tests.sh",
     "check": "gts check",
     "clean": "gts clean",
     "compile": "tsc -p .",

diff --git a/run_conformance_tests.sh b/run_conformance_tests.sh
@@ -0,0 +1,58 @@
+#!/bin/bash
+# Runs the conformance tests locally from https://github.com/GoogleCloudPlatform/functions-framework-conformance
+#
+# Servers may fail to shutdown between tests on error, leaving port 8080 bound.
+# You can see what's running on port 8080 by running `lsof -i :8080`. You can
+# run `kill -9 <PID>` to terminate a process.
+#
+# USAGE:
+# ./run_conformance_tests.sh [client_version]
+#
+# client_version (optional):
+# The version of the conformance tests client to use, formatted as "vX.X.X".
+# Defaults to the latest version of the repo, which may be ahead of the
+# latest release.
+
+CLIENT_VERSION=$1
+if [ $CLIENT_VERSION ]; then
+    CLIENT_VERSION="@$CLIENT_VERSION"
+else
+    echo "Defaulting to latest client."
+    echo "Use './run_conformance_tests vX.X.X' to specify a specific release version."
+    CLIENT_VERSION="@latest"
+fi
+
+function print_header() {
+    echo
+    echo "========== $1 =========="
+}
+
+# Fail if any command fails
+set -e
+
+# Navigate to the conformance tests and install the local FF.
+cd test/conformance;
+npm i;
+
+print_header "INSTALLING CLIENT$CLIENT_VERSION"
+echo "Note: only works with Go 1.16+ by default, see run_conformance_tests.sh for more information."
+# Go install @version only works on go 1.16+, if using a lower Go version
+# replace command with:
+# go get github.com/GoogleCloudPlatform/functions-framework-conformance/client$CLIENT_VERSION && go install github.com/GoogleCloudPlatform/functions-framework-conformance/client
+go install github.com/GoogleCloudPlatform/functions-framework-conformance/client$CLIENT_VERSION
+echo "Done installing client$CLIENT_VERSION"
+
+print_header "HTTP CONFORMANCE TESTS"
+client -buildpacks=false -type=http -cmd='npm start -- --target=writeHttp --signature-type=http' -start-delay 5 -validate-mapping=true
+
+print_header "BACKGROUND EVENT CONFORMANCE TESTS"
+client -buildpacks=false -type=legacyevent -cmd='npm start -- --target=writeLegacyEvent --signature-type=event' -start-delay 5 -validate-mapping=true
+
+print_header "CLOUDEVENT CONFORMANCE TESTS"
+client -buildpacks=false -type=cloudevent -cmd='npm start -- --target=writeCloudEvent --signature-type=cloudevent' -start-delay 5 -validate-mapping=true
+
+print_header "DECLARATIVE HTTP CONFORMANCE TESTS"
+client -buildpacks=false -cmd='npm start -- --target=writeHttpDeclarative' -start-delay 5 -validate-mapping=true
+
+print_header "DECLARATIVE CLOUDEVENT CONFORMANCE TESTS"
+client -buildpacks=false -type=cloudevent -cmd='npm start -- --target=writeCloudEventDeclarative' -start-delay 5 -validate-mapping=true