"All testers do exploratory testing. Some do it more deliberately and in intentionally skilled ways." - Cem Kaner
"Measurement, then, is just one means to an end. That end is: the marshaling of good evidence to inform our assessments of quality that can be used to make business decisions. (e.g. when to ship the product; how good is the product team; are problems with the product or the team persisting, accumulating, or getting resolved?)" - James Bach
Go Language by itself is a powerful and high-performance language built for scalability. It has many libraries that can be used to create automation, but there are some high-level test strategies that can be more easily implemented (with less lines of code) using a generic RPA framework to complement the strengths of Go. For example, the following strategies are not easy to quickly create within a day or two using only pure Go Language code.
- Model-Based Tests using Graphwalker's visual .graphml files, combined with Chaos Tests
- Load Tests combined with Chaos Tests
- Mutational Fuzz Tests combined with Model-Based Tests
- Slack-bots that report failures on all of the above
- Listener Interface that monitors every test and performs actions during runtime
- Control specific groups of tests or isolate the automation to run only one test
- Automation metrics and results easily combined into a single detailed log
- Powerful built-in automation documentation traceability from the log that can tie CI pipeline URLs to acceptance criteria and test IDs (Jira, Test Management Platforms... TestRail etc.)
This repository is a collection of Go Language test tools and strategies enhanced with Robot Framework, an open source RPA (Robotic Process Automation) and generic automation framework. The goal is to demonstrate how these two sides can work together to form a more powerful test strategy than just one side by itself. Here is a list of the Go Language tools and components that are being utilized. The examples utilizing these components and tools can also be used without RPA for hands-on exploratory testing.
- go test, the built-in Go Language test tool
- httpstat and go-httpstat, a Go Language tool and library (similar to
curl) that gathers various metrics of HTTP requests - vegeta, a versatile and customizable HTTP load testing tool and library
- toxiproxy, a TCP Chaos Proxy tool
In the future I plan to expand this toolkit into the following areas.
- Experiment with the f1 performance test framework. Try to create some examples.
- Experiment with the robotgo cross-platform GUI automation framework. Try to create some examples.
- Experiment with the py4go project. Try to create some examples.
All of the examples in this repository are designed to run within a Docker container. It is the only requirement to run everything explained below in this walkthrough section. The following documentation can be used to install Docker.
Most of the tests will be targeting a prism mock API server through a TCP Chaos Proxy called toxiproxy, the rest of the tests (Functional Tests specifically) will be targeting the prism server directly. The generic-automation.robot file (combined with the ToxiproxyChaosTestLibrary.py and PrismMockServerLibrary.py RPA automation libraries) manages the setup and teardown of these tools. The documentation links below provide more details.
- https://github.com/stoplightio/prism/tree/master/docs/getting-started
- https://github.com/Shopify/toxiproxy?tab=readme-ov-file#usage
The generic-automation.robot file uses the ToxiproxyChaosTestLibrary.py to create an exploratory-testing-proxy that will concurrently run the following "toxics". In toxiproxy a "toxic" is the Chaos Testing mechanism that is challenging the server's resilience by simulating real-world network problems.
The Functional Tests are handled by the functional-tests_test.go and httpstat-test-runner.go files. For MacOS and Linux users, you can run these tests with the following command from a terminal.
bash ./run-go-rpa-tests.sh Run-Specific-Tests-Inside-Docker Start-Go-Language-Functional-Tests
Windows users can run the following commands.
docker-compose -f docker-compose.yml builddocker-compose run docker-test-runner run-go-rpa-tests.sh Start-Go-Language-Functional-Tests
The results of the go test runner is set to provide JSON output. The following script can analyze the JSON results and generate clearer results in an HTML log file.
bash ./run-go-rpa-tests.sh Run-Specific-Tests-Inside-Docker Analyze-Functional-Tests-Generate-HTML-Logs
Optionally, the httpstat-test-runner.go file can be used for hands-on exploratory testing using the following commands.
- (MacOS/Linux users)
bash ./run-go-rpa-tests.sh Run-Specific-Tests-Inside-Docker Manual-Scripted-Tests-In-Docker - (Windows users)
docker-compose -f docker-compose.yml build, then rundocker-compose run docker-test-runner run-go-rpa-tests.sh Manual-Scripted-Tests-In-Docker
The Load Tests are handled by the vegeta-load-test-runner.go and VegetaLoadTestLibrary.py files. The script below runs two types of Load Tests, (1) a ramp-up load test runner and (2) a cusomizable load test runner. All of the Load Tests go through toxiproxy first before reaching the prism mock server.
bash ./run-go-rpa-tests.sh Run-Specific-Tests-Inside-Docker Start-Chaos-Proxy-Load-Tests
Windows users can run the following commands.
docker-compose -f docker-compose.yml builddocker-compose run docker-test-runner run-go-rpa-tests.sh Start-Chaos-Proxy-Load-Tests
The Model-based Tests are handled by the generic-automation.robot and GraphwalkerModel.graphml files. In addition to the toxiproxy Chaos Proxy automation mentioned above, the Graphwalker Model-based Tests displayed in the screenshot below use the httpstat-test-runner.go file and a Mutational Fuzzer called Radamsa.
For MacOS and Linux users, you can run these Model-based Tests with the following command from a terminal.
bash ./run-go-rpa-tests.sh Run-Specific-Tests-Inside-Docker Start-Graphwalker-Model-Based-Tests
Windows users can run the following commands.
docker-compose -f docker-compose.yml builddocker-compose run docker-test-runner run-go-rpa-tests.sh Start-Graphwalker-Model-Based-Tests
More information about Model-based Testing can be found in the links below.
- https://graphwalker.github.io/#articles
- https://www.harryrobinson.net/
- https://testoptimal.com/ref/starwest-2006-mbt-tutorial.pdf
[RPA Bonus Features] Slack-bot + Listener Interface + Specific Tag Runner + Combined Test Metrics Results
The Slack-bot is a part of the generic-automation.robot file, and will send a Slack message using slacktee when any tests fail. The screenshots below demonstrate how a Model-based Test failure is sent to a specific Slack channel.
The DurationTrackingListener.py uses the Robot Framework Listener Interface to monitor the execution of all tests. Robot Framework's Listener Interface has many capapbilities, more details about it can be found in the links below.
- https://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#listener-interface
- https://www.youtube.com/watch?v=lKu-9WKtYcg
When the DurationTrackingListener.py is set to a very low number (for example, max_seconds=1) this will very easily show the test monitoring capabilities of an RPA framework. The screenshots below demonstrate what happens when I added a 30 second delay.
Another useful side-effect of having the generic-automation.robot file act as an overseer of all tests is that it can combine the results of all tests into a single HTML log file, or run specific tests by their Test_ID (found in test management tools such as Zephyr Scale or TestRail) or Jira_ID. This is useful for a quick overview of all test results/metrics in one place and see which Test_ID/Jira_ID each test is mapped to. The screenshots of the combined-test-results-log.html below demonstrates these features.
The run-go-rpa-tests.sh script is a tool runner script that can run all of the above workflows in a Docker container. The script has the following options for running on a local work station or in GitHub Actions CI.
You can view a help menu (without triggering any automation) by running 'bash ./run-go-rpa-tests.sh -h' or 'bash ./run-go-rpa-tests.sh --help'
bash ./run-go-rpa-tests.sh Stop-Containers-And-Build-Docker-Container-With-Compose
bash ./run-go-rpa-tests.sh Run-Specific-Tests-Inside-Docker Start-Go-Language-Functional-Tests
bash ./run-go-rpa-tests.sh Run-Specific-Tests-Inside-Docker Start-Chaos-Proxy-Model-Based-Tests
bash ./run-go-rpa-tests.sh Run-Specific-Tests-Inside-Docker Start-Chaos-Proxy-Load-Tests
bash ./run-go-rpa-tests.sh Run-Specific-Tests-Inside-Docker Analyze-Functional-Tests-Generate-HTML-Logs
bash ./run-go-rpa-tests.sh Run-Specific-Tests-Inside-Docker Start-All-Tests
bash ./run-go-rpa-tests.sh Run-Specific-Tests-Inside-Docker Start-With-Test-ID-Or-Jira-ID Test_ID3
bash ./run-go-rpa-tests.sh Run-Specific-Tests-Inside-Docker Start-With-Test-ID-Or-Jira-ID Jira_ID2
bash ./run-go-rpa-tests.sh Run-Specific-Tests-Inside-Docker Manual-Scripted-Tests-In-Docker