Distributed Execution Proposal

[na--]

I know that #140 already exists! 😅 However, that issue has a lot history, unrelated discussion and comments, so at this point it isn't very suitable for actually tracking the work that has been done and that remains to be done on the topic...

#140 is also a generic tracking issue for the "distributed execution" capability and will remain open until that is natively supported by k6, however that happens. While I needed some central place that allows me to link up and explain all of the parts of my specific proposal for how to get there. More specifically, this issue is for tracking the full implementation and delivery of a stable version of the distributed execution PoC that originally started as #2438 and was later further developed in #2816. From the design document that explains it, to the PRs that implement parts of it, to the missing parts that haven't been implemented yet. This issue is for all of that, and I'll just link to it in a #140 comment 😅

I will also be taking a long vacation/sabbatical for the next few months and then gradually moving to another team. Because of that (and somewhat prompted by @ragnarlonn's comments in #140 😅), I thought it made sense to try and write down my thoughts and ideas on the topic. It's very unlikely that I will be able to finish the distributed execution work myself, so I've tried to make what exists as code and ideas in my head as easy to adopt and built upon as it's practical to do before I go away for a while.

Since I won't be around (I might not have been as diligent otherwise... 😅), I've written a design document and refactored the original proofs of concept into backwards-compatible (and so, hopefully merge-able) PRs 🤞

Here is a (hopefully mostly complete) list of tasks that remain:

Planning

Beta Give feedback

1. Refactor the original PoC so that HDR histograms are the last commit and distributed execution can be split off into multiple atomic and self-sufficient commits/PRs that are completely backwards compatible.
2. Merge minor prerequisite refactoring commits (Refactor various metric APIs to make distributed execution easier #3191) and update xk6-output-prometheus-remote to resolve conflicts with API usage of internal k6 APIs (Update the k6 dependency to the latest master version xk6-output-prometheus-remote#133, Update xk6-output-prometheus-remote to its latest main commit #3210).
3. Agree and adopt (i.e. merge) the design document Add a design doc for distributed execution and test suites #3217. Or at least do so provisionally, if no major issues are found with it and nothing better is suggested by someone else. Since all of this should be considered experimental until stated otherwise, so it could be discarded if a better approach is found later, even if some of the PRs were already merged.

Experimental version

Beta Give feedback

1. Merge Introduce execution.Controller with a local no-op implementation #3204.
2. Add Error handling via a dedicated pull request to Basic and super-experimental support for distributed execution #3205
3. Add a good enough level of test coverage via a dedicated pull request to Basic and super-experimental support for distributed execution #3205
4. Merge Basic and super-experimental support for distributed execution #3205.
5. Add support for mutual gRPC authentication between k6 agent and k6 coordinator

Integrate with other k6 products

Beta Give feedback

1. Work on transitioning k6-operator
2. Use the experimental version on Grafana Cloud k6 for the new distributed execution

Local metrics

Beta Give feedback

1. Support Cloud metrics output (aka k6 run -o cloud script.js). It requires probably the implementation of cloudRunLocally command #3282.
2. Implement HDR/sparse histogram support (Use HDR histograms for calculating percentiles in thresholds and summary stats #763); [WIP] Proof of concept changes (test suites, HDR histograms) #2816 has a PoC (moved as the last commit), but more research is definitely needed, I picked the library I used without too much scrutiny. Or implement the Open Telemetry output if it is confirmed as a solution.
3. Support Thresholds and End-to-end test result refining Support thresholds and the end-of-test summary in distributed execution #3213

UX refinement

Beta Give feedback

1. Add more flexibility beyond --instance-count parameter, to specify how a test is segmented between instances and how each instance is matched to each execution segment. Hopefully with some thoughts on how that can work together with test suites (Test suites / execute multiple scripts with k6 #1342, also in the design doc) down the line, or at least not interfere.

I've written these tasks in the order I think it makes sense to do them to get the end result with the least amount of effort and time. Though a lot of them can also be done out of order (e.g. HDR histograms can be done first, as was the case originally in the PoC, or tests can be added before some existing PR is merged 😊 😅).

This whole issue and items in the list above can be reordered, removed, added to and checked off at the discretion of whoever picks up this work. And this whole issue can be closed if the approach is considered nonviable and a better alternative exists, without affecting #140.