Testing Plan

[wileyj]

Testing Plan

Questions for the Devs

• Which repos/branches are required to build the binaries/docker images we'll need to be running?
• How to configure the signer binary? (https://github.com/stacks-network/stacks-core/blob/next/testnet/stacks-node/src/tests/signer.rs#L65-L153)
• Any specific configuration changes needed for the stacks-blockchain? Stackerdb?

On-demand Testing (closed-membership network)

What to test

Regression Tests (Existing)

• stacks-increase works as expected
• stacks-unlock works as expected
• stacking to a segwit and taproot address works in pox-4
• stacking to a segwit and taproot address receives payout
• verify that btc wire format tx for delegate-stx works

Feature Tests

• Voting
• StackerDB maintenance
• Traits
  • Old trait regression tests
  • New trait feature tests
  • Cross-epoch contract-calls, with traits defined in different epochs
• TBD

Frameworks

Testground is the best fit for an off the shelf solution, and is designed for testing distributed systems. The tests will:

• spin up n+1 nodes in a managed k8s cluster or larger VM
• simulate network issues and other potential breakage scenarios
• Expose various metrics/logs per test plan

Questions

• What is an extremely simple test case we can use to iterate with on the framework?
• Potentially variable per test plan
  • What metrics are we measuring?
  • What should be the minimum number of nodes?

Help Needed

• Writing new tests for testground
• Translating current tests to something we can run via testground

Timeframe

Late December/Early January for first iteration

Persistent Testing (closed-membership network)

This will be a close approximation of what we can expect in a real-world environment.

Network Requirements

• Persistent chainstate data across reboots
• Single seed node to be used by all subsequent nodes
• Single Bitcoin regtest node with a miner starting from genesis
• Single stacks miner using the Bitcoin regtest network
• 5-20 VM's (ability to scale up as needed, potentially to 150 VM's)
• Geographically disperse deployments
  • 50% NA
  • 30% EU
  • 20% AP
• Mix of public and nat'ed nodes ( first iteration can have ports 20443-20444 ingress firewalled )
• Command and control method to execute scripts/tests across all nodes
  • Ex: https://aws.amazon.com/batch/

What to test

• How quickly are blocks propagated to all nodes
• How much time a DKG round takes to perform
• How much time a signing round takes to perform
• What happens if a DKG signer is not responding or otherwise behaving unexpectedly

Frameworks

We’ll use AWS cloud VMs to host the persistent tests. AWS is the obvious option since it will give us access to multiple regions around the globe, provide access to tools to bootstrap the network (cloudformation/terraform etc), and also provides some of the requirements i.e. the command and control tool without having to create anything bespoke.

Stacks blockchain binary will be built from source on each VM by a script run via cron.

Questions

• What are the configuration requirements for the stacks-node binary
• What are the configuration of the signer binary
• How and what are we measuring?
  • DKG round time
  • Signing round time
  • Block propagation time

Options for Retrieving Metrics

• Option 1: Collect the targeted data that is of interest from this framework, whether that be using the existing prometheus framework and only collect a few specific metrics.
• Option 2: Create a sidecar script that collects a few datapoints/grepped log entries and ships them somewhere.

Timeframe

Late December / Early January for first iteration

Appendix

AWS Cloudwatch Agent: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Install-CloudWatch-Agent.html
AWS Batch: https://aws.amazon.com/batch/
Testground Rust test example: https://github.com/testground/testground/blob/master/plans/example-rust/src/main.rs
Testground metric collection plan example: https://docs.testground.ai/master/#/runner-library/cluster-k8s/monitoring?id=infrastructure-monitoring
Testground example plans for rust and javascript: https://github.com/testground/testground/tree/master/plans
Previous Testground plan for 2.0: https://github.com/stacks-network/stacks-blockchain-testground

[wileyj]

@stacks-network/blockchain-team @CharlieC3 @saralab

[friedger]

Not sure whether that is related to this discussion. I would like to test with devenv and run deposit and withdrawal tests: https://github.com/stacks-network/sbtc/tree/main/devenv

[wileyj]

thanks! it's not directly related, but those are definitely two tests we can/should be running with the proposed on-demand framework

[jferrant]

Should test

• DKG round
• Signing round
  These are two sep things.

[wileyj]

updated, thanks!

[wileyj]

@CharlieC3 re: testground, @zone117x shared this that we should consider: libp2p/test-plans#103

[CharlieC3]

It's a fair point, testgrounds doesn't necessarily bring any functionality to the tech stack we can't implement through other means. The primary benefit I see is with testgrounds it's all assembled under one roof, and it's an ecosystem that's already proven to work. We could technically bring scale and network emulation to the table in other ways, albeit may not be as well-defined a path.

I also understand that it comes with its own SDK, but from what I can tell we only used the SDK minimally in the past, just as a means to start the private stx network. And since we already have this much written out, adopting it to a new deployment might be trivial. Without testgrounds, we'd still find ourselves writing some kind of automation to orchestrate everything.

I think what's difficult is with testgrounds there is no middle ground I'm picking up. Meaning, the MVP of a testing framework using testgrounds is not much different from the full-fledged implementation of it. Without testgrounds, we could start small and manual, and gradually improve things over time.

I'm not familiar with the complexity that comes with testgrounds in totality though, I think only you are @wileyj. If there's something I have wrong or am missing, please let me know!

[wileyj]

nothing you've said is incorrect - but i think we should take a second look before introducing a new tool.
@zone117x pointed out that he's been doing something similar to what we need on a small scale using docker-compose.

i became curious after reading the link above, and i think it's worth a second look to see if will meet our needs - the biggest question that we've asked to help us here is minimum number of nodes. If that answer for that specific framework is 10 or so....docker-compose is a very compelling solution that we have a lot of experience with (or even k8s).

the main issue (and i agree with this) is that with testground, it's a new tool we'd have to learn while still running the same shell-type scripts etc that we would already run - with the added complexity of running them using golang and in a new framework.

i don't have an opinion yet - just sharing what was sent to me so we can at last take another look and adjust our plan if we feel something like docker-compose is a better fit for what we need today.

[wileyj]

I also understand that it comes with its own SDK, but from what I can tell we only used the SDK minimally in the past, just as a means to start the private stx network. And since we already have this much written out, adopting it to a new deployment might be trivial. Without testgrounds, we'd still find ourselves writing some kind of automation to orchestrate everything.

trivial depends on the type of test and the expected result/output. the code you're referring to was specifically tailored for net-test (and even then, it required some changes for it to run correctly in testground).

@moodmosaic also had an interesting idea that might be useful to @kantai @jcnelson around how to run the tests, taking ideas from his property testing for sbtc. if we were to align on "this is how we'll run the tests" - then where we run them becomes an easier question to answer.

i'm afraid we don't know enough about what tests and how to run them, so we're making quite a few assumptions that may end up being overkill for what's needed.

[wileyj]

@CharlieC3 i was also incorrect in that i though swarm was deprecated, it still seems very much alive: https://docs.docker.com/engine/swarm/

i think depending on the size of the network, this might be a good option in the timeframe we've been given since we already have experience with docker and docker-compose.
basically it's just k8s with less overhead. i'll have to read more, but on paper it does seem interesting for what we know our needs to be today.

still have questions though, so i'll read through the docs

[moodmosaic]

@zone117x shared this that we should consider: libp2p/test-plans#103

This is an interesting approach, thanks for sharing.

And while Docker Compose and Testground are excellent for network and performance testing of distributed systems, model-based testing offers a more focused approach, particularly valuable for systematically validating the logical correctness of the system through a model of its state and behavior.

Integrating these approaches could be beneficial: leveraging Docker Compose or Testground for their strengths in network and performance aspects, while employing model-based testing for verifying logical correctness and handling edge cases. This could provide a comprehensive testing solution, ensuring robustness under various conditions.

Model-based testing involves:

• Model State: A simplified representation of the system's state.
• Commands: Operations that can be performed, including their expected effects on the model state.
• Assertions: Checking after each command to ensure the actual system state aligns with the model.

This not only ensures that nothing adverse occurs but also verifies that essential conditions consistently hold true. I applied this method in testing sBTC asset.clar, demonstrating its effectiveness.

This dual approach (of Docker Compose, or Testground, and model-based testing) might provide the robustness we need.

[diwakergupta]

Thanks all for the discussion, esp. @wileyj and @CharlieC3 !

I wanted to add some quick thoughts, apologies if it's redundant / if folks are already aligned on these:

• It's far more important to agree on the what to test vs. the how to test it.
• Let's assume basic functional tests will be a part of the development process. So the testing here should focus more on end-to-end, integration and performance tests. Further, this workstream should prioritize multi-node testing (vs. single node).
• If we're blocked on getting a better handle on either the test scope or the code, let's use that time to build out the capabilities we know we will need. And we can test those capabilities to answer some of the questions above for the existing Stacks blockchain.

For instance, if we had good tooling and automation in place to:

• quickly spin up a network of Stacks (and bitcoin) nodes, with configurable topologies of miners, different network conditions etc
• instantiate and / or drive the network through various events (e.g. stacking enabled, run through a few cycles)
• generate different workloads against this network (e.g. lots of transfers vs. different types of contract calls), on Stacks as well as the burnchain (e.g. flash blocks or high fees)
• capture relevant metrics / logs
• quickly & reliably reset the network (from scratch or from a last known good state)

Then we could start building baselines on the current blockchain, and iteratively test Nakamoto builds against those baselines as features get added. Things like (many of these have already been mentioned above, so I'm listing these to be illustrative not exhaustive):

• how quickly does the consensus algorithm converge
• how long does block production take under different types of mempool conditions
• how long does block propagation take under different mempool and network conditions

It will be a bunch of work to build out these capabilities, so there should be no blockers on Nakamoto work IMO :)

[jcnelson]

Literally all of this can be done in some form with net-tests.

[diwakergupta]

That's great to hear!