QA Testing & Simulations

[colin-axner]

Currently the structure of IBC testing at the ecosystem level is undergoing active development and exploration. Different testing frameworks have been initially developed at different parts of the IBC stack.

In ibc-go we have the testing package. This package is great for writing some heavy unit tests. Since IBC unit tests usually require a complex network setup, the testing package automates many of these steps allowing us to easily mock an application state to ensure we are processing proofs and interacting with counterparties in the exact same way the practical environment will. The v2 of this testing package now includes support for external application developers which should help IBC applications minimize technical debt and maximize code coverage.

There also exist model based testing such as those in the transfer keeper. These are great for ensuring implementation compatibility at the module level. We hope to eventually these tests for each IBC module and sub-module. MBT tests have already found security critical bugs and I expect them to serve as a good foundation for code correctness.

At the relayer level, there usually exist some end to end integration tests and developer tools for bootstrapping networks by running actual gaia nodes. These tests are very useful for covering all of the areas of our code since the testing package does not cover all functionality. Relayers will likely fine tune these tests for their own development needs and their own relayer interface. The tests are very useful for manual testing features which may be hard to write end to end tests for.

Simulation testing is something we eventually want to achieve. This would be done in a language and implementation agnostic way. A framework for the entire IBC ecosystem to rely upon. It would likely sit above the relayer part of the stack

This discussion is intended to document testing use cases and possible solutions

[colin-axner]

In this thread I will document the QA testing I perform before the ibc-go v1.0.0 release

So far, my primary concern is version compatibility between chains running the old IBC module in the SDK and the new ibc-go module. I have begun with using Hermes. I started by attempting to use the scripts, which resulted in revealing two issues, 1 in the gaia binary and 1 with an unsupported key format due to a change in the SDK.

After the problems were fixed by the gaia team and hermes team, I have begun testing the updated relayer against the older versions of the gaia nodes. Next I will test the updated relayer against the newer version. If both succeed, I will do the same for the ts-relayer and finally end this batch of testing by setting up a network with chains using different versions and verify that both hermes and ts-relayer function properly

[colin-axner]

Tested the ts-relayer against, new gaia binaries. Only did a check that creation of channels and sending a packet + relay/ack worked and the tokens were received.

Next I will setup a old gaia connected to new gaia. Test with both hermes and ts-relayer and see if I can send a tokens across. Might test timeouts.

Ideally all this in the future can be tested via integration tests. Something along the lines of using gaiad manager to create the testnet environment and then running a set of relayer integrations against those running networks

[colin-axner]

For ts-relayer between old/new gaia binaries, I tested sending token to the counterparty and back as well as timing out packets

I did the same with hermes