Understanding the "DifferentialTest" test contract, particularly the "stateless" modifier

[PaulRBerg]

I was browsing through your repo on the 1.2 branch, and the DifferentialTest contract and the stateless modifier caught my attention.

I am curious what is the goal with this.

Is it test classification in two buckets - (i) stateful and (ii) stateless?

Is it readability in test logs, i.e. not showing state change logs while still being able to run tests as usual?

It might be worth it to document and explain this in the NatSpec comments of the DifferentialTest contract 🙏

[emo-eth]

@PaulRBerg This is a great question with a nuanced answer, due both to how the Seaport repo is set up and quirks with the Foundry execution environment.

The core reason is because the repo contains two implementations of Seaport:

• One is the "optimized" version, which lives in the contracts directory, heavily relies assembly for gas and bytecode optimizations, and is meant to be compiled with the latest version of solc and the --via-ir pipeline (0.8.17 in 1.2)

• The other is the "reference" implementation, which is meant to mirror the "optimized" version's functionality 1:1*, uses no assembly, and is meant to be compiled with an older, more established version of solc and without the --via-ir pipeline (0.8.14 in 1.2).

The Foundry fuzz tests are meant to give confidence that these two implementations indeed match up - so we run the same test with the same inputs for each implementation. However, fulfilling an order with one version of Seaport means the overall state changes - it becomes very inconvenient (if not entirely impractical) to manually undo the state changes effected by fulfilling the order (un-transfering NFTs, resetting token and ether balances, etc).

When these tests were first written, Foundry did not have the vm.snapshot and vm.revertTo cheatcodes, which seem like good solutions to this problem - but actually aren't, due to the fact that HEVM assertion failures are stored in a state variable. This storage slot gets checked by Foundry after the test runs (this is why tests continue running even after an assertion fails). Simply reverting state to a snapshot means that that state variable also gets reverted – so if the first test against, e.g. the reference implementation, fails an assertion, but the second run against the optimized version does not, the test will happily pass because the value of that failure slot was reverted and did not get re-set to indicated the test failed to foundry.

The stateless modifier reverts at the end of a function with the value of the HEVM failure slot - and the various external test functions wrap the actual execution logic in a try/catch which then assert that the revert data matches that of a successful failure slot. This lets us fuzz on inputs, revert state to re-run that same logic, and also ensure that our assertions all pass for both runs.

*In a few cases, on invalid inputs, the optimized and reference implementations will revert with different errors, since the optimized version of Seaport is often handling multiple validation steps at the same time in the assembly blocks - but for successful calls, the idea is that the functionality is exactly the same, and errors wherever possible.

[PaulRBerg]

Thanks again @jameswenzel, perfect answer. All clear now.

It's interesting to see the lengths you went to in order to keep a "reference" implementation that is free of the latest bells and whistles of the Solidity compiler.

Part of my reason for asking about the DifferentialTest was the same as in #808 (my trying to put together a Foundry template based on your set-up in this repo), but another part was the fact that the functionality of the stateless modifier seems to be something that would be useful in general. Thus I wonder if it would be worth considering upstreaming it in forge-std - doing this would take a burden off your shoulder in terms of maintenance (by separating concerns, at least), and it would also make this feature available to the wider Foundry community.