Imitate state archival invariants in testutils

[dmkozh]

Currently host doesn't contain any state archival-related logic besides providing a way to bump the entries. It's up to the embedder (Core) to enforce the state archival invariants, such as never passing expired entries to the host. We probably don't want to change this logic beyond maybe enforcing the invariants better (stellar/rs-soroban-env#1216).

Test framework, however, doesn't try to enforce and maintain these invariants, which causes confusing behavior of host when e.g. ledger sequence is bumped or storage is set up with expired entries. So we should make sure that expired entries are treated as missing (in case of temp storage) or as errors (in case of persistent storage). This will probably need some env-side support (conditioned on testutils).

[leighmcculloch]

I think this probably is entirely an env side feature and we should move the issue there. The SDK is for the most part rather lightweight in its test utilities, just adding sugar and glue, but the solids of the test utilities is really the host.

[leighmcculloch]

After reading your comment on this other issue (quoted below), I better understand what is being proposed here. Yes implementing this in the SnapshotSource would be an option. I'd still probably do it in the Env, but it can be used here.

Of course, neither of these invariants holds in the test environment at the moment. I think we should approach this holistically and probably try imitating the Core behavior via customizing Snapshot implementation. That would take some time though, so I've opened some issues: #1162, stellar/rs-soroban-env#1216

Ref: stellar/rs-soroban-env#1211 (comment)

[dmkozh]

Yeah, I've been thinking about using the snapshot (which provides a way to inject some custom storage behavior). But while writing this I've realized that it might not be sufficient. I wouldn't sweat on the project too much - the end result is SDK usability, so I'd keep this here. We can add env support as needed.

[brson]

I have added a workaround for the test environment seeing expired storage entries to soroban-token-fuzzer (in brson/soroban-token-fuzzer@6a2dc17#diff-4e7dcf48a54935577ceeef30454dd5161735a3659dc613b6190490df86eef4bbR654).

When the test environment sees expired entries it generates "accessing no-longer-live entry" errors that would not be generated in production.

The way soroban-token-fuzzer advances time is by exporting a snapshot from the env, mutating the sequence number and timestamp of the snapshot, then creating a new Env from the mutated snapshot.

As part of this process, it now iterates over the snapshot ledger entries and deletes expired ones:

fn purge_expired_entries(snapshot: &mut Snapshot) {
    snapshot.ledger.ledger_entries.retain(|entry| {
        let (_key, (_entry, expiration_ledger)) = entry;

        if let Some(expiration_ledger) = expiration_ledger {
            if *expiration_ledger < snapshot.ledger.sequence_number {
                false
            } else {
                true
            }
        } else {
            // what does it mean for storage to not have an expiration ledger?
            true
        }
    });
}

This is easy enough to do in a test case now that I understand the problem better.

From an end-user perspective, it would possibly be more convenient if this was done automatically somewhere, but I can also understand why the Env wouldn't automatically munge a snapshot on import.

I also suspect a lot of testers aren't using snapshots to advance time, but just mutating the env's sequence number and timestamp directly, as that is what most of the docs do. This also results in seeing expired entries.

There could maybe be dedicated testutils functions for advancing time, since it is tricky to do correctly.

fwiw soroban-token-fuzzer is doing some pretty sophisticated things now with the sdk's testutils, and is potentially a useful resource for others who are trying to do more advanced contract testing: it advances time reliably, generates working contract and account addresses, and generates randomized auths for both contract and account addresses.