State table verification #169
Conversation
cmd/export_ledger_entry_changes.go
Outdated
| } | ||
| } | ||
| } | ||
|
|
||
| for checkpointLedgers := range verifyOutputs { | ||
| _, err := verify.VerifyState(ctx, verifyOutputs[checkpointLedgers], archive, checkpointLedgers, verifyBatchSize) |
There was a problem hiding this comment.
Do you know where the verifyBatchSize comes from? Do we know that 50000 is big enough?
There was a problem hiding this comment.
This value can be changed. It's basically as big as possible but small enough to not cause OOM crash.
internal/utils/verify/verify.go
Outdated
| return false, errors.Wrap(err, "addLiquidityPoolsToStateVerifier failed") | ||
| } | ||
|
|
||
| return true, nil |
There was a problem hiding this comment.
There should be Verify() call after Write()'ing all ledger entries.
There was a problem hiding this comment.
@bartekn calling Verify() against StateVerifier.currentEntries makes sense when these entries come from the Captive Core? I'm struggling to implement this method as it is because the logic I used in this file is different than the original one.
If implementing this is really necessary, could you shed some light on how to?
There was a problem hiding this comment.
This method needs one argument which is the total number of ledger entries in the user's storage (in our case: BigQuery). This is necessary because due to a bug users can have extra entries. If the provided number is equal to the total number streamed state is correct, if not it means you have extra entries. Obviously you can fool this by passing the number of entries you got from state verifier but it's better to count all entries in the storage.
Overview
This implements Horizon's
verifypackage tostellar-etlcommandexport_ledger_entry_changes. It only verifies entries from checkpoint ledgers (every 64 ledgers, every ~5 minutes).Changes
Raw*fields typesAccountOutput,OfferOutput,PoolOutput,ClaimableBalanceOutputandTrustlineOutputLedgerIsCheckpoint()function toutilspackageTransformedOutputtype tointernal/transform/schema.goto assist the Horizon'sverifypackage implementationinternal/utils/verify.goto work withstellar-etlCloses GH#279
Closes GH#280
Closes GH#281