assumeutxo: Get rid of faked nTx and nChainTx values #29370

ryanofsky · 2024-02-02T16:18:52Z

The PopulateAndValidateSnapshot function introduced in f6e2da5 from #19806 has been setting fake nTx and nChainTx values that can show up in RPC results (#29328) and make CBlockIndex state hard to reason about, because it is difficult to know whether the values are real or fake.

Revert to previous behavior of setting nTx and nChainTx to 0 when the values are unknown, instead of faking them. Also drop no-longer needed BLOCK_ASSUMED_VALID flag.

Dropping the faked values also fixes assert failures in the CheckBlockIndex (pindex->nChainTx == pindex->nTx + prev_chain_tx) check that could happen previously if forked or out-of-order blocks before the snapshot got submitted while the snapshot was being validated. The PR includes two commits adding tests for these failures and describing them in detail.

Compatibility note: This change could cause new -checkblockindex failures if a snapshot was loaded by a previous version of Bitcoin Core and not fully validated, because fake nTx values will have been saved to the block index. It would be pretty easy to avoid these failures by adding some compatibility code to LoadBlockIndex and changing nTx values from 1 to 0 when they are fake (when (pindex->nStatus & BLOCK_VALID_MASK) < BLOCK_VALID_TRANSACTIONS), but a little simpler not to worry about being compatible in this case.

DrahtBot · 2024-02-02T16:18:55Z

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage

For detailed information about the code coverage, see the test coverage report.

Reviews

See the guideline for information on the review process.

Type	Reviewers
ACK	Sjors, mzumsande, maflcko, achow101
Concept ACK	fjahr

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#29656 (chainparams: Change nChainTx type to uint64_t by fjahr)
#29617 (test: Validate UTXO snapshot with coin height > base height & amount > MAX_MONEY supply by jrakibi)
#29553 (assumeutxo: Add dumptxoutset height param, remove shell scripts by fjahr)
#28339 (validation: improve performance of CheckBlockIndex by mzumsande)
#27006 (reduce cs_main scope, guard block index 'nFile' under a local mutex by furszy)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

maflcko · 2024-02-02T16:47:18Z

Backwards compatibility code in LoadBlockIndex to override nTx == 1 values set by previous code when they are fake (when (pindex->nStatus & BLOCK_VALID_MASK) < BLOCK_VALID_TRANSACTIONS)

Not sure. This can only happen on test-chains, so I think it would be cleaner to just require the developers, if there are any, to finish their background sync, or to wipe their chainstate, or the blocksdir, or the whole test data dir.

DrahtBot · 2024-02-02T17:16:09Z

🚧 At least one of the CI tasks failed. Make sure to run all tests locally, according to the
documentation.

Possibly this is due to a silent merge conflict (the changes in this pull request being
incompatible with the current code in the target branch). If so, make sure to rebase on the latest
commit of the target branch.

Leave a comment here, if you need help tracking down a confusing failure.

_{Debug: https://github.com/bitcoin/bitcoin/runs/21160915955}

ryanofsky · 2024-02-02T18:44:52Z

Not sure. This can only happen on test-chains, so I think it would be cleaner to just require the developers, if there are any, to finish their background sync, or to wipe their chainstate, or the blocksdir, or the whole test data dir.

It makes sense that having backwards compatibility may not be worth it, but I don't see why test chains would set nTx to a real value without also setting BLOCK_VALID_TRANSACTIONS. It seems like we should be able to use BLOCK_VALID_TRANSACTIONS to know if nTx is real or fake, as long as we are checking for BLOCK_VALID_TRANSACTIONS, directly, and not using the IsValid() helper which is also influenced by BLOCK_FAILED flags.

Updated 4c70d99 -> a3228f0 (pr/nofake.1 -> pr/nofake.2, compare) to fix broken validation_block_tests due to a buggy check, also making some minor cleanups

maflcko

lgtm

src/chain.h

src/test/validation_chainstatemanager_tests.cpp

src/test/util/chainstate.h

src/test/validation_chainstatemanager_tests.cpp

src/validation.cpp

maflcko · 2024-02-05T15:00:52Z

Also, it would be good to include the test to ensure the crash no longer happens.

#29261 (comment)

ryanofsky

Thanks for the close review! Will work on your suggestions

src/chain.h

src/test/util/chainstate.h

src/test/validation_chainstatemanager_tests.cpp

src/validation.cpp

ryanofsky

Updated a3228f0 -> 3bdbc3f (pr/nofake.2 -> pr/nofake.3, compare) implementing all suggestions, adding test coverage for the RPCs, and adding a new commit that drops the no longer needed BLOCK_ASSUMED_VALID flag.
Updated 3bdbc3f -> 603a92c (pr/nofake.3 -> pr/nofake.4, compare) removing more BLOCK_ASSUMED_VALID references

src/test/util/chainstate.h

src/test/validation_chainstatemanager_tests.cpp

src/validation.cpp

src/chain.h

src/validation.cpp

test/functional/feature_assumeutxo.py

maflcko · 2024-02-06T09:04:07Z

Needs rebase

ryanofsky · 2024-02-07T01:07:56Z

Needs rebase

Yes, thanks for recognizing the CI failures.

Rebased 603a92c -> 3f0b860 (pr/nofake.4 -> pr/nofake.5, compare) with test fixes needed after silent conflict with #29354

maflcko · 2024-02-07T08:25:29Z

Could include the test #29370 (comment) , or did you want to leave that for later?

Add ubsan suppressions for integer overflows in the getchaintxstats RPC. getchainstatstx line "int nTxDiff = pindex->nChainTx - past_block.nChainTx" can trigger ubsan integer overflows when assumeutxo snapshots are loaded, from subtracting unsigned values and assigning the result to a signed int. The overflow behavior probably exists in current code but is hard to trigger because it would require calling getchainstatstx at the right time with specific parameters as background blocks are being downloaded. But the overflow behavior becomes easier to trigger in the upcoming commit removing fake nChainTx values, so a suppression needs to be added before then for CI to pass. getchainstatstx should probably be improved separately in another PR to not need this suppression, and handle edge cases and missing nChainTx values more carefully.

…nected block Use Assume macro as suggested bitcoin#29370 (comment)

The checks are changing slightly in the next commit, so try to explains the ones that exist to avoid confusion (bitcoin#29370 (comment))

The `PopulateAndValidateSnapshot` function introduced in f6e2da5 from bitcoin#19806 has been setting fake `nTx` and `nChainTx` values that can show up in RPC results (see bitcoin#29328) and make `CBlockIndex` state hard to reason about, because it is difficult to know whether the values are real or fake. Revert to previous behavior of setting `nTx` and `nChainTx` to 0 when the values are unknown, instead of faking them. This commit fixes at least two assert failures in the (pindex->nChainTx == pindex->nTx + prev_chain_tx) check that would happen previously. Tests for these failures are added separately in the next two commits. Compatibility note: This change could result in -checkblockindex failures if a snapshot was loaded by a previous version of Bitcoin Core and not fully validated, because fake nTx values will have been saved to the block index. It would be pretty easy to avoid these failures by adding some compatibility code to `LoadBlockIndex` and changing `nTx` values from 1 to 0 when they are fake (when `(pindex->nStatus & BLOCK_VALID_MASK) < BLOCK_VALID_TRANSACTIONS`), but a little simpler not to worry about being compatible in this case.

Add a test for a CheckBlockIndex crash that would happen before previous "assumeutxo: Get rid of faked nTx and nChainTx values" commit. The crash was an assert failure in the (pindex->nChainTx == pindex->nTx + prev_chain_tx) check that would previously happen if a snapshot was loaded, and a block was submitted which forked from the chain before the snapshot block and after the last downloaded background chain block. This block would not be marked assumed-valid because it would not be an ancestor of the snapshot, and it would have nTx set, nChainTx unset, and prev->nChainTx set with a fake value, so the assert would fail. After the fix, prev->nChainTx is unset instead of being set to a fake value, so the assert succeeds. This test was originally posted by maflcko in bitcoin#29261 (comment) Co-authored-by: MarcoFalke <*~=`'#}+{/-|&$^_@721217.xyz>

Add a test for a CheckBlockIndex crash that would happen before previous "assumeutxo: Get rid of faked nTx and nChainTx values" commit. The crash was an assert failure in the (pindex->nChainTx == pindex->nTx + prev_chain_tx) check that would previously happen if the snapshot block was submitted after loading the snapshot and downloading a few blocks after the snapshot. In that case ReceivedBlockTransactions() previously would overwrite the nChainTx value of the submitted snapshot block with a fake value based on the previous block, so the (pindex->nChainTx == pindex->nTx + prev_chain_tx) check would later fail on the first block after the snapshot. This test was originally posted by Martin Zumsande <mzumsande@gmail.com> in bitcoin#29370 (comment) Co-authored-by: Martin Zumsande <mzumsande@gmail.com>

Flag adds complexity and is not currently used for anything.

…nected block Use Assume macro as suggested bitcoin#29370 (comment)

The checks are changing slightly in the next commit, so try to explains the ones that exist to avoid confusion (bitcoin#29370 (comment))

Add a test for a CheckBlockIndex crash that would happen before previous "assumeutxo: Get rid of faked nTx and nChainTx values" commit. The crash was an assert failure in the (pindex->nChainTx == pindex->nTx + prev_chain_tx) check that would previously happen if the snapshot block was submitted after loading the snapshot and downloading a few blocks after the snapshot. In that case ReceivedBlockTransactions() previously would overwrite the nChainTx value of the submitted snapshot block with a fake value based on the previous block, so the (pindex->nChainTx == pindex->nTx + prev_chain_tx) check would later fail on the first block after the snapshot. This test was originally posted by Martin Zumsande <mzumsande@gmail.com> in bitcoin#29370 (comment) Co-authored-by: Martin Zumsande <mzumsande@gmail.com>

ryanofsky

Rebased 345ac98 -> 9d9a745 (pr/nofake.20 -> pr/nofake.21, compare) due to conflict with #29478

ryanofsky · 2024-03-18T18:10:28Z

src/test/util/chainstate.h

-                // and with nTx > 0 (since we aren't setting the pruned flag);
-                // see CheckBlockIndex().
-                pindex->nStatus |= BLOCK_ASSUMED_VALID;
+                // Remove all data and validity flags by just setting


re: #29370 (comment)

The comment above (Reset the HAVE_DATA flags...) could be removed or at least changed to match the changed code here.

The comment above is still accurate and still describes the main thing this is trying to do. Now just other flags are reset as well. Happy to change any of the comments if you have a specific suggestion though.

Sjors · 2024-03-19T10:42:06Z

re-ACK 9d9a745

I also tried generating and loading a signet snapshot with this PR combined with #29612.

mzumsande

Tested ACK 9d9a745

I reviewed the code and tested this running with checkblockindex on signet.

maflcko · 2024-03-20T09:56:54Z

src/validation.cpp

+                    // and the transactions were not pruned (pindexFirstMissing
+                    // is null), it is a potential candidate. The check
+                    // excludes pruned blocks, because if any blocks were
+                    // pruned between pindex the current chain tip, pindex will


nit: 9b97d5b: pindex *and* the current

src/validation.cpp

maflcko

ACK 9d9a745 🎯

Show signature

Signature:

untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
trusted comment: ACK 9d9a7458a2570f7db56ab626b22010591089c312 🎯
m+XWEuj+TMxTAm2vqwQ2KlIlqiGQf/ADm9CzC8dphRCYKKQo4VacrJ9sy4MIZjiOABr94CzwXsoVSmSPlrElBw==

maflcko · 2024-03-20T11:35:41Z

test/functional/feature_assumeutxo.py

+        # setting and testing nChainTx values, and it exposed previous bugs.
+        snapshot_hash = n0.getblockhash(SNAPSHOT_BASE_HEIGHT)
+        snapshot_block = n0.getblock(snapshot_hash, 0)
+        n1.submitblock(snapshot_block)


nit in ef174e9:

I wonder if it would be better to randomly either submit or not submit the block, because the test should be passing in both cases.

If a crash were to happen, it would be intermittent. However, it would still be deterministic, because the randomness seed is printed/logged.

achow101 · 2024-03-20T16:49:08Z

ACK 9d9a745

fjahr · 2024-03-20T19:11:15Z

Post merge utACK 9d9a745

…nected block Use Assume macro as suggested bitcoin#29370 (comment)

The checks are changing slightly in the next commit, so try to explains the ones that exist to avoid confusion (bitcoin#29370 (comment))

Add a test for a CheckBlockIndex crash that would happen before previous "assumeutxo: Get rid of faked nTx and nChainTx values" commit. The crash was an assert failure in the (pindex->nChainTx == pindex->nTx + prev_chain_tx) check that would previously happen if the snapshot block was submitted after loading the snapshot and downloading a few blocks after the snapshot. In that case ReceivedBlockTransactions() previously would overwrite the nChainTx value of the submitted snapshot block with a fake value based on the previous block, so the (pindex->nChainTx == pindex->nTx + prev_chain_tx) check would later fail on the first block after the snapshot. This test was originally posted by Martin Zumsande <mzumsande@gmail.com> in bitcoin#29370 (comment) Co-authored-by: Martin Zumsande <mzumsande@gmail.com>

…nected block Use Assume macro as suggested bitcoin/bitcoin#29370 (comment)

The checks are changing slightly in the next commit, so try to explains the ones that exist to avoid confusion (bitcoin/bitcoin#29370 (comment))

Add a test for a CheckBlockIndex crash that would happen before previous "assumeutxo: Get rid of faked nTx and nChainTx values" commit. The crash was an assert failure in the (pindex->nChainTx == pindex->nTx + prev_chain_tx) check that would previously happen if the snapshot block was submitted after loading the snapshot and downloading a few blocks after the snapshot. In that case ReceivedBlockTransactions() previously would overwrite the nChainTx value of the submitted snapshot block with a fake value based on the previous block, so the (pindex->nChainTx == pindex->nTx + prev_chain_tx) check would later fail on the first block after the snapshot. This test was originally posted by Martin Zumsande <mzumsande@gmail.com> in bitcoin/bitcoin#29370 (comment) Co-authored-by: Martin Zumsande <mzumsande@gmail.com>

ryanofsky mentioned this pull request Feb 2, 2024

assumeutxo: nTx and nChainTx in RPC results are off #29328

Open

DrahtBot added the CI failed label Feb 2, 2024

ryanofsky force-pushed the pr/nofake branch from 4c70d99 to a3228f0 Compare February 2, 2024 18:36

This was referenced Feb 2, 2024

doc: Assert that assumed-valid blocks are not fully valid in CheckBlockIndex() #29355

Closed

refactor: Remove nChainTx from consensus #29349

Closed

DrahtBot mentioned this pull request Feb 2, 2024

redeclare nChainTx to use uint64_t #29331

Closed

DrahtBot removed the CI failed label Feb 2, 2024

maflcko approved these changes Feb 5, 2024

View reviewed changes

maflcko reviewed Feb 5, 2024

View reviewed changes

src/validation.cpp Show resolved Hide resolved

ryanofsky commented Feb 5, 2024

View reviewed changes

ryanofsky force-pushed the pr/nofake branch from a3228f0 to 3bdbc3f Compare February 5, 2024 22:14

ryanofsky commented Feb 5, 2024

View reviewed changes

ryanofsky force-pushed the pr/nofake branch from 3bdbc3f to 603a92c Compare February 5, 2024 22:43

DrahtBot added the CI failed label Feb 5, 2024

DrahtBot mentioned this pull request Feb 6, 2024

validation: improve performance of CheckBlockIndex #28339

Open

mzumsande reviewed Feb 6, 2024

View reviewed changes

src/validation.cpp Show resolved Hide resolved

maflcko reviewed Feb 6, 2024

View reviewed changes

test/functional/feature_assumeutxo.py Outdated Show resolved Hide resolved

ryanofsky mentioned this pull request Feb 6, 2024

Update nChainTx to 64bit type #29258

Open

ryanofsky force-pushed the pr/nofake branch from 603a92c to 3f0b860 Compare February 7, 2024 01:03

DrahtBot removed the CI failed label Feb 7, 2024

ryanofsky force-pushed the pr/nofake branch from 3f0b860 to 2f4e7e8 Compare February 7, 2024 19:08

ryanofsky mentioned this pull request Feb 7, 2024

assumeutxo: nTx and nChainTx violations in CheckBlockIndex #29261

Closed

ryanofsky and others added 7 commits March 18, 2024 11:28

validation: Check GuessVerificationProgress is not called with discon…

0fd915e

…nected block Use Assume macro as suggested bitcoin#29370 (comment)

doc: Improve comments describing setBlockIndexCandidates checks

9b97d5b

The checks are changing slightly in the next commit, so try to explains the ones that exist to avoid confusion (bitcoin#29370 (comment))

assumeutxo: Remove BLOCK_ASSUMED_VALID flag

9d9a745

Flag adds complexity and is not currently used for anything.

ryanofsky force-pushed the pr/nofake branch from 345ac98 to 9d9a745 Compare March 18, 2024 18:00

ryanofsky added a commit to ryanofsky/bitcoin that referenced this pull request Mar 18, 2024

validation: Check GuessVerificationProgress is not called with discon…

302b878

…nected block Use Assume macro as suggested bitcoin#29370 (comment)

ryanofsky commented Mar 18, 2024

View reviewed changes

DrahtBot removed the Needs rebase label Mar 18, 2024

DrahtBot mentioned this pull request Mar 19, 2024

chainparams: Change nChainTx type to uint64_t #29656

Open

DrahtBot requested a review from fjahr March 19, 2024 10:42

mzumsande reviewed Mar 19, 2024

View reviewed changes

maflcko reviewed Mar 20, 2024

View reviewed changes

DrahtBot requested a review from maflcko March 20, 2024 10:07

maflcko approved these changes Mar 20, 2024

View reviewed changes

achow101 self-assigned this Mar 20, 2024

achow101 merged commit b50554b into bitcoin:master Mar 20, 2024
16 checks passed

mzumsande pushed a commit to mzumsande/bitcoin that referenced this pull request Apr 1, 2024

validation: Check GuessVerificationProgress is not called with discon…

be8406d

…nected block Use Assume macro as suggested bitcoin#29370 (comment)

janus pushed a commit to BitgesellOfficial/bitgesell that referenced this pull request Apr 6, 2024

validation: Check GuessVerificationProgress is not called with discon…

877976a

…nected block Use Assume macro as suggested bitcoin/bitcoin#29370 (comment)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

assumeutxo: Get rid of faked nTx and nChainTx values #29370

assumeutxo: Get rid of faked nTx and nChainTx values #29370

ryanofsky commented Feb 2, 2024 •

edited

DrahtBot commented Feb 2, 2024 •

edited

maflcko commented Feb 2, 2024

DrahtBot commented Feb 2, 2024

ryanofsky commented Feb 2, 2024

maflcko left a comment

maflcko commented Feb 5, 2024

ryanofsky left a comment

ryanofsky left a comment •

edited

maflcko commented Feb 6, 2024

ryanofsky commented Feb 7, 2024

maflcko commented Feb 7, 2024

ryanofsky left a comment

ryanofsky Mar 18, 2024

Sjors commented Mar 19, 2024

mzumsande left a comment

maflcko Mar 20, 2024

maflcko left a comment

maflcko Mar 20, 2024

achow101 commented Mar 20, 2024

fjahr commented Mar 20, 2024

assumeutxo: Get rid of faked nTx and nChainTx values #29370

assumeutxo: Get rid of faked nTx and nChainTx values #29370

Conversation

ryanofsky commented Feb 2, 2024 • edited

DrahtBot commented Feb 2, 2024 • edited

Code Coverage

Reviews

Conflicts

maflcko commented Feb 2, 2024

DrahtBot commented Feb 2, 2024

ryanofsky commented Feb 2, 2024

maflcko left a comment

Choose a reason for hiding this comment

maflcko commented Feb 5, 2024

ryanofsky left a comment

Choose a reason for hiding this comment

ryanofsky left a comment • edited

Choose a reason for hiding this comment

maflcko commented Feb 6, 2024

ryanofsky commented Feb 7, 2024

maflcko commented Feb 7, 2024

ryanofsky left a comment

Choose a reason for hiding this comment

ryanofsky Mar 18, 2024

Choose a reason for hiding this comment

Sjors commented Mar 19, 2024

mzumsande left a comment

Choose a reason for hiding this comment

maflcko Mar 20, 2024

Choose a reason for hiding this comment

maflcko left a comment

Choose a reason for hiding this comment

maflcko Mar 20, 2024

Choose a reason for hiding this comment

achow101 commented Mar 20, 2024

fjahr commented Mar 20, 2024

ryanofsky commented Feb 2, 2024 •

edited

DrahtBot commented Feb 2, 2024 •

edited

ryanofsky left a comment •

edited