-
Notifications
You must be signed in to change notification settings - Fork 35.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
wallet: fix unrelated parent conflict doesn't cause child tx to be marked as conflict #29680
base: master
Are you sure you want to change the base?
Conversation
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers. Code CoverageFor detailed information about the code coverage, see the test coverage report. ReviewsSee the guideline for information on the review process.
If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update. ConflictsReviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first. |
🚧 At least one of the CI tasks failed. Make sure to run all tests locally, according to the Possibly this is due to a silent merge conflict (the changes in this pull request being Leave a comment here, if you need help tracking down a confusing failure. |
abaa8ca
to
4853bb5
Compare
ba8c831
to
532d25f
Compare
Draft until I've gotten some more feedback on the approach. |
cc @achow101? |
I'm actually wondering now if it would be sufficient to just have the |
@achow101 I think this makes sense. Once we add conflicting block hash for conflicts then we can safely mark wallet tx as conflicted which should solve the issue. What would we still need the replacement txid for? EDIT When you say "replacement txid", are you referring to the txid of the tx causing the wallet tx to kicked out? if so, I believe the current fix implemented here does exactly what you're describing, See 635e2e4 |
Yes, we need the replacement txid for instances where the tx is removed by replacement rather than a block conflict.
It appears to also be watching for replacements of those replacements too, and I think that is unnecessary. |
Thanks @achow101. Yes, I added that when I realized that if the replacement is also replaced then the check in the |
532d25f
to
47750d3
Compare
@achow101 I took this out because the new replacement is not guaranteed to conflict with the original wallet transaction Added I had to use Curious to see what others think. EDIT This PR now modifies Now marking this PR as ready for review. |
47750d3
to
8ee9629
Compare
🚧 At least one of the CI tasks failed. Make sure to run all tests locally, according to the Possibly this is due to a silent merge conflict (the changes in this pull request being Leave a comment here, if you need help tracking down a confusing failure. |
8ee9629
to
e868b2d
Compare
Putting in draft while I fix falling test |
e868b2d
to
e004ecf
Compare
faea0a5
to
6bd69c3
Compare
6bd69c3
to
ea66044
Compare
Concept ACK Seems like a reasonable approach to the problem described in #29435 , will dig in more. Using a variant as a replacement for the Enum seems a bit odd at first glance and requires a lot of duplicate code for each struct. Perhaps another approach could be a class for EDIT: this approach might also make it easier to do the first commit as a scripted-diff |
Thanks for the review @josibake. Will this approach still require the definition of structs to hold each data for each reason? |
I don't think so: only the Conflict and Replace reasons need extra data, and from what I understand it can only be one reason at any given time. Given that, you could have a field on the new class for removal data which is a std::variant of the possible data types. For example, |
@josibake Makes sense but I'm skeptical about how this affects the removal reason usage. For example, you can just define |
The benefits I see are we can keep the existing Enum and code for converting the Enum to a string. This would also allow updating the function signatures with a scripted-diff, which makes larger refactors like this easier to verify as a reviewer. It also feels like a more natural fit to me and my gut feeling is that having a class to encapsulate the removal reasons and necessary data will be more maintainable and extensible going forward. |
@josibake Makes sense. I'll create a RemovalReason class |
The drawback of a separate data field is that there is no longer a guarantee that data accompanying the reason is always present when the reason is set, and always absent when it is not set. So it is possible for code to only partially initialize the RemovalReason class or initialize it in an inconsistent state. In general I like the idea of replacing enums with variants for more safety, and to be able to express rules about state in type definitions. But variants in c++ are more awkward than sum types in other languages, and I did not look into this specific case, so maybe the tradeoffs in this case are not worth it. |
Seems easily addressed with a constructor, no? Something like: class RemovedReason {
public:
MemPoolRemovalReason m_reason;
std::variant<std::monostate, CTxReference, BlockData> m_extra_data;
// Constructor for reasons that don't require extra data
RemovedReason(MemPoolRemovalReason r) : reason(r) {
if (requiresExtraData(r)) {
throw std::invalid_argument("reason X requires data Y etc");
}
}
// Constructor needing CTxRef
RemovedReason(RemovalReason r, const CTxRef& data) : reason(r), extra_data(data) {
if (!IsA(r)) {
throw std::invalid_argument("CTxRef is required for reason A, got bla");
}
}
// Constructor needing BlockData
RemovedReason(RemovalReason r, const BlockData& data) : reason(r), extra_data(data) {
if (!IsB(r)) {
throw std::invalid_argument("BlockData is required for reason B, got bla");
}
} Maybe this is starting to be more complicated than just having a struct per each reason? But I'd still argue this is a better approach in that it keeps all the logic for mempool removal reasons in one place and avoids duplicating code on each struct. |
Same thing I was thinking. Using the class right now looks like it will make things more complicated. Maybe we should leave the class for a future change where it becomes necessary? @josibake @ryanofsky |
Yes, but those are manual constraints that you are writing by hand rather than automatic constraints expressed implicitly in the data definition. Depending on the constructors it may also only provide runtime checking rather than compile-time checking like in your example. And if the struct is mutable could allow invalid representations of state after construction. I don't know what is best in this particular case, I would just stand up for: struct MyState1 { int data; };
struct MyState2 { bool flag; };
struct MyState3 {};
using MyState = std::variant<MyState1, MyState2, MyState3>; as a good alternative to: enum class MyState {
STATE1,
STATE2,
STATE3,
};
class MyData {
MyState m_state,
// ... more data and methods...
}; in many cases.
I'm not sure the answer to this, but it is probably worth experimenting and choosing the approach that seems simplest. |
Fair point, in that bugs could be introduced by someone not writing these pre-checks correctly / efficiently. I'll admit I'm not fully convinced that the struct per state approach isn't going to be harder to maintain / extend in the future, but given that I haven't convinced you guys on that point and you have convinced me there is an advantage per the struct per state approach, I'll retract my suggestion we change it 😄 |
This allows the mempool to send additional data with TransactionRemovedFromMempool event. Now, we can send conflicting_block_hash and conflicting_block_height for Conflicts and replacement_tx for Replacements.
Detect replacement of wallet txs and wait for confirmation of replacement tx before marking wallet tx as conflicted
ea66044
to
7da8f98
Compare
test/functional/wallet_conflicts.py
Outdated
assert_equal(current_wallet.gettransaction(child_txid)["confirmations"], 0) | ||
|
||
# Make a conflict spending parent | ||
conflict_psbt = def_wallet.walletcreatefundedpsbt(inputs=[gp_utxo], outputs=[{def_wallet.getnewaddress(): 2}], fee_rate=parent_feerate*3)["psbt"] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
node0 2024-05-20T11:47:13.485435Z [httpworker.2] [rpc/request.cpp:222] [parse] [rpc] ThreadRPCServer method=walletcreatefundedpsbt user=__cookie__
test 2024-05-20T11:47:13.486000Z TestFramework (ERROR): JSONRPC error
Traceback (most recent call last):
File "/ci_container_base/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/test_framework/test_framework.py", line 132, in main
self.run_test()
File "/ci_container_base/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/wallet_conflicts.py", line 41, in run_test
self.test_unknown_parent_conflict()
File "/ci_container_base/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/wallet_conflicts.py", line 486, in test_unknown_parent_conflict
conflict_psbt = def_wallet.walletcreatefundedpsbt(inputs=[gp_utxo], outputs=[{def_wallet.getnewaddress(): 2}], fee_rate=parent_feerate*3)["psbt"]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/ci_container_base/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/test_framework/coverage.py", line 50, in __call__
return_val = self.auth_service_proxy_instance.__call__(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/ci_container_base/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/test_framework/authproxy.py", line 129, in __call__
raise JSONRPCException(response['error'], status)
test_framework.authproxy.JSONRPCException: Invalid amount (-3)
Watch for wallet transaction conflicts triggered by adding conflicting blocks
7da8f98
to
f7ec03e
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Still reviewing the later commits, but had some initial feedback/questions for the first commit.
|
||
/** Reason why a transaction was removed from the mempool, | ||
/** Reasons why a transaction was removed from the mempool, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
in 7debac0 ("Change MemPoolRemovalReason to variant type"):
nit: I think it's more correct to leave this as "Reason." "Reasons" implies a single transaction can have multiple reasons for being removed at the same time.
@@ -19,7 +19,7 @@ | |||
#include <unordered_map> | |||
#include <utility> | |||
|
|||
std::string RemovalReasonToString(const MemPoolRemovalReason& r) noexcept; | |||
// std::string RemovalReasonToString(const MemPoolRemovalReason& r) noexcept; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
in 7debac0 ("Change MemPoolRemovalReason to variant type"):
Can remove this line.
class CValidationInterface; | ||
class CScheduler; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
in 7debac0 ("Change MemPoolRemovalReason to variant type"):
AFAICT, these are unused (I was able to compile this commit fine without them). Maybe leftover from a different approach?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same here. I'll take them out
@@ -56,7 +56,6 @@ class CKeyID; | |||
class CPubKey; | |||
class Coin; | |||
class SigningProvider; | |||
enum class MemPoolRemovalReason; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
in 7debac0 ("Change MemPoolRemovalReason to variant type"):
Don't we also need to #include kernel/mempool_removal_reasons.h
, for Include What You Use (IWYU)? I'll admit, I'm totally sure what our conventions are on this. cc @TheCharlatan or @maflcko
@@ -3057,7 +3057,7 @@ bool Chainstate::ConnectTip(BlockValidationState& state, CBlockIndex* pindexNew, | |||
Ticks<MillisecondsDouble>(time_chainstate) / num_blocks_total); | |||
// Remove conflicting transactions from the mempool.; | |||
if (m_mempool) { | |||
m_mempool->removeForBlock(blockConnecting.vtx, pindexNew->nHeight); | |||
m_mempool->removeForBlock(blockConnecting.vtx, pthisBlock->GetHash(), pindexNew->nHeight); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
in 7debac0 ("Change MemPoolRemovalReason to variant type"):
Was a bit surprised that pthisBlock->GetHash()
isn't returning a member variable and instead is calculating the hash each time its called. Worth mentioning we are adding an extra hash to ConnectTip
. Probably negligible but wanted to mention it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@josibake I looked around a bit and there might be something here. GetHash()
is called before this point
- when we read the block from disk
bitcoin/src/node/blockstorage.cpp
Line 1092 in e163d86
if (block.GetHash() != index.GetBlockHash()) { - when
ActivateBestChain
is calledLine 3354 in e163d86
if (!ActivateBestChainStep(state, pindexMostWork, pblock && pblock->GetHash() == pindexMostWork->GetBlockHash() ? pblock : nullBlockPtr, fInvalidFound, connectTrace)) {
There might be some gain in caching GetHash() but I think that has to be addressed on its own. I'll have to measure the runtimes and see its worth a PR.
I did also discover that pindexNew
which is a CBlockIndex
does store the blockhash
, see
Line 244 in e163d86
uint256 GetBlockHash() const |
@@ -15,7 +16,7 @@ | |||
|
|||
BOOST_FIXTURE_TEST_SUITE(mempool_tests, TestingSetup) | |||
|
|||
static constexpr auto REMOVAL_REASON_DUMMY = MemPoolRemovalReason::REPLACED; | |||
static const auto REMOVAL_REASON_DUMMY = ReplacedReason(nullptr); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
in 7debac0 ("Change MemPoolRemovalReason to variant type"):
I think it would be better to pass a dummy CTxRef
here instead of nullptr.
explicit ConflictReason(const uint256& conflicting_block_hash, int conflicting_block_height) : conflicting_block_hash(conflicting_block_hash), conflicting_block_height(conflicting_block_height) {} | ||
std::string toString() const noexcept { | ||
return "conflict"; | ||
} | ||
}; | ||
|
||
struct ReplacedReason { | ||
CTransactionRef replacement_tx; | ||
|
||
explicit ReplacedReason(const CTransactionRef replacement_tx) : replacement_tx(replacement_tx) {} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
in 7debac0 ("Change MemPoolRemovalReason to variant type"):
Any reason why conflicting_block_hash
is passed by reference but replacement_tx
isn't?
Would also be nice if we prevented these objects from be constructed with nullptrs
. I'm not sure if we have a convention in our codebase around this or other examples to point to, but would be worth looking into.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Any reason why
conflicting_block_hash
is passed by reference butreplacement_tx
isn't?
CTransactionRef
is already a pointer
bitcoin/src/primitives/transaction.h
Lines 423 to 424 in e163d86
typedef std::shared_ptr<const CTransaction> CTransactionRef; | |
template <typename Tx> static inline CTransactionRef MakeTransactionRef(Tx&& txIn) { return std::make_shared<const CTransaction>(std::forward<Tx>(txIn)); } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would also be nice if we prevented these objects from be constructed with
nullptrs
. I'm not sure if we have a convention in our codebase around this or other examples to point to, but would be worth looking into.
It looks like I may be able to do this by deleting the constructor
// Deleted constructor
ReplacedReason(std::nullptr_t) = delete;
I'll test it
uint256 conflicting_block_hash; | ||
unsigned int conflicting_block_height; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
in 7debac0 ("Change MemPoolRemovalReason to variant type"):
I haven't finished reviewing the later commits yet, but seems odd to pass both conflicting_block_hash
and conflicting_block_height
. Seems like we should be able to only use conflicting_block_hash
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The TxStateBlockConflicted
requires both that's why I added the both of them
🐙 This pull request conflicts with the target branch and needs rebase. |
This PR implements a fix for the issue described in #29435.
The problem is that the wallet is unable to abandon transactions that have unrelated parent conflicts. The solution implemented here, augments the mempool transaction
REPLACED
signal with the double-spending transaction which the wallet stores and watches for in Block notifications. A map is added to the wallet to track conflicting tx ids and their child transactions. The entry is erased when the double-spending tx is removed from MemPool.