Transactions stuck in mempool

[byte-bandit]

Running on cometbft 0.37.2.

I'm seeing multiple transactions in the mempool that appear to be stuck. There are no logs about transactions failing to be included in a block. From what I can tell, the transactions succeeded in basic validation and ante handlers when being added, but now they fail to be included into a block.

As far as I can tell, there is nothing unusual about the TX contents.

Is there any way to investigate the mempool further and build an understanding of why those transactions fail to be included in a block?

[thanethomson]

Have you managed to reliably reproduce this issue? If so, could you please provide more info re: your configuration, transactions, etc.? If not it's really hard for us to say why that's happening.

If you want to investigate the mempool further, what sorts of insights would help you in reproducing and debugging the problem? (Any specific metrics, logs, etc. that could help?)

[byte-bandit]

Cheers for getting back to me! We can reliably reproduce this on our public test net, but there's quite a bit of customisation involved. From what we can tell, this behaviour started after creating our first custom Ante handler, which performs some stateful checks. It appears that for some transactions, the checks pass when transactions are entering the mempool, but then fail to be valid again for inclusion in a block indefinitely. Any log output as to WHEN a transaction is being attempted to be included as well as WHY it fails to be included would help.

That being said, we're currently upgrading our CosmosSDK version to 0.50 which will bring in CometBFT 0.38 as well. We're going to focus on this upgrade for now. In case the behaviour cannot be resolved by upgrading, we'll come back to this.

[thanethomson]

Yeah please keep us updated!

In the meantime we'll look into ensuring we log when we're attempting to include a transaction in a block but ends up not being included. cc @hvanz @lasarojc @sergio-mena