Slow getTxOutProof RPC

[monsterbitar]

As part of building an indexing server for certain transaction types I need to fetch merkle proofs. With a reasonably strong machines (AMD FX(tm)-8120) but a relatively slow storage medium i find the RPC requests for merkle proofs to be much slower than I'd expect.

My measurements across a large swath of transactions (where I request proof for only a single transaction at a time) ranges from 300ms to 600ms.

From looking at the code in https://github.com/BitcoinUnlimited/BitcoinUnlimited/blob/release/src/rpc/rawtransaction.cpp I can't see anything obvious that would cause this, except maybe a "LOCK(cs_main);" which only happens if you do not specify which block.

When I do specify a block the performance increases dramatically.

Does this mean that BU either needs a faster way to lookup TXID <-> BLOCK or is the problem that it's locking something despite this being a read-only process? (or something else entirely?)

[monsterbitar]

I also don't see why you're checking for coin spent status: !coin->IsSpent()

The inclusion proof shouldn't change based on the UTXO status, right?

[Greg-Griffith]

@monsterbitar Hi,
CoinAccessor locks the cs_utxo. IIRC cs_utxo and cs_main are the two most heavily used locks and are more than likely the cause of your slowdown.
The CoinAccessor isnt guaranteed to find a coin. In the situation where it doesnt find anything it will return the emptyCoin object. emptyCoin has the properties of isSpent() == true, height == 0, and fCoinbase == false. the isSpent() and height > 0 checks are simply testing for emptyCoin.

EDIT: to clarify, what i mean by "The CoinAccessor isnt guaranteed to find a coin." if you give it a txid where all outputs are spent or the txid simply doesnt exist it will return emptyCoin

[monsterbitar]

Thanks for clarifying why it checks isSpent(). Is there some other way to get a block pointer that doesn't require such a heavy lock?

For now I've worked around it in my application by always submitting the blockhash to the function and the performance with the block hash available is about two orders of magniture better.

[Greg-Griffith]

i dont think so... it seems already do it in a pretty logical order. if no block hash supplied then check the utxo set because its in memory, if its not in there use GetTransaction whcih will need to pull data from disk. But GetTransaction doesnt do anything the CoinAccessor wont already have done unless you are running a txindex. I am unsure how else you would find the block that contains a transaction without using one of those methods.

[monsterbitar]

In my case I am running with txindex=1, but either way since I already have the blockhash from a previous call I can just supply it.

Would it be faster to skip the lock and go directly for disk, but only if txindex is enabled?

[gandrewstone]

@monsterbitar What is your processing time for the data returned from getrawblocktransactions? If it is not insignificant, you could fire off the next getrawblocktransactions while processing the current one. This would parallelize your work with our disk access -- so you may effectively be able to do all your processing in 0 time.

[monsterbitar]

@gandrewstone my processing time was about a 1/10th of the cost, but was entirely synchronous. I thought about why I did it like this and my take is that I need the blockchain order validated and can't do the blocks in parallel with my current structures.

I could parallelize the processing of the transactions though, so I did that this morning. Thanks for the suggestion, I'll take some time to think about if I can restructure to parallelize the blocks as well.