BTCPay FastSync Generator and Validator

[Kukks]

Sorry @pavlenex , I'm ditching the template for this one :)

With the latest drama around potential widespread adoption of SPVs, I feel like it's about time we bring the FastSync system that @NicolasDorier built to a wider audience, just as @rockstardev has been wanting for a while.

FastSync currently allows users to download the utxo snapshot uploaded by maintainers of the btcpayserver-docker. A user may also set an environment variable that links to a utxo set download link.

I propose we add a UI in BtcPay to generate, validate and provide these utxo sets with the underlying full bitcoin node. This allows us to get rid of the direct dependency and trust of btcpay developers and instead make it slightly more independent. It also allows independent users to help spread and verify utxo snapshots while verifying them.

[rockstardev]

Such great idea... I actually wanted to dedicate part of my talk to getting people to sign these sets.

If there was UI for generation of UTXO set it would save some time. We just need to make sure people are generate snapshots at the same time... thinking about it - I guess this page you sketched can also include output of bitcoin-cli.sh gettxoutsetinfo.

[Kukks]

Yeah, we can output everything for clarity to the admin executing creation and validation. Maybe even display more data for end user but it shouldn't be a lot of effort in any case.

How big is the utxo snapshot? I am wondering if it is too big to ask btcpay hosts to keep on their server

[rockstardev]

@Kukks latest one seems to be 3.3GB:

http://utxosets.blob.core.windows.net/public/utxo-snapshot-bitcoin-mainnet-565305.tar

Here is the reference: https://github.com/btcpayserver/btcpayserver-docker/blob/master/contrib/FastSync/load-utxo-set.sh#L26

P.S. I need to update my signature, good guy @junderw added his sig in the meantime.

[Kukks]

Yeah so at that cost it may be hard when you consider the penny-pinching people do to save 12 usd a year on storage costs.

[Kukks]

Ok, so I need to work on #239 first just so hosts can integrate external storage providers. I've got some code ready for a few different ones like amazon, google & azure and local that I can integrate

[NicolasDorier]

This is awesome such an idea. So if I understand, you will be able to ask to other btcpay nodes if they validated the snapshot? And provide in BTCPay the tool to validate yourself the snapshot?

But this is super hard, because you need to automate popping up a new full node, downloading the snapshot, synching it and finally comparing the utxo hashes.

[rockstardev]

You were not kidding when you said gettxoutsetinfo is expensive operation @NicolasDorier... 4 minutes initially on my beast of machine, then 1 minute every consecutive run.

Also, I've tried exposing results in separate page in Server Settings - the question is how we execute to bitcoin-cli... because it's different if setup is done through Docker or manual. I'll keep working on "Trusty" part that's independent of BtcPayServer (tweeting out gettxoutsetinfo) and we'll go further on how to expose info in BtcPayServer during one of our future meets.

[Kukks]

I'm still wondering how twitter bot would work. Wouldn't it mean that there is a < 10minute time window to verify the utxoset?

[junderw]

is this guaranteed to be bit for bit deterministic on the same block?

[NicolasDorier]

@Kukks yes, this mean that we should run automatically the gettxoutsetinfo prevemptively on all BTCPay installs at the same time, so they can verify at posteriori...

@junderw yes, it is guaranteed.

[junderw]

So should I sign the utxosetinfo hash or should I sign the tar of the snapshot?

Last time I signed the tar file, but this issue is an idea for people sharing the utxosetinfo hash?

Cool.

[NicolasDorier]

@junderw yeah FastSync is about signing the tar, I think this can't be prevented.

But this issue is about something different. This is about making people monitoring their UTXOSet hash periodically. A bad utxoset snapshot would be quickly spotted.

[Kukks]

I think the goal is of this is best described as:

• Decentralizing FastSync snapshots through btcpay deployments to reduce dependency on one repo.
• Decentralized verification of snapshots by pairing through public announcements

[rockstardev]

For Twitter bot - I am actually launching it today; will test it for a bit and then when it's polished send you link to repository so that you can run it @NicolasDorier and @Kukks if you want.

[rockstardev]

OK, I got the first pass through:

https://twitter.com/TxoutSetInfo

Now, moving onto scheduler... after I have it ready then it's publishing source code and exposing API so that others can send info from their own bitcoind.

I plan to have listening service that aggregates received data and then tweets out consensus + linked tweet mentioning everyone who participated in consensus.

[NicolasDorier]

@rockstardev I think you should do it at NBXplorer level.

[NicolasDorier]

Just making an API which can be used if --track-txoutsetinfo is available, which would just get it, save in DB and expose it.

[rockstardev]

Yeah, I planned on trying to do it on NBXplorer level; I'm just worrying if it's too resource intensive and how I would handle running command on every block - what should I do when new blocks comes in while I'm calculating and all that.

Otherwise - update on progress - I deployed service that Tweets out data to LunaNode so that it's accessible for others also, and I have Bitcoin Core 0.17.1 running on separate machine, triggering collection every 10 minutes.

I'll demo FastSync today during my talk and will keep dogfooding - plan to run another Bitcoin Core full node locally (different version) and aggregate published data. Will keep you all updated!

[Kukks]

We shouldn't do it on every block though, maybe every 72 blocks/ twice a day at most right? I'm still trying to wrap my head at how you want to automate it and decentralize this :D

[rockstardev]

I would do it on every block for this Twitter service because we never know when someone is using FastSync... and I don't want to keep users waiting on data.

As for running this as service on BTCPayServer host - yeah, you are right... which is why I'm mentioning those resource constraints. Anyways, we'll figure it out - let's me just survive my talk today - so much to prepare ;)

[NicolasDorier]

please don't do it that often, the goal is that people actually check. If you do too often, people don't care. Once a month should be enough. So it is a kind of "event".