Possibility to create the filters on the client side

[csiki2]

The client should be able to create the block filters on its own if requested by the user.

Description

Currently the client gets the filters from the backend, this is a fast and convenient solution during the first start since downloading the whole blockchain would take a lot of time and space.
However if the user requests (through settings or at the start), the client should be able to generate the filters on its own without the help of the backend.

Reasoning

There would be multiple advantages of this feature:

• Storage: even though the client would need to download the blockchain once, it wouldn't need to store it, which is unpractical in most of the cases
• Privacy: if an user is very serious about the privacy, they have the option to do so for the filtering
• Continuous filter generation: even with starting to download filters from the backend, later the client would be able to generate the filters on its own (if the available net is sufficient ofc.), the estimated download size would be 200-400MB daily
• It would decrease the load of the backend
• It would be a huge step toward to make the WalletWasabi to be able to run without the backend
• Later, by completely removing the mandatory backend requirement would make possible to ask bitcoin.org to reevaluate the validation part of their comparison table

Designs

Probably the client should store where the filter comes from along the filter itself.
If I understand correctly for a filter generation we need the previous filter. We might be able add "filter points" along with the software to start generating new filters from (the size increase of the install package would be negligible, let's say every 1000s filter).
The rational behind this that downloading the beginning of the blockchain has no real benefit for a few years old wallet (even if it would give positive results, it would be false positive) and would make possible to begin generating recent filters right at the start.

[molnard]

• It would be a huge step toward to make the WalletWasabi to be able to run without the backend

I think this would be a powerful feature. You can use Wasabi 100% without the backend (except CJ). Especially nowadays...

Relevant post from twitter: https://twitter.com/PavelTheCoder/status/1784779670363181442

• Later, by completely removing the mandatory backend requirement would make possible to ask bitcoin.org to reevaluate the validation part of their comparison table

Yes.

[kiminuo]

@csiki2 Do you plan to implement this? Or will @molnard implement this?

[csiki2]

I planned to do it, but when I got to @lontivero that I would start it, he stated to be a bad idea.
So I don't know what to think now, we will need to discuss this on the next meeting.

[kiminuo]

I planned to do it, but when I got to @lontivero that I would start it, he stated to be a bad idea.

For what reason?

[csiki2]

According to him the whole purpose of the filter is to not download the block.
I think, you might not loaded all your wallets at the time so it would drop the block and the biggest advantage of the filter is the small storage size for later checks.

[kiminuo]

Oh yes, right, I'm an idiot :-D. So it's like this:

1. You want to have filters because they are much smaller than blocks.
2. Basically you have a set of scriptPubKeys (~ your coins) and you check all scriptPubKeys against all block filters (more or less) and if there is a match then you need to download the corresponding block.

-> It would be more useful to have a decentralized mechanism for distributing of block filters.

[csiki2]

Tricky question. Doing decentralized, but keeping the privacy and the ensure that the filters are valid.

[kiminuo]

I'm not sure why the question is tricky.

You have blockchain blocks. Full nodes download all blocks. Lightweight wallets download only certain blocks and they are not keen to let to know people what blocks they need for privacy reasons.

Downloading all block filters seem similar to downloading all bitcoin blocks. If downloaded over Tor, you just tell the world that "hey, there is a person who wants block filters, so it might be a wasabi user but it can be anyone in the world" (feels ok).

The remaining part is block filters' validity and that is AFAIK described in https://github.com/bitcoin/bips/blob/master/bip-0157.mediawiki#filter-headers. Also block filters are determistict so as the BIP mentions: A light client can then download an entire block if the filter matches the data it is watching for. So if one downloads a batch of filters and then downloads a block in that range to confirm the data is valid, one has a good chance to receive valid data.

[csiki2]

So, do we have known (free) filter providers to turn to?

[turbolay]

What to do with synchronization is for sure one of the most challenging question we have to answer, and rushing towards a solution would be unwise. Also, there will never be a perfect solution and it will always be about tradeoffs, so probably we need to implement a multi-faceted solution users can choose from.

Generating filters on client side sounds like a really appealing solution, but major constraints are at play here. Creating the filters is extremely time consuming. A full index creation is even an order of magnitude longer than syncing a full node. So you can rightfully say: "Yes but we could start from a checkpoint". Then what about starting from a checkpoint to sync a full node? The downside is the same. So next argument is "Yeah but we would save storage". Then what about using a pruned node? A pruned node has only few extra downside than what you're suggesting to do, but it has more upside.

Something to think about: not opening Wasabi for 24h would mean 144 blocks to download and create filters from. What about UX here? This operation will probably last for 1h. Even with major optimization like parallelization of the downloads, as we will now be able to perform a full parallelization, it will be time consuming, CPU and Memory intensive. It is already hard for the bulk of users to accept to wait a bit more for their privacy instead of sending their xpub to an electrum server, it would be an order of magnitude worse if you always need to wait 10-100x more after a good night of sleep. And this regardless of the size of your wallet, whilst currently a small wallet synchronizes almost instantly in Wasabi.

About the checkpoints: In order to keep an ok-ish onboarding experience, we would need snapshots really often, maybe once a day at least. Well if we have a mechanism for snapshotting that often the filters, why don't push a bit more and serve the filters as we are doing now?

Where I'm going with this: I, and I believe most of us, would like to guarantee 2 things:

• Wasabi stays (one of?) the best privacy wallet.
• Wasabi stays beginner friendly and keeps providing an acceptable UX.

With a synchronization solution solely based on the creation of filters client side, the 2nd point is not possible, as all users will have to perform hours of initial synchronization, or tens of minutes each time they launch their wallet, which makes it too easy and incentivizes too much to switch to a wallet with a xpub to server bases synchronization, which would take only few seconds.

I know that I'm not suggesting any solution here. And creating filters from client side is not to be discarded at all either. I just want to make a point that we will probably not be able to find a single solution that answers to all the constraints and user requirements that we have.

[csiki2]

I think, all of us can agree that there will be no holy grail here. If it was easy, the other wallets would have that solution already (at least the leading ones).
On UX side, I can only think that we shouldn't block the possibility to proceed (if the user asks for it for example), showing that the wallet might be inaccurate (ofc. this can lead to complications at sending BTC).