Option for custom rates endpoint

[btcmaine]

I would like to provide my own endpoint for coin prices. I can put a bounty on this feature of 0.01 BTC if done within 30 days as an addon. Would be a great advanced feature for the main release as well.

For the past 3 years I have solved the issue by invoicing the specific BTC amount, but this fix makes many features of BTCPay unusable for me.

[bolatovumar]

@btcmaine you want to be able to specify your own custom endpoint which would return coin prices instead of having to pick one from the list provided by BTCPay? Do I understand you correctly?

[Zaxounette]

I can at least confirm that I understand this the same way as you Umar.

[btcmaine]

Yes, this is correct.

[bolatovumar]

@btcmaine why is it that you need custom rates? I'm trying to understand what we are trying to improve here. Is there something you are trying to do that's not possible with existing rate providers?

[btcmaine]

Bitcoin is a real commodity. My company has operated a custom indexed price and spread for many years and this is good for us, but it means we cannot use any of the included endpoints. Further, it would be best for everyone to be able to set their own because in plenty of areas around the world the price is unstable and exchanges cannot be trusted as a good baseline. The improvement is to cut dependence on exchanges and indexes that are used as providers. We have got around this for a long time by invoicing in BTC as I said before, but it locks us out of other features, like payment requests.

[NicolasDorier]

Please provide more details about the UX to do this. Where should it be configurable, what to input where

[btcmaine]

In this instance there's no need for auth and the format can be dev choosing. The point is that it's easier to create the endpoint than try and provide support for customization to btcpay. The endpoint is expected to be created specifically for the purpose of providing rates to btcpay, not to just type in some exchange that isn't supported.

[bolatovumar]

and the format can be dev choosing

@btcmaine it can't be in any format, it would have to be in some sort of pre-agreed format that BTCPay expects (there isn't one form it expects currently, we build a custom adapter for each exchange).

[btcmaine]

You're missing the point. Whoever builds could choose because it doesn't matter. The endpoint would be built to fit whatever btcpay expects in this specific configuration (which doesn't currently exist). This way anyone could easily create the endpoint to feed whatever rates they choose. Having this option in btcpay helps the project by allowing more custom use cases (like ours) without an extreme burden to develop custom btcpay integration, just create the endpoint and point btcpay to it.

[bolatovumar]

@btcmaine yeah, I get what you mean. For this to work we would need to come to an agreement as to what shape the API should supply the data in (I could come up with this). Also, would this endpoint be public or would we need to have an auth mechanism? I imagine you wouldn't want to have a public API that anyone can use.

[btcmaine]

There is no problem with BTCPay assuming it's public, there is a level of confidence given here that the user can work around it, the real hurdle to to keep the user from having to keep a forked BTCPay server going and updated.

If the user doesn't want it to be public they can host it locally or relay it locally. We use a json file for our index and get it when needed, it's hosted locally, but even if it wasn't and it were behind auth it would be easy enough to create a local copy for BTCpay to look at.

[IT4Bitcoin]

This is an underrated requested feature. We cannot use Argentinean, Cuban, Belorussia and Venezuela exchange rate properly because the lack of this. I really need to integrate yadio.io wich is the API for all countries without forex market.

[dubdam]

+1 Not being able to choose an API that provides real exchange rates makes BTCPay unusuable for countries under heavy financial repression if we'd like to denominate prices in local currency. Yadio.io API would solve this

[bolatovumar]

@IT4Bitcoin @dubdam I'm adding Yadio as a provider in this PR: #4048

[Amperstrand]

I'm also looking into this.

The use case is that you have provider that guarantees a specific exchange rate. So if you are invoicing 1 euro with BTCPayserver and received XXXXX sats and forward it immediately to that provider, you are guaranteed 1 euro.

It is important because if you use the wrong exchange rate and receive 0.99 euro or 1.01 euro, the fiat accounting won't match up. So you need to use the exchange rate which is published by that provider.

One way to approach to solve this is to make a provider for localhost. So for example http://localhost:1234/btc_usd would give you the exchange rate for USD. Then a simple python webserver could listen on localhost and be responsible for fetching the exchange rate. This way, you can tell btcpayserver to use the the localhost provider, and then just make sure you have a webserver running on the same server as btcpayserver which is responsible for supplying whatever exchange rate you want.

Another approach is to tell BTCPayserver to use coingecko, but MITM the connection. For this to work, you need to set up a webserver that emulates https://api.coingecko.com/api/v3/exchange_rates, but add api.coingecko.com to /etc/hosts so that btcpayserver goes to fetch your spoofed coingecko rates instead of the real ones. You also need to add a self signed SSL certificate that you tell the BTCPayserver host to trust. And of course you need to provide your rates in the same way format that coingecko does.

For the MITM approach to work:

• the self signed SSL certificate needs to be trusted by BTCPayserver
• api.coingecko.com needs to be redirected through /etc/hosts so that the rate is fetched from your server instead of the real one
• I need to check if the rate is in fact being pulled from the json found at https://api.coingecko.com/api/v3/exchange_rates

[AdilElFarissi]

Hi there :)
i use this (GET) endpoint for my LSL scripts... i think may help in this case:
BTCPayServerURL/api/rates?storeId={YourStoreId}&currencyPairs=BTC_USD,BTC_EUR

this return a Json with the current rate that you can use calculate the crypto amount on the fly... you can get the rate of all the available paires in the data source separated by coma (,).

best wishes :)

[Amperstrand]

It needs to be the rate provided by an external provider because the store does its accounting in fiat. So when the merchant invoices 3 euro for a coffee it expects exactly 3 euro to be credited to its fiat bank account.

Yet another option would be to make a native C# rate provider for the rates endpoint, but this would require recompiling BTCPayServer and/or getting the rate provider included in a future version. I just want a simple way to tell my instance : here, use this rate.

Another use case is that if I was running my own rates provider at https://localhost:1234/rates.json as a python script, that python script could poll 4 different rates providers behind the scenes and take whatever rate is worse. This means that even if one of the 4 providers has an issue where they are either unavailable or are displaying a bugged rate, I still have 3 other providers to fall back on.

Will try to get the MITM solution working, but the best thing would be to write a custom rates provider that is pointed at localhost, and then have a small python script serving up whatever rates to use there.

[Amperstrand]

Implementing a man-in-the-middle approach and emulating https://api.coingecko.com/api/v3/exchange_rates actually worked, but I suspect a C# provider that does a similar thing except without the MITM part is probably a better long term solution.

Proof of concept here: https://github.com/amperstrand/mitm-rates

I also discovered that btcpayserver will use a cached exchange rate if it fails to fetch an exchange rate. Will this is probably usually a good fail-safe, it might in some cases be preferable to fail safe and refuse to provide invoices if the exchange rate is not available.

[AdilElFarissi]

You don't need more than what already exist in BTCPay to do it ^^
One of the problems that BTCPay solve is the data sources API limitations by requesting and caching the rates... so, when the rates are requested by the apps they get the cached rates and BTCPay do not request the rate from the data sources for each invoice because if there is 100 processed invoices BTCPay will have to call the data source 100 times... (over limits)

If i can suggest solutions there are 2 and it depends on your background environment:

1. create 4 stores and configure a different data source for each. in this case you can use the endpoint mentioned above and you have just to change the store id... then, request the invoice from the store where the rate is the correct one...

2. use the Greenfield API to get the rates from the same store by updating the data source and requesting the rates 4 times... then, you update a last time to switch to the source where the rate is correct before requesting the invoice...
   https://docs.btcpayserver.org/API/Greenfield/v1/#operation/Stores_GetStoreRates
   https://docs.btcpayserver.org/API/Greenfield/v1/#operation/Stores_UpdateStoreRateConfiguration

best wishes :)

[Amperstrand]

https://github.com/amperstrand/mitm-rates now includes a readme bash script which should mostly just work.

It deploys a docker container that can man in the middle the rates between btcpayserver and the following providers:

• https://api.coingecko.com/api/v3/exchange_rates (multiple currencies rates, but no bid/ask)
• https://bylls.com/api/price?from_currency=BTC&to_currency=CAD (single exchange rate, no bid/ask)
• https://public.bitbank.cc/tickers (supports multiple currencies and bid/ask for each one)

I have also made a fork of btcpayserver where I have copied the bitbank rates provider but pointed it to https://customrates.local/tickers instead:

• https://github.com/Amperstrand/btcpayserver/blob/master/BTCPayServer.Rating/Providers/CustomRateProvider.cs

I would like to ask for feedback from @btcmaine @Zaxounette @IT4Bitcoin @dubdam and others that might find this useful:

1. Is a solution that redirects public.bitbank.cc to a custom endpoint through editing /etc/hosts and installing a selfsigned SSL cert good enough for now, or should I lobby to include a rates provider with a configurable endpoint
2. What format should a custom rate provider use? I like that it can support an arbitrary number of currencies and that it supports bid/ask, so one that looks like bitbank seems like a good choice. Ideally, I would also like to see support for amount so that you might get a different price if you are paying 1 dollar vs 1000. In the strike docs, you essentially ask for a quote where you state the amount: https://docs.strike.me/walkthrough/receiving-payments.

The MITM style endpoint works well enough for me, but I might keep working on this if there is interest from others.

I think it might be possible to adapt https://github.com/amperstrand/mitm-rates to talk to strike and actually receive a quote with your API key so that the amount of fiat that you are asking from the customer is the amount of fiat that you get.

I understand that API limits are a potential issue, but this responsibility would be moved to the custom rates provider. The use case that is not being covered by BTCPayserver as it exists without this is that it is not possible to connect it to a rates provider that will give you a guaranteed conversion rate to fiat which would be useful for shops that do all of their accounting in fiat and just want to add Bitcoin as a payment option.