Sudden 502 bad gateway

[SireWilliamson]

I updated btcpay by pulling the repo. I ran build.sh.
No errors or warnings.
Started the server. I now receive 502 bad gateway errors when trying to access the server.
The nginx logs say:
2023/11/14 04:11:47 [error] 14028#14028: *33 connect() failed (111: Connection refused) while connecting to upstream, client: xxx.xxx.xxx.xxx, server: mybtcpayserv.com, request: "GET / HTTP/2.0", upstream: "http://127.0.0.1:23000/", host: "mybtcpayserv.com"

From https://github.com/btcpayserver/btcpayserver
1f9fbbe..96569ae master -> origin/master

• [new tag] BTCPayServer.Client/v1.7.3 -> BTCPayServer.Client/v1.7.3
• [new tag] v1.11.4 -> v1.11.4
• [new tag] v1.11.5 -> v1.11.5
• [new tag] v1.11.5-rc1 -> v1.11.5-rc1
• [new tag] v1.11.6 -> v1.11.6
• [new tag] v1.11.7 -> v1.11.7
• [new tag] v1.11.7-rc1 -> v1.11.7-rc1
  Updating 1f9fbbe..96569ae
  Fast-forward

Any ideas what might be causing this?

[dennisreimann]

I assume you are running the manual deployment and upgraded from v1.11.3? Guessing the version by the tags that have been pulled, but you might not have build and run anm earlier version.

The nginx log indicates that BTCPay Server is not running or reachable, can you check the BTCPay Server and NBXplorer logs as well and see why these services might not start or fail? These commands should give you an overview:

sudo journalctl -xe --unit nbxplorer
sudo journalctl -xe --unit btcpay

[SireWilliamson]

The services say they are running. I thought maybe some thing had changed from 1.7.3 to 1.11.5
It says my BTCPayServer.Client did not upgrade. Is that supposed to be?
"BTCPayServer.Client/v1.7.3 -> BTCPayServer.Client/v1.7.3"

● btcpay.service - BTCPay Server
Loaded: loaded (/etc/systemd/system/btcpay.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2023-11-14 14:36:48 UTC; 785ms ago
Main PID: 722308 (run.sh)
Tasks: 14 (limit: 9493)
Memory: 35.4M
CGroup: /system.slice/btcpay.service
├─722308 /bin/bash /webservice/btcpay/run.sh --conf=/etc/btcpay/btcpay.config
└─722310 dotnet BTCPayServer.dll --conf=/etc/btcpay/btcpay.config
[714142]: info: Configuration: Data Directory: /btcpay/.btcpayserver/Main
[714142]: info: Configuration: Configuration File: /etc/btcpay/btcpay.config
[714142]: info: BTCPayServer.Plugins.PluginManager: Loading plugins from /btcpay/.btcpayserver/Plugins
[714142]: info: BTCPayServer.Plugins.PluginManager: Adding and executing plugin BTCPayServer - 1.11.7
[714142]: info: BTCPayServer.Plugins.PluginManager: Adding and executing plugin BTCPayServer.Plugins.Shopify - 1.11.7
[714142]: info: BTCPayServer.Plugins.PluginManager: Adding and executing plugin BTCPayServer.Plugins.PointOfSale - 1.11.7
[714142]: info: BTCPayServer.Plugins.PluginManager: Adding and executing plugin BTCPayServer.Plugins.PayButton - 1.11.7
[714142]: info: BTCPayServer.Plugins.PluginManager: Adding and executing plugin BTCPayServer.Plugins.NFC - 1.11.7
[714142]: info: BTCPayServer.Plugins.PluginManager: Adding and executing plugin BTCPayServer.Plugins.Crowdfund - 1.11.7
[714142]: info: Configuration: Supported chains: BTC

● nbxplorer.service - NBXplorer daemon
Loaded: loaded (/etc/systemd/system/nbxplorer.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2023-11-14 14:42:50 UTC; 389ms ago
Main PID: 730478 (run.sh)
Tasks: 17 (limit: 9493)
Memory: 27.0M
CGroup: /system.slice/nbxplorer.service
├─730478 /bin/bash /webservice/nbxplorer/run.sh --conf=/etc/nbxplorer/nbxplorer.config --automigrate
└─730480 dotnet run --no-launch-profile --no-build -c Release --project NBXplorer/NBXplorer.csproj -- --conf=/etc/nbxplorer/nbxplorer.config --automigrate

systemd[1]: Started NBXplorer daemon.

-- The job identifier is 939875 and the job result is done.
Nov 14 14:28:00 systemd[1]: Started NBXplorer daemon.
-- Subject: A start job for unit nbxplorer.service has finished successfully
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support

-- A start job for unit nbxplorer.service has finished successfully.

-- The job identifier is 939875.
[710147]: info: Configuration: Data Directory: /btcpay/.nbxplorer/Main
[710147]: info: Configuration: Configuration File: /etc/nbxplorer/nbxplorer.config
[710147]: info: Configuration: Network: Mainnet
[710147]: info: Configuration: Supported chains: BTC
[710147]: info: Configuration: DBCache: 50 MB
[710147]: info: Configuration: Network: Mainnet
[710147]: info: Configuration: Supported chains: BTC
[710147]: info: Configuration: DBCache: 50 MB

[SireWilliamson]

I've also tried pulling a fresh copy of BTCPay server, and building. Made no difference.

[SireWilliamson]

Looks like it might be an issue in nbxplorer
run.sh[13651]: Severity: ERROR
run.sh[13651]: SqlState: 23505
run.sh[13651]: MessageText: duplicate key value violates unique constraint "nbxv1_evts_pkey"
run.sh[13651]: Detail: Detail redacted as it may contain sensitive data. Specify 'Include Error Detail' in the connection string to include this information.
run.sh[13651]: SchemaName: public
run.sh[13651]: TableName: nbxv1_evts
run.sh[13651]: ConstraintName: nbxv1_evts_pkey
run.sh[13651]: File: nbtinsert.c
run.sh[13651]: Line: 563
run.sh[13651]: Routine: _bt_check_unique

[NicolasDorier]

@SireWilliamson I need you to give us more logs of NBX.

But 502 bad gateway is probably unrelated. The 502 bad gateway is typically sent by reverse proxies such as nginx when they don't manage to connect to the internal server.

You error show "http://127.0.0.1:23000/", why 23000?

[dennisreimann]

@NicolasDorier It's the vale in the sample nginx config — I don't find where we specify the BTCPay Server port there, which port is the default?

[SireWilliamson]

What I found was that BTCPay says it is running, but it is actually timing out quite some time later. It tries to start the process 5 times before it just stops. I think it fails to start because of the nbxplorer issue. nbxplorer seems to just try over and over, but keeps getting the duplicate key error. I'll try to get some more detailed logs.

I'm not completely clear on what nbxplorer keeps track of. If I just reset my nbxplorer DB what info do I lose?

23000 is what is in the nginx config in the "Extended Manual"
I've been running with the same config for 3 years. No issues with the config.

[SireWilliamson]

So I was right.
Because NBXplorer was not actually running properly, BTCPay never got to the point where it was broadcasting on :23000.
NBXplorer wasn't running correctly, because every time it started it was trying to re-migrate from DBTrie. Since the default is NOT to delete the old DBTrie after migration, the automigrate was getting confused. There should be a flag stating that migration was already completed, the first time it is done, so that if someone tries that option again, it just says, "This was already done." and skips, even if the old DBTrie still exists. OR It should by default delete the old DBTrie once migration is completed.

Thanks for your help though. I hope others, if they encounter this problem, know how to fix it now.