Cause of failed cancel request in log?

[mbrancato]

I'm seeing a lot of these with pgbouncer 1.20.1 and PostgreSQL 15. I'm unclear as to the cause of this and what the solution would be.

2023-10-25 11:08:47.934 UTC [1] LOG C-0x7fc0e2efe3a0: (nodb)/(nouser)@10.32.79.178:60072 closing because: failed cancel request (age=0s)

[JelteF]

That can happen when your clients send cancellations, but the keys are unknown by PgBouncer. One common reason this can occur is if you have a load balancer in front of multiple pgbouncer processes. If that applies to you should configure peering: https://www.pgbouncer.org/config.html#section-peers

[mbrancato]

Hi @JelteF

No, in this case there is no load balancer. Is is just like this:
app -> PgBouncer -> cloudsql proxy -> Pg 15

[JelteF]

Two other possible reasons:

1. Your application is trying to cancel queries for a connection that is already closed
2. unlikely but possible: some attacker/misbehaving application is sending random cancelation requests

Do other clients get disconnected close before the message your seeing.

Btw to be clear, while this error might sound bad it's not really a huge problem. It means that the cancellation request that was received by pgbouncer could not be processed, because it does not match any of the currently connected clients.

[mbrancato]

@JelteF thanks for the help

After looking more, it is being load balanced behind a Kubernetes Service resource, but normally we haven't seen this issue.

I'm looking into connection leaking on the library side. It appears to be churning connections all over the place.

[JelteF]

If you have multiple pgbouncer Pods load balanced behind a Kubernetes Service then it's almost certainly because you have not setup peering. It should only start happening when you actually have more than 1 Pods and your application is sending cancellation requests.