Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Configure and enforce max connections per user at runtime #707

Open
wants to merge 13 commits into
base: master
Choose a base branch
from

Conversation

Justin-Kwan
Copy link

@Justin-Kwan Justin-Kwan commented Apr 16, 2022

Description

This PR introduces a new command to enforce max user connections at runtime. In addition to existing max_user_connections per user on pgbouncer.ini file reload, PgBouncer operators can now execute SET USER <user> = 'max_user_connections=<new limit> pool_mode=<session|transaction|statement>' to update a user's max connections across all pools that the user is in.

When the pgbouncer.ini file is reloaded, or the above command is submitted, PgBouncer will now drop the user's connections as a reaction to these new values. PgBouncer will prioritize killing any idle and unused connections, and then kill the oldest active connections across all pools owned by the user to achieve the configured connection limit.

In addition, this PR also updates the SHOW USERS; command to display the maximum configured connections per user to increase visibility when throttling users.

In a production multi-tenant cluster, we have seen the need for PgBouncer to enforce maximum connections per user across all pools. When our upstream services (PG users) acquires too many connections while executing expensive and/or repetitive queries, this often puts strain on our Postgres primary server and starves CPU/Disk IO resources at the server. To remediate this, currently we manually constrain the user to maximum number of Postgres connections at server side via ALTER USER "some_bad-user" WITH CONNECTION LIMIT 123;. We've seen a clear correlation between the connection pool size (concurrency) per tenant denoted in green and database resource load denoted in yellow:

production connections to load correlation (1)

Having this option in PgBouncer would be very convenient because:

  1. Postgres does not actively kill existing connections that surpass the new limit (which means load does not immediately reduce)
  2. This opens the option for operators to build custom scripts that can automate per user connection throttling in PgBouncer via the SET USER ... command

Note

This PR follows from #706 and has some overlapping commits. The previous PR should merge before this one.

cc @viggy28

@Justin-Kwan Justin-Kwan changed the title Justin/feature/enforce max user connections Enforce Max User Connections at Runtime Apr 16, 2022
@Justin-Kwan Justin-Kwan force-pushed the justin/feature/enforce_max_user_connections branch from beffc77 to c12bd87 Compare April 16, 2022 00:14
@Justin-Kwan Justin-Kwan changed the title Enforce Max User Connections at Runtime Configure and Throttle Max User Connections at Runtime Apr 16, 2022
@Justin-Kwan Justin-Kwan changed the title Configure and Throttle Max User Connections at Runtime Configure and enforce max user connections at runtime Apr 16, 2022
@Justin-Kwan Justin-Kwan force-pushed the justin/feature/enforce_max_user_connections branch 8 times, most recently from f43dd22 to 2d4500f Compare April 18, 2022 18:21
@Justin-Kwan Justin-Kwan force-pushed the justin/feature/enforce_max_user_connections branch 3 times, most recently from 07248c8 to 82f9bbe Compare April 21, 2022 21:34
@Justin-Kwan Justin-Kwan force-pushed the justin/feature/enforce_max_user_connections branch 8 times, most recently from 1580fd7 to adf171c Compare April 24, 2022 22:45
@Justin-Kwan Justin-Kwan changed the title Configure and enforce max user connections at runtime Configure and enforce max connections per user at runtime Apr 26, 2022
@Justin-Kwan Justin-Kwan force-pushed the justin/feature/enforce_max_user_connections branch from 1cf4190 to 821b2e5 Compare May 10, 2022 06:47
@Justin-Kwan Justin-Kwan force-pushed the justin/feature/enforce_max_user_connections branch from 821b2e5 to 9f4e309 Compare May 10, 2022 06:52
@Justin-Kwan Justin-Kwan force-pushed the justin/feature/enforce_max_user_connections branch from 9f4e309 to 0b2c3f4 Compare May 11, 2022 16:49
@Justin-Kwan Justin-Kwan force-pushed the justin/feature/enforce_max_user_connections branch 2 times, most recently from b48531c to 2add413 Compare May 11, 2022 20:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants