Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[bitnami/postgresql-ha] Standard postgresql-ha deployment fails when username/password are provided #24226

Closed
Sediket opened this issue Mar 6, 2024 · 8 comments · May be fixed by #24593
Closed

[bitnami/postgresql-ha] Standard postgresql-ha deployment fails when username/password are provided #24226

Sediket opened this issue Mar 6, 2024 · 8 comments · May be fixed by #24593
Assignees
@carrodher @migruiz4
Labels
postgresql-ha solved stale 15 days without activity tech-issues The user has a technical issue about an application triage Triage is needed

Comments

@Sediket
Copy link

Sediket commented Mar 6, 2024

Name and Version

bitnami/postgresql-ha 13.6.0

What architecture are you using?

amd64

What steps will reproduce the bug?

Done in Killercoda: https://killercoda.com/playgrounds/scenario/kubernetes

The pvcs are not present from a previous install.

#Update Helm
curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh

#Install bitnami/postgresql-ha
helm show values oci://registry-1.docker.io/bitnamicharts/postgresql-ha > values.yaml

#modify values to include username/passwords in global section and enabled debuging:
global:
  imageRegistry: ""
  ## E.g.
  ## imagePullSecrets:
  ##   - myRegistryKeySecretName
  ##
  imagePullSecrets: []
  storageClass: ""
  postgresql:
    username: "post"
    password: "password"
    database: "db"
    repmgrUsername: "rep"
    repmgrPassword: "password"
    repmgrDatabase: "rep-db"
    existingSecret: ""
  ldap:
    bindpw: ""
    existingSecret: ""
  pgpool:
    adminUsername: "pg"
    adminPassword: "password"

#install: 
helm install my-release oci://registry-1.docker.io/bitnamicharts/postgresql-ha -f values.yaml

#logs:
controlplane $ k logs my-release-postgresql-ha-postgresql-0 --follow
postgresql-repmgr 23:35:34.07 INFO  ==> 
postgresql-repmgr 23:35:34.07 INFO  ==> Welcome to the Bitnami postgresql-repmgr container
postgresql-repmgr 23:35:34.08 INFO  ==> Subscribe to project updates by watching https://github.com/bitnami/containers
postgresql-repmgr 23:35:34.11 INFO  ==> Submit issues and feature requests at https://github.com/bitnami/containers/issues
postgresql-repmgr 23:35:34.11 INFO  ==> 
postgresql-repmgr 23:35:34.13 DEBUG ==> Configuring libnss_wrapper...
postgresql-repmgr 23:35:34.18 DEBUG ==> Copying files from /opt/bitnami/postgresql/conf.default to /opt/bitnami/postgresql/conf
postgresql-repmgr 23:35:34.19 INFO  ==> ** Starting PostgreSQL with Replication Manager setup **
postgresql-repmgr 23:35:34.26 INFO  ==> Validating settings in REPMGR_* env vars...
postgresql-repmgr 23:35:34.27 INFO  ==> Validating settings in POSTGRESQL_* env vars..
postgresql-repmgr 23:35:34.28 INFO  ==> Querying all partner nodes for common upstream node...
postgresql-repmgr 23:35:34.30 DEBUG ==> Checking node 'my-release-postgresql-ha-postgresql-0.my-release-postgresql-ha-postgresql-headless.default.svc.cluster.local:5432'...
psql: error: connection to server at "my-release-postgresql-ha-postgresql-0.my-release-postgresql-ha-postgresql-headless.default.svc.cluster.local" (192.168.1.59), port 5432 failed: Connection refused
        Is the server running on that host and accepting TCP/IP connections?
postgresql-repmgr 23:35:34.33 DEBUG ==> Skipping: failed to get primary from the node 'my-release-postgresql-ha-postgresql-0.my-release-postgresql-ha-postgresql-headless.default.svc.cluster.local:5432'!
postgresql-repmgr 23:35:34.36 DEBUG ==> Checking node 'my-release-postgresql-ha-postgresql-1.my-release-postgresql-ha-postgresql-headless.default.svc.cluster.local:5432'...
psql: error: could not translate host name "my-release-postgresql-ha-postgresql-1.my-release-postgresql-ha-postgresql-headless.default.svc.cluster.local" to address: Name or service not known
postgresql-repmgr 23:35:34.41 DEBUG ==> Skipping: failed to get primary from the node 'my-release-postgresql-ha-postgresql-1.my-release-postgresql-ha-postgresql-headless.default.svc.cluster.local:5432'!
postgresql-repmgr 23:35:34.42 DEBUG ==> Checking node 'my-release-postgresql-ha-postgresql-2.my-release-postgresql-ha-postgresql-headless.default.svc.cluster.local:5432'...
psql: error: could not translate host name "my-release-postgresql-ha-postgresql-2.my-release-postgresql-ha-postgresql-headless.default.svc.cluster.local" to address: Name or service not known
postgresql-repmgr 23:35:34.46 DEBUG ==> Skipping: failed to get primary from the node 'my-release-postgresql-ha-postgresql-2.my-release-postgresql-ha-postgresql-headless.default.svc.cluster.local:5432'!
postgresql-repmgr 23:35:34.47 INFO  ==> There are no nodes with primary role. Assuming the primary role...
postgresql-repmgr 23:35:34.48 INFO  ==> Preparing PostgreSQL configuration...
postgresql-repmgr 23:35:34.49 DEBUG ==> Injecting a new postgresql.conf file...
postgresql-repmgr 23:35:34.50 INFO  ==> postgresql.conf file not detected. Generating it...
postgresql-repmgr 23:35:34.70 DEBUG ==> Injecting a new pg_hba.conf file...
postgresql-repmgr 23:35:34.71 INFO  ==> Preparing repmgr configuration...
postgresql-repmgr 23:35:34.74 DEBUG ==> Node ID: '1000', Rol: 'primary', Primary Node: ':5432'
postgresql-repmgr 23:35:34.75 INFO  ==> Initializing Repmgr...
postgresql-repmgr 23:35:34.77 INFO  ==> Initializing PostgreSQL database...
postgresql-repmgr 23:35:34.78 DEBUG ==> Copying files from /bitnami/postgresql/conf to /opt/bitnami/postgresql/conf
postgresql-repmgr 23:35:34.80 INFO  ==> Custom configuration /opt/bitnami/postgresql/conf/postgresql.conf detected
postgresql-repmgr 23:35:34.81 INFO  ==> Custom configuration /opt/bitnami/postgresql/conf/pg_hba.conf detected
postgresql-repmgr 23:35:34.81 DEBUG ==> Ensuring expected directories/files exist...
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /bitnami/postgresql/data ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.

Success. You can now start the database server using:

    /opt/bitnami/postgresql/bin/pg_ctl -D /bitnami/postgresql/data -l logfile start

postgresql-repmgr 23:35:36.61 INFO  ==> Starting PostgreSQL in background...
waiting for server to start.... done
server started
CREATE DATABASE
postgresql-repmgr 23:35:36.84 INFO  ==> Creating user post
CREATE ROLE
postgresql-repmgr 23:35:36.87 INFO  ==> Granting access to "post" to the database "db"
GRANT
ALTER DATABASE
postgresql-repmgr 23:35:36.92 INFO  ==> Setting ownership for the 'public' schema database "db" to "post"
ALTER SCHEMA
postgresql-repmgr 23:35:36.95 INFO  ==> Creating replication user rep
CREATE ROLE
postgresql-repmgr 23:35:36.98 INFO  ==> Stopping PostgreSQL...
waiting for server to shut down.... done
server stopped
postgresql-repmgr 23:35:37.31 INFO  ==> Configuring replication parameters
postgresql-repmgr 23:35:37.35 INFO  ==> Configuring fsync
postgresql-repmgr 23:35:37.36 INFO  ==> Starting PostgreSQL in background...
waiting for server to start.... done
server started
postgresql-repmgr 23:35:37.50 INFO  ==> Creating repmgr user: rep
psql: error: connection to server on socket "/tmp/.s.PGSQL.5432" failed: FATAL:  password authentication failed for user "postgres"
postgresql-repmgr 23:35:37.53 INFO  ==> Stopping PostgreSQL...
waiting for server to shut down.... done
server stopped

# the pod restarts:
controlplane $ k logs my-release-postgresql-ha-postgresql-0 --follow
postgresql-repmgr 23:35:38.59 INFO  ==> 
postgresql-repmgr 23:35:38.59 INFO  ==> Welcome to the Bitnami postgresql-repmgr container
postgresql-repmgr 23:35:38.59 INFO  ==> Subscribe to project updates by watching https://github.com/bitnami/containers
postgresql-repmgr 23:35:38.59 INFO  ==> Submit issues and feature requests at https://github.com/bitnami/containers/issues
postgresql-repmgr 23:35:38.60 INFO  ==> 
postgresql-repmgr 23:35:38.60 DEBUG ==> Configuring libnss_wrapper...
postgresql-repmgr 23:35:38.61 DEBUG ==> Copying files from /opt/bitnami/postgresql/conf.default to /opt/bitnami/postgresql/conf
postgresql-repmgr 23:35:38.62 INFO  ==> ** Starting PostgreSQL with Replication Manager setup **
postgresql-repmgr 23:35:38.64 INFO  ==> Validating settings in REPMGR_* env vars...
postgresql-repmgr 23:35:38.65 INFO  ==> Validating settings in POSTGRESQL_* env vars..
postgresql-repmgr 23:35:38.66 INFO  ==> Querying all partner nodes for common upstream node...
postgresql-repmgr 23:35:38.66 DEBUG ==> Checking node 'my-release-postgresql-ha-postgresql-0.my-release-postgresql-ha-postgresql-headless.default.svc.cluster.local:5432'...
psql: error: connection to server at "my-release-postgresql-ha-postgresql-0.my-release-postgresql-ha-postgresql-headless.default.svc.cluster.local" (192.168.1.59), port 5432 failed: Connection refused
        Is the server running on that host and accepting TCP/IP connections?
postgresql-repmgr 23:35:38.68 DEBUG ==> Skipping: failed to get primary from the node 'my-release-postgresql-ha-postgresql-0.my-release-postgresql-ha-postgresql-headless.default.svc.cluster.local:5432'!
postgresql-repmgr 23:35:38.68 DEBUG ==> Checking node 'my-release-postgresql-ha-postgresql-1.my-release-postgresql-ha-postgresql-headless.default.svc.cluster.local:5432'...
psql: error: could not translate host name "my-release-postgresql-ha-postgresql-1.my-release-postgresql-ha-postgresql-headless.default.svc.cluster.local" to address: Name or service not known
postgresql-repmgr 23:35:38.70 DEBUG ==> Skipping: failed to get primary from the node 'my-release-postgresql-ha-postgresql-1.my-release-postgresql-ha-postgresql-headless.default.svc.cluster.local:5432'!
postgresql-repmgr 23:35:38.70 DEBUG ==> Checking node 'my-release-postgresql-ha-postgresql-2.my-release-postgresql-ha-postgresql-headless.default.svc.cluster.local:5432'...
psql: error: could not translate host name "my-release-postgresql-ha-postgresql-2.my-release-postgresql-ha-postgresql-headless.default.svc.cluster.local" to address: Name or service not known
postgresql-repmgr 23:35:38.71 DEBUG ==> Skipping: failed to get primary from the node 'my-release-postgresql-ha-postgresql-2.my-release-postgresql-ha-postgresql-headless.default.svc.cluster.local:5432'!
postgresql-repmgr 23:35:38.72 INFO  ==> There are no nodes with primary role. Assuming the primary role...
postgresql-repmgr 23:35:38.72 INFO  ==> Preparing PostgreSQL configuration...
postgresql-repmgr 23:35:38.73 DEBUG ==> Injecting a new postgresql.conf file...
postgresql-repmgr 23:35:38.73 INFO  ==> postgresql.conf file not detected. Generating it...
postgresql-repmgr 23:35:38.82 DEBUG ==> Injecting a new pg_hba.conf file...
postgresql-repmgr 23:35:38.83 INFO  ==> Preparing repmgr configuration...
postgresql-repmgr 23:35:38.84 DEBUG ==> Node ID: '1000', Rol: 'primary', Primary Node: ':5432'
postgresql-repmgr 23:35:38.84 INFO  ==> Initializing Repmgr...
postgresql-repmgr 23:35:38.84 INFO  ==> Initializing PostgreSQL database...
postgresql-repmgr 23:35:38.85 DEBUG ==> Copying files from /bitnami/postgresql/conf to /opt/bitnami/postgresql/conf
postgresql-repmgr 23:35:38.85 INFO  ==> Custom configuration /opt/bitnami/postgresql/conf/postgresql.conf detected
postgresql-repmgr 23:35:38.86 INFO  ==> Custom configuration /opt/bitnami/postgresql/conf/pg_hba.conf detected
postgresql-repmgr 23:35:38.86 DEBUG ==> Ensuring expected directories/files exist...
postgresql-repmgr 23:35:38.88 INFO  ==> Deploying PostgreSQL with persisted data...
postgresql-repmgr 23:35:38.91 INFO  ==> Configuring replication parameters
postgresql-repmgr 23:35:38.98 INFO  ==> Configuring fsync
postgresql-repmgr 23:35:39.00 DEBUG ==> Skipping repmgr configuration...
postgresql-repmgr 23:35:39.00 INFO  ==> ** PostgreSQL with Replication Manager setup finished! **

postgresql-repmgr 23:35:39.03 INFO  ==> Starting PostgreSQL in background...
waiting for server to start.... done
server started
postgresql-repmgr 23:35:39.15 INFO  ==> ** Starting repmgrd **
[2024-03-06 23:35:39] [NOTICE] repmgrd (repmgrd 5.3.3) starting up
[2024-03-06 23:35:39] [ERROR] connection to database failed
[2024-03-06 23:35:39] [DETAIL] 
connection to server at "my-release-postgresql-ha-postgresql-0.my-release-postgresql-ha-postgresql-headless.default.svc.cluster.local" (192.168.1.59), port 5432 failed: FATAL:  database "rep-db" does not exist

[2024-03-06 23:35:39] [DETAIL] attempted to connect using:
  user=rep password=password connect_timeout=5 dbname=rep-db host=my-release-postgresql-ha-postgresql-0.my-release-postgresql-ha-postgresql-headless.default.svc.cluster.local port=5432 fallback_application_name=repmgr options=-csearch_path=

Are you using any custom parameters or values?

Yes, username/password and db in the global part of the values.yaml and enabled debugging

global:
  imageRegistry: ""
  ## E.g.
  ## imagePullSecrets:
  ##   - myRegistryKeySecretName
  ##
  imagePullSecrets: []
  storageClass: ""
  postgresql:
    username: "post"
    password: "password"
    database: "db"
    repmgrUsername: "rep"
    repmgrPassword: "password"
    repmgrDatabase: "rep-db"
    existingSecret: ""
  ldap:
    bindpw: ""
    existingSecret: ""
  pgpool:
    adminUsername: "pg"
    adminPassword: "password"
    existingSecret: ""

What is the expected behavior?

all pods to deploy and ready without issues, as seen with a default install: helm install my-release oci://registry-1.docker.io/bitnamicharts/postgresql-ha

What do you see instead?

pods crashing with attached logs, can't authenticate

On pods first startup:

psql: error: connection to server on socket "/tmp/.s.PGSQL.5432" failed: FATAL:  password authentication failed for user "postgres"

On pods following startus:

[2024-03-06 23:35:39] [ERROR] connection to database failed
[2024-03-06 23:35:39] [DETAIL] 
connection to server at "my-release-postgresql-ha-postgresql-0.my-release-postgresql-ha-postgresql-headless.default.svc.cluster.local" (192.168.1.59), port 5432 failed: FATAL:  database "rep-db" does not exist

[2024-03-06 23:35:39] [DETAIL] attempted to connect using:
  user=rep password=password connect_timeout=5 dbname=rep-db host=my-release-postgresql-ha-postgresql-0.my-release-postgresql-ha-postgresql-headless.default.svc.cluster.local port=5432 fallback_application_name=repmgr options=-csearch_path=

Additional information

the pvcs are not present from a previous install.
@Sediket Sediket added the tech-issues The user has a technical issue about an application label Mar 6, 2024
@github-actions github-actions bot added the triage Triage is needed label Mar 6, 2024
@javsalgar javsalgar changed the title Standard postgresql-ha deployment fails when username/password are provided [bitnami/postgresql-ha] Standard postgresql-ha deployment fails when username/password are provided Mar 8, 2024
@github-actions github-actions bot removed the triage Triage is needed label Mar 8, 2024
@github-actions github-actions bot assigned migruiz4 and unassigned javsalgar Mar 8, 2024
@migruiz4 migruiz4 mentioned this issue Mar 21, 2024
Open
4 tasks
@migruiz4
Copy link
Member

Hi @Sediket,

Thank you for reporting this issue. There was an issue with the Postgresql-ha chart where the postgres user password is not being set when using global.postgresql.username/postgresql.username.

I was able to reproduce your issue and submitted this PR with a fix: #24593

Please give it a try and let me know if it works for you.

As I mentioned in the PR description, it is possible this behavior was intentional and the PR doesn't get merged. In that case, your issue could be resolved by setting posgresql.postgresPassword / global.posgresql.postgresPassword

P.S.: The database name 'rep-db' will cause a failure because of the dash, but changing it with repdb fixes it.

@github-actions GitHub Actions
Copy link

github-actions bot commented Apr 6, 2024

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

@github-actions github-actions bot added the stale 15 days without activity label Apr 6, 2024
@github-actions GitHub Actions
Copy link

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.

@bitnami-bot bitnami-bot closed this as not planned Won't fix, can't repro, duplicate, stale Apr 11, 2024
@smolinari
Copy link

This should stay open, as it probably relates to #24593.

Scott

@javsalgar javsalgar reopened this Apr 18, 2024
@javsalgar javsalgar removed the solved label Apr 18, 2024
@github-actions github-actions bot added the triage Triage is needed label Apr 18, 2024
@javsalgar javsalgar removed the stale 15 days without activity label Apr 18, 2024
@github-actions GitHub Actions
Copy link

github-actions bot commented May 4, 2024

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

@github-actions github-actions bot added the stale 15 days without activity label May 4, 2024
@github-actions GitHub Actions
Copy link

github-actions bot commented May 9, 2024

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.

@github-actions github-actions bot added the solved label May 9, 2024
@bitnami-bot bitnami-bot closed this as not planned Won't fix, can't repro, duplicate, stale May 9, 2024
@smolinari
Copy link

There is a PR set to fix this in #24593.

Can someone review it please and either merge or close it, if it isn't a good change?

Scott

@8ball030
Copy link

Thanks @smolinari i hope this get merged soon

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@carrodher carrodher

@migruiz4 migruiz4

Labels
postgresql-ha solved stale 15 days without activity tech-issues The user has a technical issue about an application triage Triage is needed
Projects
None yet
Milestone
No milestone
7 participants
@javsalgar @smolinari @bitnami-bot @carrodher @Sediket @8ball030 @migruiz4