pgbouncer cannot connect to server

[michaelromagne]

What happened?:

While installing pachyderm with helm using minikube, the patchd pod is running but not ready:

~/perso/advent-of-code-submissions-2023/day20_pachyderm (main*) » kubectl get pods -o wide                                                                   michaelromagne@GGFRLTL035
NAME                                         READY   STATUS    RESTARTS        AGE   IP            NODE       NOMINATED NODE   READINESS GATES
console-859f59ddcd-99tb9                     1/1     Running   0               26m   10.244.0.7    minikube   <none>           <none>
etcd-0                                       1/1     Running   0               26m   10.244.0.11   minikube   <none>           <none>
pachd-84db85d6c8-8dknb                       0/1     Running   4 (4m24s ago)   26m   10.244.0.8    minikube   <none>           <none>
pachyderm-kube-event-tail-5db949bbdf-rxscs   1/1     Running   0               26m   10.244.0.4    minikube   <none>           <none>
pachyderm-loki-0                             1/1     Running   0               26m   10.244.0.10   minikube   <none>           <none>
pachyderm-promtail-92hng                     1/1     Running   0               26m   10.244.0.6    minikube   <none>           <none>
pachyderm-proxy-774f6947f-6rcmz              1/1     Running   0               26m   10.244.0.3    minikube   <none>           <none>
pg-bouncer-5959f447f4-5t6zl                  1/1     Running   0               26m   10.244.0.5    minikube   <none>           <none>
postgres-0                                   1/1     Running   0               26m   10.244.0.9    minikube   <none>           <none>

When looking at logs, I see that it can't connect to pgbouncer :

{"severity":"info","time":"2023-12-30T14:06:44.422227285Z","caller":"pachd/main.go:40","message":"pachd: starting","mode":"full"}
{"severity":"info","time":"2023-12-30T14:06:44.422665734Z","logger":"maxprocs","caller":"maxprocs/maxprocs.go:47","message":"maxprocs: Leaving GOMAXPROCS=20: CPU quota undefined"}
{"severity":"info","time":"2023-12-30T14:06:44.422707041Z","caller":"pachd/setup.go:55","message":"gc: set gc percent","value":100}
{"severity":"info","time":"2023-12-30T14:06:44.422722598Z","caller":"pachd/builder.go:504","message":"memlimit: setting GOMEMLIMIT (95% of the k8s value)","limit":"29 GiB","setFrom":"kubernetes limit"}
{"severity":"info","time":"2023-12-30T14:06:44.422734765Z","caller":"pachd/setup.go:42","message":"version info","versionInfo":{"major":2,"minor":8,"micro":2,"additional":"","git_commit":"1ed48c5527ae2488b71f756a8ef4c9b435fb9805","git_tree_modified":"false","build_date":"2023-12-13T18:36:04Z","go_version":"go1.21.5","platform":"amd64"}}
{"severity":"info","time":"2023-12-30T14:06:44.422746257Z","caller":"pachd/setup.go:82","message":"no Jaeger collector found (JAEGER_COLLECTOR_SERVICE_HOST not set)"}
{"severity":"info","time":"2023-12-30T14:06:44.423110054Z","logger":"WaitUntilReady","caller":"dbutil/db.go:146","message":"WaitUntilReady: span start"}
{"severity":"info","time":"2023-12-30T14:06:44.423152398Z","logger":"WaitUntilReady","caller":"dbutil/db.go:150","message":"waiting for db to be ready..."}
{"severity":"info","time":"2023-12-30T14:06:44.426375101Z","logger":"WaitUntilReady","caller":"dbutil/db.go:156","message":"db is not ready","reason":"failed to connect to `host=pg-bouncer user=pachyderm database=pachyderm`: server error (FATAL: pgbouncer cannot connect to server (SQLSTATE 08P01))"}
{"severity":"info","time":"2023-12-30T14:06:45.429887849Z","logger":"WaitUntilReady","caller":"dbutil/db.go:156","message":"db is not ready","reason":"failed to connect to `host=pg-bouncer user=pachyderm database=pachyderm`: server error (FATAL: pgbouncer cannot connect to server (SQLSTATE 08P01))"}
{"severity":"info","time":"2023-12-30T14:06:47.438052288Z","logger":"WaitUntilReady","caller":"dbutil/db.go:156","message":"db is not ready","reason":"failed to connect to `host=pg-bouncer user=pachyderm database=pachyderm`: server error (FATAL: pgbouncer cannot connect to server (SQLSTATE 08P01))"}
{"severity":"info","time":"2023-12-30T14:06:48.440415915Z","logger":"WaitUntilReady","caller":"dbutil/db.go:156","message":"db is not ready","reason":"failed to connect to `host=pg-bouncer user=pachyderm database=pachyderm`: server error (FATAL: pgbouncer cannot connect to server (SQLSTATE 08P01))"}
{"severity":"info","time":"2023-12-30T14:06:50.447811065Z","logger":"WaitUntilReady","caller":"dbutil/db.go:156","message":"db is not ready","reason":"failed to connect to `host=pg-bouncer user=pachyderm database=pachyderm`: server error (FATAL: pgbouncer cannot connect to server (SQLSTATE 08P01))"}
{"severity":"info","time":"2023-12-30T14:06:51.451026565Z","logger":"WaitUntilReady","caller":"dbutil/db.go:156","message":"db is not ready","reason":"failed to connect to `host=pg-bouncer user=pachyderm database=pachyderm`: server error (FATAL: pgbouncer cannot connect to server (SQLSTATE 08P01))"}
{"severity":"info","time":"2023-12-30T14:06:53.4558778Z","logger":"WaitUntilReady","caller":"dbutil/db.go:156","message":"db is not ready","reason":"failed to connect to `host=pg-bouncer user=pachyderm database=pachyderm`: server error (FATAL: pgbouncer cannot connect to server (SQLSTATE 08P01))"}
{"severity":"info","time":"2023-12-30T14:06:54.457363977Z","logger":"WaitUntilReady","caller":"dbutil/db.go:156","message":"db is not ready","reason":"failed to connect to `host=pg-bouncer user=pachyderm database=pachyderm`: server error (FATAL: pgbouncer cannot connect to server (SQLSTATE 08P01))"}
{"severity":"info","time":"2023-12-30T14:06:56.463097584Z","logger":"WaitUntilReady","caller":"dbutil/db.go:156","message":"db is not ready","reason":"failed to connect to `host=pg-bouncer user=pachyderm database=pachyderm`: server error (FATAL: pgbouncer cannot connect to server (SQLSTATE 08P01))"}
{"severity":"info","time":"2023-12-30T14:06:57.464621089Z","logger":"WaitUntilReady","caller":"dbutil/db.go:156","message":"db is not ready","reason":"failed to connect to `host=pg-bouncer user=pachyderm database=pachyderm`: server error (FATAL: pgbouncer cannot connect to server (SQLSTATE 08P01))"}
{"severity":"info","time":"2023-12-30T14:06:59.470780783Z","logger":"WaitUntilReady","caller":"dbutil/db.go:156","message":"db is not ready","reason":"failed to connect to `host=pg-bouncer user=pachyderm database=pachyderm`: server error (FATAL: pgbouncer cannot connect to server (SQLSTATE 08P01))"}

When looking at pgbouncer pod logs :

Wrote authentication credentials to /tmp/userlist.txt
Create pgbouncer config in /tmp
    ################## Auto generated ##################
    [databases]
    *=host=postgres.default.svc.cluster.local     port=5432 user=pachyderm

    [pgbouncer]
    listen_addr = 0.0.0.0
    listen_port = 5432
    auth_file = /tmp/userlist.txt
    auth_type = md5
    pool_mode = transaction
    max_client_conn = 100000
    default_pool_size = 80
    ignore_startup_parameters = extra_float_digits
    admin_users = pachyderm
    idle_transaction_timeout = 300

    # TLS settings
    server_tls_sslmode = disable
                ################## end file ##################
    Starting /usr/bin/pgbouncer /tmp/pgbouncer.ini...
2023-12-30 13:50:19.690 UTC [1] LOG kernel file descriptor limit: 1048576 (hard: 1048576); max_client_conn: 100000, max expected fd use: 100012
2023-12-30 13:50:19.690 UTC [1] LOG listening on 0.0.0.0:5432
2023-12-30 13:50:19.690 UTC [1] LOG listening on unix:/tmp/.s.PGSQL.5432
2023-12-30 13:50:19.690 UTC [1] LOG process up: PgBouncer 1.16.1, libevent 2.1.12-stable (epoll), adns: udns 0.4, tls: OpenSSL 1.1.1n  15 Mar 2022
2023-12-30 13:50:44.161 UTC [1] LOG C-0x5601ce1daec0: (nodb)/(nouser)@127.0.0.1:55384 registered new auto-database: db=pachyderm
2023-12-30 13:50:44.161 UTC [1] LOG C-0x5601ce1daec0: pachyderm/pachyderm@127.0.0.1:55384 login attempt: db=pachyderm user=pachyderm tls=no
2023-12-30 13:50:59.357 UTC [1] LOG S-0x5601ce1e1db0: pachyderm/pachyderm@212.95.74.75:5432 closing because: connect failed (age=15s)
2023-12-30 13:51:14.166 UTC [1] LOG C-0x5601ce1db0f0: pachyderm/pachyderm@127.0.0.1:37780 login attempt: db=pachyderm user=pachyderm tls=no
2023-12-30 13:51:14.166 UTC [1] LOG C-0x5601ce1db0f0: pachyderm/pachyderm@127.0.0.1:37780 closing because: pgbouncer cannot connect to server (age=0s)
2023-12-30 13:51:14.166 UTC [1] WARNING C-0x5601ce1db0f0: pachyderm/pachyderm@127.0.0.1:37780 pooler error: pgbouncer cannot connect to server
2023-12-30 13:51:14.358 UTC [1] LOG S-0x5601ce1e1fe0: pachyderm/pachyderm@212.95.74.75:5432 closing because: connect timeout (age=15s)
2023-12-30 13:51:17.921 UTC [1] LOG C-0x5601ce1db0f0: pachyderm/pachyderm@10.244.0.8:40174 login attempt: db=pachyderm user=pachyderm tls=no
2023-12-30 13:51:17.921 UTC [1] LOG C-0x5601ce1db0f0: pachyderm/pachyderm@10.244.0.8:40174 closing because: pgbouncer cannot connect to server (age=0s)
2023-12-30 13:51:17.921 UTC [1] WARNING C-0x5601ce1db0f0: pachyderm/pachyderm@10.244.0.8:40174 pooler error: pgbouncer cannot connect to server
2023-12-30 13:51:18.924 UTC [1] LOG C-0x5601ce1db0f0: pachyderm/pachyderm@10.244.0.8:40190 login attempt: db=pachyderm user=pachyderm tls=no
2023-12-30 13:51:18.924 UTC [1] LOG C-0x5601ce1db0f0: pachyderm/pachyderm@10.244.0.8:40190 closing because: pgbouncer cannot connect to server (age=0s)
2023-12-30 13:51:18.924 UTC [1] WARNING C-0x5601ce1db0f0: pachyderm/pachyderm@10.244.0.8:40190 pooler error: pgbouncer cannot connect to server
2023-12-30 13:51:19.694 UTC [1] LOG stats: 0 xacts/s, 0 queries/s, in 0 B/s, out 0 B/s, xact 0 us, query 0 us, wait 0 us
2023-12-30 13:51:19.927 UTC [1] LOG C-0x5601ce1db0f0: pachyderm/pachyderm@10.244.0.8:40192 login attempt: db=pachyderm user=pachyderm tls=no
2023-12-30 13:51:19.927 UTC [1] LOG C-0x5601ce1db0f0: pachyderm/pachyderm@10.244.0.8:40192 closing because: pgbouncer cannot connect to server (age=0s)
2023-12-30 13:51:19.927 UTC [1] WARNING C-0x5601ce1db0f0: pachyderm/pachyderm@10.244.0.8:40192 pooler error: pgbouncer cannot connect to server

I basically see that pgbouncer can not connect to postgres pod. I did not see this kind of issue here (which makes me think that I may miss something 🥲 ), and pgbouncer issues did not bring me any help. Could you help on this please ?

What you expected to happen?:

patchd to start

How to reproduce it (as minimally and precisely as possible)?:

I just followed this doc https://docs.pachyderm.com/latest/get-started/first-time-setup/

Environment?:

• Kubernetes version (use kubectl version): 1.26.10

• Pachyderm CLI and pachd server version (use pachctl version): 2.7.3, unable to see pachd version due to the error

• Cloud provider (e.g. aws, azure, gke) or local deployment (e.g. minikube vs dockerized k8s): Minikube, but the same error happened on aws

• If you deployed with helm, the values you used (helm get values pachyderm):
  SER-SUPPLIED VALUES:
  deployTarget: LOCAL
  proxy:
  enabled: true
  host: localhost
  service:
  type: LoadBalancer

• OS (e.g. from /etc/os-release):
  PRETTY_NAME="Ubuntu 22.04.3 LTS"
  NAME="Ubuntu"
  VERSION_ID="22.04"
  VERSION="22.04.3 LTS (Jammy Jellyfish)"
  VERSION_CODENAME=jammy
  ID=ubuntu
  ID_LIKE=debian
  HOME_URL="https://www.ubuntu.com/"
  SUPPORT_URL="https://help.ubuntu.com/"
  BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
  PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
  UBUNTU_CODENAME=jammy

• Others: