Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Postgres integration fails to submit postgres.can_connect failure for database when connection is refused on network level in Azure #16412

Open
anordby opened this issue Dec 13, 2023 · 0 comments

Comments

@anordby
Copy link

anordby commented Dec 13, 2023

Trace log from datadog event:

2023-12-13 12:06:41 CET | CORE | TRACE | (pkg/collector/runner/expvars/expvars.go:129 in AddCheckStats) | Adding stats for postgres:63f61c8625e257c8
2023-12-13 12:06:41 CET | CORE | DEBUG | (pkg/collector/worker/check_logger.go:60 in CheckFinished) | check:postgres | Done running check
2023-12-13 12:06:42 CET | CORE | TRACE | (pkg/collector/scheduler/job.go:181 in process) | Bucket ticked... current index: 0
2023-12-13 12:06:42 CET | CORE | TRACE | (pkg/collector/scheduler/job.go:197 in process) | Jobs in bucket: [disk]
2023-12-13 12:06:42 CET | CORE | DEBUG | (pkg/collector/worker/check_logger.go:43 in CheckStarted) | check:disk | Running check...

Environment: Debian Linux Stretch 64 bit, Azure VM. Running datadog agent 7.49.1

Steps to reproduce the issue:

  1. Install datadog agent and set up (successful) monitoring of Postgres database with Azure managed "Azure Database for PostgreSQL servers"
  2. Close connectivity to Azure PostgreSQL server by: removing all vnet openings, set deny public access etc so that database is unreachable.
  3. Check Datadog agent trace log and DataDog monitor of PostgreSQL integration status postgres.can_connect, result for your databse will not be there while a connection failure is expected

Expected result: expect a failure on the postgresql.can_connect result, similar to what happens here on another VM where credentials are wrong:

023-12-13 11:15:16 UTC | CORE | ERROR | (pkg/collector/python/datadog_agent.go:129 in LogMessage) | postgres:8d9a572b38a26f09 | (postgres.py:764) | Unable to collect postgres metrics.
Traceback (most recent call last):
  File "/opt/datadog-agent/embedded/lib/python3.9/site-packages/datadog_checks/postgres/postgres.py", line 744, in check    self._connect()
  File "/opt/datadog-agent/embedded/lib/python3.9/site-packages/datadog_checks/postgres/postgres.py", line 573, in _connect
    self.db = self._new_connection(self._config.dbname)
  File "/opt/datadog-agent/embedded/lib/python3.9/site-packages/datadog_checks/postgres/postgres.py", line 557, in _new_connection
    conn = psycopg2.connect(**args)
  File "/opt/datadog-agent/embedded/lib/python3.9/site-packages/psycopg2/__init__.py", line 127, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: FATAL:  password authentication failed for user "datadog"
2023-12-13 11:15:16 UTC | CORE | DEBUG | (pkg/collector/python/datadog_agent.go:135 in LogMessage) | postgres:8d9a572b38a26f09 | (postgres.py:223) | Cleaning state
2023-12-13 11:15:16 UTC | CORE | TRACE | (pkg/collector/python/memory.go:76 in MemoryTracker) | Memory Tracker - ptr: 0x7f71d007a4d0, sz: 56, op: 0
2023-12-13 11:15:16 UTC | CORE | TRACE | (pkg/collector/python/memory.go:76 in MemoryTracker) | Memory Tracker - ptr: 0x7f71d003b640, sz: 28, op: 0
2023-12-13 11:15:16 UTC | CORE | TRACE | (pkg/collector/python/memory.go:76 in MemoryTracker) | Memory Tracker - ptr: 0x7f71d0047540, sz: 11, op: 0
2023-12-13 11:15:16 UTC | CORE | TRACE | (pkg/collector/python/memory.go:76 in MemoryTracker) | Memory Tracker - ptr: 0x7f71d0023d10, sz: 37, op: 0
2023-12-13 11:15:16 UTC | CORE | TRACE | (pkg/collector/python/memory.go:76 in MemoryTracker) | Memory Tracker - ptr: 0x7f71d003ceb0, sz: 10, op: 0
2023-12-13 11:15:16 UTC | CORE | TRACE | (pkg/collector/python/memory.go:76 in MemoryTracker) | Memory Tracker - ptr: 0x7f71d003d050, sz: 16, op: 0
2023-12-13 11:15:16 UTC | CORE | TRACE | (pkg/collector/python/memory.go:76 in MemoryTracker) | Memory Tracker - ptr: 0x7f71d0094a20, sz: 99, op: 0
2023-12-13 11:15:16 UTC | CORE | TRACE | (pkg/aggregator/sender.go:346 in ServiceCheck) | Service check submitted:  postgres.can_connect :  CRITICAL  for hostname:  jumpbox.4rvjf015rjdurjvumtkvfmayzh.ax.internal.cloudapp.net  tags:  [pghost:moller-db0-prod-pgdb pgenv:prod server:db0-prod-pgdb.local.moller.no port:5432 db:smaxint_prod dd.internal.resource:database_instance:jumpbox.4rvjf015rjdurjvumtkvfmayzh.ax.internal.cloudapp.net]

Happens all the time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant