Support log downloads for AWS Aurora Serverless v1

[eduardocalazansjr]

I'm trying to integrate the pganalyze collector to my Aurora Serverless Cluster but it can't find the db instance since the command aws rds describre-db-instances doesn't return my instance, but aws rds describe-db-clusters does so.

My collector keeps outputing

E [default] Rds/Logs: Could not find RDS instance: DBInstanceNotFound: DBInstance xxx not found.

[lfittl]

@eduardocalazansjr Thanks for reaching out!

At this moment we don't support Aurora Serverless for system metrics / log monitoring, but you can still connect over Postgres directly - simply leave out the AWS specific settings. You can add "disable_logs: 1" to your configuration to avoid the error message.

[eduardocalazansjr]

@lfittl but the 'explain plains' doesn't work too, right?

[lfittl]

@eduardocalazansjr Thats correct - EXPLAIN plans rely on Log Insights data (i.e. also needs special integration in this case)

[moltar]

At this moment we don't support Aurora Serverless for system metrics / log monitoring

What is so fundamentally different about Aurora Serverless?

Now that v2 is GA, I think it's worth re-visiting this again. (although it isn't serverless anymore, as minimum scaling setting is enforced)

[moltar]

Btw, I seem to get this on the RDS Aurora Provisioned type as well:

E [default] Could not collect logs for server: could not collect logs: Error finding RDS instance: DBInstanceNotFound: DBInstance foo-bar not found.

[lfittl]

@moltar I realize its been a while, but if you are still running into issues, this comes down to the difference between clusters and instances.

In AWS Aurora, the cluster does not have database error logs - only the individual instance. When you specify the cluster hostname as the "db_host" in pganalyze, we will try to lookup the cluster name as an instance, which fails with the error you're seeing.

The way to resolve this is to use the instance hostname instead (e.g. from the writer instance), which we'll then be able to use for downloading logs.

The good news is that Aurora Serverless v2 actually provides support for the RDS log download API - the limitation that this issue is about only affected v1 (which did not support that API, so you would have had to go through CloudWatch to get the logs - with v2 going through CloudWatch is not necessary).

Here is the config for a successful test I just ran:

[server1]
#db_host = pganalyze-serverless-v2-test.cluster-ACCOUNTID.us-east-1.rds.amazonaws.com
db_host = pganalyze-serverless-v2-test-instance-1.ACCOUNTID.us-east-1.rds.amazonaws.com
db_name = postgres
db_username = pganalyze
db_password = REMOVED
db_sslrootcert = rds-ca-2019-root
db_sslmode = verify-full

Note the commented out cluster hostname, and instead using the instance hostname. We are exploring ways to make this easier in the future, since it's a very common issue.

(updating the title of this issue to clarify the missing support only affects Aurora Serverless v1, not v2)