Replies: 2 comments 2 replies
-
You should raise it to MWAA team. It's hard to say what the reason is, but logging configuration is done by the MWAA team and they are the best to help you (and undoubtedly you have support agreement for the paid service you use, so they should be able to investigate and help). |
Beta Was this translation helpful? Give feedback.
-
We were able to get the logging to work by adding the following code at the beginning (so run as part of the top level) import logging
logging.getLogger("airflow.providers.postgres.hooks.postgres.PostgresHook").setLevel(logging.INFO) I was wondering if the code fix we have figured out above, gives anyone an idea as to what in the AWS infrastructure code (or maybe its in Airflow code) might be causing this behavior. |
Beta Was this translation helpful? Give feedback.
-
Apache Airflow version
Other Airflow 2 version (please specify below)
If "Other Airflow 2 version" selected, which one?
2.6.3
What happened?
After upgrading to MWAA 2.6.3, we have found that some of the logs that we used to get previous suddenly stopped showing up. We believe we have isolated it to calling sql hook methods that execute a query, which then causes task logs to stop showing up.
This the logs we expect:
The important pieces are
[2024-03-28, 15:58:11 MDT] {{sql.py:374}} INFO - Running statement: SELECT now(), parameters: None
[2024-03-28, 15:58:11 MDT] {{sql.py:383}} INFO - Rows affected: 1
These 2 lines disappear, if we call pg_hook.run or get_records as top-level code. In the following code, if I were to uncomment the following line, in the code that follows, then the sql.py logs stop showing up!:
This used to work correctly in the previous version of MWAA we had. But seems to have started after we upgraded to 2.6.3.
The postgresql provider is: apache-airflow-providers-postgres 5.5.1.
Is there a quick fix for this issue? Can I enable the logging within this dag? I have tried setting log_sql=True when creating a pg_hook, but that did not help.
What you think should happen instead?
The sql logs should still be captured even if the sql code is run from the top-level code of the DAG.
How to reproduce
DAG code to repro this issue has been provided above.
Uncomment the line #rows_d = pg_hook.get_records(query) to repro the behavior.
Operating System
AWS MWAA
Versions of Apache Airflow Providers
apache-airflow-providers-postgres 5.5.1
apache-airflow-providers-common-sql 1.5.2
Deployment
Amazon (AWS) MWAA
Deployment details
No response
Anything else?
No response
Are you willing to submit PR?
Code of Conduct
Beta Was this translation helpful? Give feedback.
All reactions