-
Notifications
You must be signed in to change notification settings - Fork 123
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Can't install astronomer-cosmos[dbt-redshift]==1.3.2 in MWAA(airflow 2.8.1) #953
Comments
To address the dependency conflict issue with
This setup ensures that dbt and its dependencies are managed in isolation, mitigating the conflict and allowing for a smoother integration with MWAA.
|
Great!,It works to me.But now I got a new problem.my execution mode is kubernetes(eks 1.27),task seems not running on k8s.Pod status changes from Pending to Completed takes 0s.
Below is my dag python code.
|
The immediate completion of Kubernetes pods without executing the tasks as expected in your Airflow DAG could be due to several factors. Here's a structured approach to troubleshoot and potentially resolve the issue:
To further diagnose, delve into the Airflow task logs and Kubernetes pod logs and events for error messages or warnings that provide more insight. This targeted approach should help identify and resolve the root cause of the issue.
|
@ysjjovo what do those K8s pods logs say? |
no logs generated in K8s pod,Here is the screenshot.
Airflow Graph show All success.
|
@ysjjovo, thanks for the details! Could you confirm which version of Also, could you test if your DAG works as expected in a local k8s cluster (e.g., with Kind, as described in these docs), it would be great for us to isolate the issue from a bug specific to MWAA, or a problem in the docker image / or in Cosmos. |
I didn't specify
But I can get version from MWAA(airflow) logs.version is Here is the Dockerfile I used.
Here is the entire dbt project: https://github.com/ysjjovo/dbtcicdnew |
Evironments:
Some logs in worker thread log group 'requirements_install_ip-10-1-64-85.us-east-2.compute.internal_1715574539.5060384.log'
The text was updated successfully, but these errors were encountered: