Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for Mysql Backed airflow. #139

Open
amolsr opened this issue Jan 23, 2024 · 1 comment
Open

Add support for Mysql Backed airflow. #139

amolsr opened this issue Jan 23, 2024 · 1 comment

Comments

@amolsr
Copy link

amolsr commented Jan 23, 2024

image

@amolsr
Copy link
Author

amolsr commented Jan 23, 2024

[2024-01-23, 11:30:29 IST] {airflow-db-cleanup.py:293} INFO - Retrieving max_execution_date from XCom
[2024-01-23, 11:30:29 IST] {airflow-db-cleanup.py:306} INFO - Configurations:
[2024-01-23, 11:30:29 IST] {airflow-db-cleanup.py:307} INFO - max_date: 2024-01-13 05:59:25.875810+00:00
[2024-01-23, 11:30:29 IST] {airflow-db-cleanup.py:308} INFO - enable_delete: True
[2024-01-23, 11:30:29 IST] {airflow-db-cleanup.py:309} INFO - session: <sqlalchemy.orm.session.Session object at 0x7f618406e910>
[2024-01-23, 11:30:29 IST] {airflow-db-cleanup.py:310} INFO - _db_model: <class '.models.taskinstance.TaskInstance'>
[2024-01-23, 11:30:29 IST] {airflow-db-cleanup.py:311} INFO - state: None
[2024-01-23, 11:30:29 IST] {airflow-db-cleanup.py:312} INFO - age_check_column: ColumnAssociationProxyInstance(AssociationProxy('dag_run', 'execution_date'))
[2024-01-23, 11:30:29 IST] {airflow-db-cleanup.py:313} INFO - keep_last: False
[2024-01-23, 11:30:29 IST] {airflow-db-cleanup.py:314} INFO - keep_last_filters: None
[2024-01-23, 11:30:29 IST] {airflow-db-cleanup.py:315} INFO - keep_last_group_by: None
[2024-01-23, 11:30:29 IST] {airflow-db-cleanup.py:317} INFO -
[2024-01-23, 11:30:29 IST] {airflow-db-cleanup.py:319} INFO - Running Cleanup Process...
[2024-01-23, 11:30:29 IST] {taskinstance.py:1775} ERROR - Task failed with exception
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/operators/python.py", line 175, in execute
return_value = self.execute_callable()
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/operators/python.py", line 192, in execute_callable
return self.python_callable(*self.op_args, **self.op_kwargs)
File "/opt/airflow/dags/maintainance/airflow-db-cleanup.py", line 324, in cleanup_function
logging.info("INITIAL QUERY : " + str(query))
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 2946, in str
return str(statement.compile(bind))
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 503, in compile
return self._compiler(dialect, *kw)
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 567, in _compiler
return dialect.statement_compiler(dialect, self, kw)
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 809, in init
Compiled.init(self, dialect, statement, kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 464, in init
self.string = self.process(self.statement, compile_kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 499, in process
return obj._compiler_dispatch(self, kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/sql/visitors.py", line 82, in _compiler_dispatch
return meth(self, kw)
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 3406, in visit_select
select_stmt, self, kwargs
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/sql/base.py", line 510, in create_for_statement
return klass.create_for_statement(statement, compiler, kw)
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/context.py", line 736, in create_for_statement
opt.process_compile_state(self)
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/strategy_options.py", line 267, in process_compile_state
and not compile_state.compile_options._for_refresh_state,
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/strategy_options.py", line 924, in _process
raiseerr,
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/strategy_options.py", line 1085, in _bind_loader
"mapper option expects " "string key or list of attributes"
sqlalchemy.exc.ArgumentError: mapper option expects string key or list of attributes
[2024-01-23, 11:30:29 IST] {taskinstance.py:1326} INFO - Marking task as FAILED. dag_id=
-db-cleanup, task_id=cleanup_TaskInstance, execution_date=20240122T000000, start_date=20240123T060029, end_date=20240123T060029
[2024-01-23, 11:30:29 IST] {logging_mixin.py:137} WARNING - /home/
/.local/lib/python3.7/site-packages/
/models/param.py:74 RemovedInAirflow3Warning: The use of non-json-serializable params is deprecated and will be removed in a future release
[2024-01-23, 11:30:29 IST] {logging_mixin.py:137} WARNING - /home/
/.local/lib/python3.7/site-packages/
/utils/email.py:152 RemovedInAirflow3Warning: Fetching SMTP credentials from configuration variables will be deprecated in a future release. Please set credentials using a connection instead.
[2024-01-23, 11:30:29 IST] {email.py:268} INFO - Email alerting: attempt 1
[2024-01-23, 11:30:30 IST] {email.py:280} INFO - Sent an alert email to ['amol.saini@infoedge.com', 'sumit.k1@gmail.com']
[2024-01-23, 11:30:30 IST] {standard_task_runner.py:105} ERROR - Failed to execute job 393423 for task cleanup_TaskInstance (mapper option expects string key or list of attributes; 32510)
[2024-01-23, 11:30:30 IST] {local_task_job.py:212} INFO - Task exited with return code 1
[2024-01-23, 11:30:30 IST] {taskinstance.py:2585} INFO - 0 downstream tasks scheduled from follow-on schedule check

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant