Skip to content
This repository has been archived by the owner on Dec 29, 2022. It is now read-only.

airflow 1.10+ broken with cloudsql(mysql) explicit_defaults_for_timestamp=off #48

Open
barney-s opened this issue Jan 28, 2019 · 2 comments

Comments

@barney-s
Copy link
Contributor

Since 1.10.0, Airflow requires MySQL to have explicit_defaults_for_timestamp flag set to ON.
CloudSQL can only support MySQL with explicit_defaults_for_timestamp=OFF for now.

@barney-s
Copy link
Contributor Author

Seeing this error:

2019-01-28 03:50:50,394] {{__init__.py:51}} INFO - Using executor CeleryExecutor
DB: mysql://v9dbimx6cr3mbgzw:***@cc-base-sql:3306/e9qcv7p27gi36b5m
[2019-01-28 03:50:50,676] {{db.py:338}} INFO - Creating tables
INFO  [alembic.runtime.migration] Context impl MySQLImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade d2ae31099d61 -> 0e2a74e0fc9f, Add time zone awareness
Traceback (most recent call last):
  File "/usr/local/bin/airflow", line 32, in <module>
    args.func(args)
  File "/usr/local/lib/python3.6/site-packages/airflow/bin/cli.py", line 1011, in initdb
    db_utils.initdb(settings.RBAC)
  File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 92, in initdb
    upgradedb()
  File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 346, in upgradedb
    command.upgrade(config, 'heads')
  File "/usr/local/lib/python3.6/site-packages/alembic/command.py", line 174, in upgrade
    script.run_env()
  File "/usr/local/lib/python3.6/site-packages/alembic/script/base.py", line 416, in run_env
    util.load_python_file(self.dir, 'env.py')
  File "/usr/local/lib/python3.6/site-packages/alembic/util/pyfiles.py", line 93, in load_python_file
    module = load_module_py(module_id, path)
  File "/usr/local/lib/python3.6/site-packages/alembic/util/compat.py", line 68, in load_module_py
    module_id, path).load_module(module_id)
  File "<frozen importlib._bootstrap_external>", line 399, in _check_name_wrapper
  File "<frozen importlib._bootstrap_external>", line 823, in load_module
  File "<frozen importlib._bootstrap_external>", line 682, in load_module
  File "<frozen importlib._bootstrap>", line 265, in _load_module_shim
  File "<frozen importlib._bootstrap>", line 684, in _load
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/local/lib/python3.6/site-packages/airflow/migrations/env.py", line 91, in <module>
    run_migrations_online()
  File "/usr/local/lib/python3.6/site-packages/airflow/migrations/env.py", line 86, in run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/usr/local/lib/python3.6/site-packages/alembic/runtime/environment.py", line 807, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/usr/local/lib/python3.6/site-packages/alembic/runtime/migration.py", line 321, in run_migrations
    step.migration_fn(**kw)
  File "/usr/local/lib/python3.6/site-packages/airflow/migrations/versions/0e2a74e0fc9f_add_time_zone_awareness.py", line 46, in upgrade
    raise Exception("Global variable explicit_defaults_for_timestamp needs to be on (1) for mysql")
Exception: Global variable explicit_defaults_for_timestamp needs to be on (1) for mysql
[2019-01-28 03:50:51,465] {{settings.py:174}} INFO - setting.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800
[2019-01-28 03:50:51,948] {{__init__.py:51}} INFO - Using executor CeleryExecutor

@barney-s
Copy link
Contributor Author

Update:
Google Composer patched airflow to work around this issue. Will wait for that to be open-sourced in airflow.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant