Skip to content
This repository has been archived by the owner on Oct 24, 2023. It is now read-only.

Great Expectations Error in Airflow execution #8

Open
JeroenSchmidt opened this issue Jul 8, 2020 · 1 comment
Open

Great Expectations Error in Airflow execution #8

JeroenSchmidt opened this issue Jul 8, 2020 · 1 comment

Comments

@JeroenSchmidt
Copy link

The airflow (GE) DAG is unable to successfully execute because the configs appear to be incompatible with the version of GE that is being installed within the container. See error below;

*** Reading local file: /usr/local/airflow/logs/ge_tutorials_dag_with_ge/task_validate_source_data/2020-07-08T14:23:36.810518+00:00/1.log
[2020-07-08 14:23:40,135] {{taskinstance.py:655}} INFO - Dependencies all met for <TaskInstance: ge_tutorials_dag_with_ge.task_validate_source_data 2020-07-08T14:23:36.810518+00:00 [queued]>
[2020-07-08 14:23:40,153] {{taskinstance.py:655}} INFO - Dependencies all met for <TaskInstance: ge_tutorials_dag_with_ge.task_validate_source_data 2020-07-08T14:23:36.810518+00:00 [queued]>
[2020-07-08 14:23:40,153] {{taskinstance.py:866}} INFO - 
--------------------------------------------------------------------------------
[2020-07-08 14:23:40,153] {{taskinstance.py:867}} INFO - Starting attempt 1 of 1
[2020-07-08 14:23:40,153] {{taskinstance.py:868}} INFO - 
--------------------------------------------------------------------------------
[2020-07-08 14:23:40,164] {{taskinstance.py:887}} INFO - Executing <Task(PythonOperator): task_validate_source_data> on 2020-07-08T14:23:36.810518+00:00
[2020-07-08 14:23:40,168] {{standard_task_runner.py:53}} INFO - Started process 1006 to run task
[2020-07-08 14:23:40,211] {{logging_mixin.py:112}} INFO - Running %s on host %s <TaskInstance: ge_tutorials_dag_with_ge.task_validate_source_data 2020-07-08T14:23:36.810518+00:00 [running]> 26f657a26611
[2020-07-08 14:23:40,252] {{taskinstance.py:1128}} ERROR - You appear to have an invalid config version (1.0).
    The version number must be at least 2. Please see the migration guide at https://docs.greatexpectations.io/en/latest/how_to_guides/migrating_versions.html
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 966, in _run_raw_task
    result = task_copy.execute(context=context)
  File "/usr/local/lib/python3.7/site-packages/airflow/operators/python_operator.py", line 113, in execute
    return_value = self.execute_callable()
  File "/usr/local/lib/python3.7/site-packages/airflow/operators/python_operator.py", line 118, in execute_callable
    return self.python_callable(*self.op_args, **self.op_kwargs)
  File "/usr/local/airflow/dags/ge_tutorials_dag_with_great_expectations.py", line 70, in validate_source_data
    context = ge.data_context.DataContext(great_expectations_context_path)
  File "/usr/local/lib/python3.7/site-packages/great_expectations/data_context/data_context.py", line 2148, in __init__
    project_config = self._load_project_config()
  File "/usr/local/lib/python3.7/site-packages/great_expectations/data_context/data_context.py", line 2186, in _load_project_config
    return DataContextConfig.from_commented_map(config_dict)
  File "/usr/local/lib/python3.7/site-packages/great_expectations/data_context/types/base.py", line 86, in from_commented_map
    config = dataContextConfigSchema.load(commented_map)
  File "/usr/local/lib/python3.7/site-packages/marshmallow/schema.py", line 723, in load
    data, many=many, partial=partial, unknown=unknown, postprocess=True
  File "/usr/local/lib/python3.7/site-packages/marshmallow/schema.py", line 886, in _do_load
    field_errors=field_errors,
  File "/usr/local/lib/python3.7/site-packages/marshmallow/schema.py", line 1189, in _invoke_schema_validators
    partial=partial,
  File "/usr/local/lib/python3.7/site-packages/marshmallow/schema.py", line 774, in _run_validator
    validator_func(output, partial=partial, many=many)
  File "/usr/local/lib/python3.7/site-packages/great_expectations/data_context/types/base.py", line 298, in validate_schema
    data["config_version"], MINIMUM_SUPPORTED_CONFIG_VERSION
great_expectations.exceptions.UnsupportedConfigVersionError: You appear to have an invalid config version (1.0).
    The version number must be at least 2. Please see the migration guide at https://docs.greatexpectations.io/en/latest/how_to_guides/migrating_versions.html
[2020-07-08 14:23:40,254] {{taskinstance.py:1185}} INFO - Marking task as FAILED.dag_id=ge_tutorials_dag_with_ge, task_id=task_validate_source_data, execution_date=20200708T142336, start_date=20200708T142340, end_date=20200708T142340
[2020-07-08 14:23:50,113] {{logging_mixin.py:112}} INFO - [2020-07-08 14:23:50,113] {{local_task_job.py:103}} INFO - Task exited with return code 1
@torsjonas
Copy link

This looks like a duplicate of #7

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants