Nightly Build Failure 2024-04-30

[zaneselvans]

Overview

MLFlow got a database integrity error, apparently due to a non-unique primary key in a row it was attempting to insert? See detailed logs below. I haven't seen this before. Maybe it has to do with a dependency update from yesterday? Or maybe it's random.

@katherinelamb @zschira is this something you've come across before with MLFlow?

Relevant logs

gsutil cp gs://builds.catalyst.coop/2024-04-30-0602-84a669d6c-main/2024-04-30-0602-84a669d6c-main-pudl-etl.log .

2024-04-30 10:13:42 +0000 - dagster - ERROR - etl_job - b325de94-12b1-4663-a39d-30e5c2a9bd3b - 1325 - RUN_FAILURE - Execution of run for "etl_job" failed. Steps failed: ['out_pudl__yearly_assn_eia_ferc1_plant_parts.ferc_to_eia.col_cleaner_cleaner_graph_2.col_cleaner_train'].
Traceback (most recent call last):
  File "/home/mambauser/env/bin/pudl_etl", line 8, in <module>
    sys.exit(pudl_etl())
             ^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.12/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/pudl/src/pudl/etl/cli.py", line 177, in pudl_etl
    raise Exception(event.event_specific_data.error)
Exception: dagster._core.errors.DagsterExecutionStepExecutionError: Error occurred while executing op "out_pudl__yearly_assn_eia_ferc1_plant_parts.ferc_to_eia.col_cleaner_cleaner_graph_2.col_cleaner_train":

Stack Trace:
  File "/home/mambauser/env/lib/python3.12/site-packages/dagster/_core/execution/plan/execute_plan.py", line 282, in dagster_event_sequence_for_step
    for step_event in check.generator(step_events):
  File "/home/mambauser/env/lib/python3.12/site-packages/dagster/_core/execution/plan/execute_step.py", line 523, in core_dagster_event_sequence_for_step
    for user_event in _step_output_error_checked_user_event_sequence(
  File "/home/mambauser/env/lib/python3.12/site-packages/dagster/_core/execution/plan/execute_step.py", line 202, in _step_output_error_checked_user_event_sequence
    for user_event in user_event_sequence:
  File "/home/mambauser/env/lib/python3.12/site-packages/dagster/_core/execution/plan/execute_step.py", line 100, in _process_asset_results_to_events
    for user_event in user_event_sequence:
  File "/home/mambauser/env/lib/python3.12/site-packages/dagster/_core/execution/plan/compute.py", line 208, in execute_core_compute
    for step_output in _yield_compute_results(step_context, inputs, compute_fn, compute_context):
  File "/home/mambauser/env/lib/python3.12/site-packages/dagster/_core/execution/plan/compute.py", line 177, in _yield_compute_results
    for event in iterate_with_context(
  File "/home/mambauser/env/lib/python3.12/site-packages/dagster/_utils/__init__.py", line 462, in iterate_with_context
    with context_fn():
  File "/home/mambauser/env/lib/python3.12/contextlib.py", line 158, in __exit__
    self.gen.throw(value)
  File "/home/mambauser/env/lib/python3.12/site-packages/dagster/_core/execution/plan/utils.py", line 84, in op_execution_error_boundary
    raise error_cls(

The above exception was caused by the following exception:
mlflow.exceptions.MlflowException: (sqlite3.IntegrityError) UNIQUE constraint failed: params.key, params.run_uuid
[SQL: INSERT INTO params ("key", value, run_uuid) VALUES (?, ?, ?)]
[parameters: [('col_cleaner.plant_name.weight', '1.0', '70cc58eac484445aa0334b5f1441b03f'), ('col_cleaner.plant_name.columns', "['plant_name']", '70cc58eac484445aa0334b5f1441b03f'), ('col_cleaner.plant_name.name_cleaner.company_cleaner.cleaning_rules_list', "['remove_word_the_from_the_end', 'remove_word_the_from_the_beginning', 'replace_amperstand_between_space_by_AND', 'replace_hyphen_by_space', 'replace ... (225 characters truncated) ... eses', 'add_space_after_closing_parentheses', 'remove_parentheses', 'remove_brackets', 'remove_curly_brackets', 'enforce_single_space_between_words']", '70cc58eac484445aa0334b5f1441b03f'), ('col_cleaner.plant_name.name_cleaner.company_cleaner.normalize_legal_terms', 'True', '70cc58eac484445aa0334b5f1441b03f'), ('col_cleaner.plant_name.name_cleaner.company_cleaner.remove_unicode', 'False', '70cc58eac484445aa0334b5f1441b03f'), ('col_cleaner.plant_name.name_cleaner.company_cleaner.output_lettercase', 'lower', '70cc58eac484445aa0334b5f1441b03f'), ('col_cleaner.plant_name.name_cleaner.company_cleaner.legal_term_location', 'LegalTermLocation.AT_THE_END', '70cc58eac484445aa0334b5f1441b03f'), ('col_cleaner.plant_name.name_cleaner.company_cleaner.remove_accents', 'False', '70cc58eac484445aa0334b5f1441b03f')  ... displaying 10 of 28 total bound parameter sets ...  ('col_cleaner.capacity_mw.columns', "['capacity_mw']", '70cc58eac484445aa0334b5f1441b03f'), ('col_cleaner.capacity_mw.column_cleaner.cleaning_function', 'zero_to_null', '70cc58eac484445aa0334b5f1441b03f')]]
(Background on this error at: https://sqlalche.me/e/20/gkpj)

Stack Trace:
  File "/home/mambauser/env/lib/python3.12/site-packages/dagster/_core/execution/plan/utils.py", line 54, in op_execution_error_boundary
    yield
  File "/home/mambauser/env/lib/python3.12/site-packages/dagster/_utils/__init__.py", line 464, in iterate_with_context
    next_output = next(iterator)
                  ^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.12/site-packages/dagster/_core/execution/plan/compute_generator.py", line 141, in _coerce_op_compute_fn_to_iterator
    result = invoke_compute_fn(
             ^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.12/site-packages/dagster/_core/execution/plan/compute_generator.py", line 129, in invoke_compute_fn
    return fn(context, **args_to_pass) if context_arg_provided else fn(**args_to_pass)
                                                                    ^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/pudl/src/pudl/analysis/record_linkage/embed_dataframe.py", line 160, in train_dataframe_cleaner
    log_dataframe_embedder_config(name_prefix, vectorizers, experiment_tracker)
  File "/home/mambauser/pudl/src/pudl/analysis/record_linkage/embed_dataframe.py", line 103, in log_dataframe_embedder_config
    experiment_tracker.execute_logging(
  File "/home/mambauser/pudl/src/pudl/analysis/ml_tools/experiment_tracking.py", line 152, in execute_logging
    logging_func()
  File "/home/mambauser/pudl/src/pudl/analysis/record_linkage/embed_dataframe.py", line 104, in <lambda>
    lambda: mlflow.log_params(
            ^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.12/site-packages/mlflow/tracking/fluent.py", line 911, in log_params
    return MlflowClient().log_batch(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.12/site-packages/mlflow/tracking/client.py", line 1134, in log_batch
    return self._tracking_client.log_batch(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.12/site-packages/mlflow/tracking/_tracking_service/client.py", line 471, in log_batch
    self.store.log_batch(
  File "/home/mambauser/env/lib/python3.12/site-packages/mlflow/store/tracking/sqlalchemy_store.py", line 1358, in log_batch
    raise e
  File "/home/mambauser/env/lib/python3.12/site-packages/mlflow/store/tracking/sqlalchemy_store.py", line 1354, in log_batch
    self._log_params(run_id, params)
  File "/home/mambauser/env/lib/python3.12/site-packages/mlflow/store/tracking/sqlalchemy_store.py", line 1097, in _log_params
    with self.ManagedSessionMaker() as session:
  File "/home/mambauser/env/lib/python3.12/contextlib.py", line 144, in __exit__
    next(self.gen)
  File "/home/mambauser/env/lib/python3.12/site-packages/mlflow/store/db/utils.py", line 155, in make_managed_session
    raise MlflowException(message=e, error_code=BAD_REQUEST)

The above exception occurred during handling of the following exception:
sqlalchemy.exc.IntegrityError: (sqlite3.IntegrityError) UNIQUE constraint failed: params.key, params.run_uuid
[SQL: INSERT INTO params ("key", value, run_uuid) VALUES (?, ?, ?)]
[parameters: [('col_cleaner.plant_name.weight', '1.0', '70cc58eac484445aa0334b5f1441b03f'), ('col_cleaner.plant_name.columns', "['plant_name']", '70cc58eac484445aa0334b5f1441b03f'), ('col_cleaner.plant_name.name_cleaner.company_cleaner.cleaning_rules_list', "['remove_word_the_from_the_end', 'remove_word_the_from_the_beginning', 'replace_amperstand_between_space_by_AND', 'replace_hyphen_by_space', 'replace ... (225 characters truncated) ... eses', 'add_space_after_closing_parentheses', 'remove_parentheses', 'remove_brackets', 'remove_curly_brackets', 'enforce_single_space_between_words']", '70cc58eac484445aa0334b5f1441b03f'), ('col_cleaner.plant_name.name_cleaner.company_cleaner.normalize_legal_terms', 'True', '70cc58eac484445aa0334b5f1441b03f'), ('col_cleaner.plant_name.name_cleaner.company_cleaner.remove_unicode', 'False', '70cc58eac484445aa0334b5f1441b03f'), ('col_cleaner.plant_name.name_cleaner.company_cleaner.output_lettercase', 'lower', '70cc58eac484445aa0334b5f1441b03f'), ('col_cleaner.plant_name.name_cleaner.company_cleaner.legal_term_location', 'LegalTermLocation.AT_THE_END', '70cc58eac484445aa0334b5f1441b03f'), ('col_cleaner.plant_name.name_cleaner.company_cleaner.remove_accents', 'False', '70cc58eac484445aa0334b5f1441b03f')  ... displaying 10 of 28 total bound parameter sets ...  ('col_cleaner.capacity_mw.columns', "['capacity_mw']", '70cc58eac484445aa0334b5f1441b03f'), ('col_cleaner.capacity_mw.column_cleaner.cleaning_function', 'zero_to_null', '70cc58eac484445aa0334b5f1441b03f')]]
(Background on this error at: https://sqlalche.me/e/20/gkpj)

Stack Trace:
  File "/home/mambauser/env/lib/python3.12/site-packages/mlflow/store/db/utils.py", line 142, in make_managed_session
    session.commit()
  File "/home/mambauser/env/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 1972, in commit
    trans.commit(_to_root=True)
  File "<string>", line 2, in commit
  File "/home/mambauser/env/lib/python3.12/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go
    ret_value = fn(self, *arg, **kw)
                ^^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 1257, in commit
    self._prepare_impl()
  File "<string>", line 2, in _prepare_impl
  File "/home/mambauser/env/lib/python3.12/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go
    ret_value = fn(self, *arg, **kw)
                ^^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 1232, in _prepare_impl
    self.session.flush()
  File "/home/mambauser/env/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 4296, in flush
    self._flush(objects)
  File "/home/mambauser/env/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 4431, in _flush
    with util.safe_reraise():
  File "/home/mambauser/env/lib/python3.12/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/home/mambauser/env/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 4392, in _flush
    flush_context.execute()
  File "/home/mambauser/env/lib/python3.12/site-packages/sqlalchemy/orm/unitofwork.py", line 466, in execute
    rec.execute(self)
  File "/home/mambauser/env/lib/python3.12/site-packages/sqlalchemy/orm/unitofwork.py", line 642, in execute
    util.preloaded.orm_persistence.save_obj(
  File "/home/mambauser/env/lib/python3.12/site-packages/sqlalchemy/orm/persistence.py", line 93, in save_obj
    _emit_insert_statements(
  File "/home/mambauser/env/lib/python3.12/site-packages/sqlalchemy/orm/persistence.py", line 1048, in _emit_insert_statements
    result = connection.execute(
             ^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1422, in execute
    return meth(
           ^^^^^
  File "/home/mambauser/env/lib/python3.12/site-packages/sqlalchemy/sql/elements.py", line 514, in _execute_on_connection
    return connection._execute_clauseelement(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1644, in _execute_clauseelement
    ret = self._execute_context(
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1850, in _execute_context
    return self._exec_single_context(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mambauser/env/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1990, in _exec_single_context
    self._handle_dbapi_exception(
  File "/home/mambauser/env/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 2357, in _handle_dbapi_exception
    raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
  File "/home/mambauser/env/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1940, in _exec_single_context
    self.dialect.do_executemany(
  File "/home/mambauser/env/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 916, in do_executemany
    cursor.executemany(statement, parameters)

The above exception was caused by the following exception:
sqlite3.IntegrityError: UNIQUE constraint failed: params.key, params.run_uuid

Stack Trace:
  File "/home/mambauser/env/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1940, in _exec_single_context
    self.dialect.do_executemany(
  File "/home/mambauser/env/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 916, in do_executemany
    cursor.executemany(statement, parameters)

Copying outputs to GCP bucket gs://builds.catalyst.coop/2024-04-30-0602-84a669d6c-main

[zaneselvans]

@zschira are we still trying to address this? Or is it intermittent enough that we'd like to ignore and close this issue for now?

[zaneselvans]

Okay, closing this since it's thankfully not recurred. If it becomes a real problem we can create a new build failure issue and address it.