You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{"log":"{\"asctime\": \"2022-04-04 06:50:17,140\", \"levelname\": \"ERROR\", \"name\": \"chaos_genius.controllers.data_source_metadata_controller\", \"message\": \"Error in metadata prefetch.\", \"lineno\": 160, \"funcName\": \"run_metadata_prefetch\", \"filename\": \"data_source_metadata_controller.py\", \"exc_info\": \"Traceback (most recent call last):\\n File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py\\\", line 1770, in _execute_context\\n self.dialect.do_execute(\\n File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py\\\", line 717, in do_execute\\n cursor.execute(statement, parameters)\\npsycopg2.OperationalError: SSL SYSCALL error: EOF detected\\n\\n\\nThe above exception was the direct cause of the following exception:\\n\\nTraceback (most recent call last):\\n File \\\"/usr/src/app/chaos_genius/controllers/data_source_metadata_controller.py\\\", line 146, in run_metadata_prefetch\\n _ = scan_db_and_save_table_info(\\n File \\\"/usr/src/app/chaos_genius/controllers/data_source_metadata_controller.py\\\", line 208, in scan_db_and_save_table_info\\n table_info = get_table_info(\\n File \\\"/usr/src/app/chaos_genius/connectors/__init__.py\\\", line 80, in get_table_info\\n table_info[\\\"columns\\\"] = db_connection.get_columns(table_name, schema)\\n File \\\"/usr/src/app/chaos_genius/connectors/base_db.py\\\", line 114, in get_columns\\n db_columns = self.inspector.get_columns(table_name=use_table, schema=use_schema)\\n File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/reflection.py\\\", line 497, in get_columns\\n col_defs = self.dialect.get_columns(\\n File \\\"\u003cstring\u003e\\\", line 2, in get_columns\\n File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/reflection.py\\\", line 55, in cache\\n ret = fn(self, con, *args, **kw)\\n File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/dialects/postgresql/base.py\\\", line 3652, in get_columns\\n domains = self._load_domains(connection)\\n File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/dialects/postgresql/base.py\\\", line 4420, in _load_domains\\n c = connection.execution_options(future_result=True).execute(s)\\n File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py\\\", line 1262, in execute\\n return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)\\n File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py\\\", line 329, in _execute_on_connection\\n return connection._execute_clauseelement(\\n File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py\\\", line 1451, in _execute_clauseelement\\n ret = self._execute_context(\\n File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py\\\", line 1813, in _execute_context\\n self._handle_dbapi_exception(\\n File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py\\\", line 1994, in _handle_dbapi_exception\\n util.raise_(\\n File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py\\\", line 211, in raise_\\n raise exception\\n File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py\\\", line 1770, in _execute_context\\n self.dialect.do_execute(\\n File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py\\\", line 717, in do_execute\\n cursor.execute(statement, parameters)\\nsqlalchemy.exc.OperationalError: (psycopg2.OperationalError) SSL SYSCALL error: EOF detected\\n\\n[SQL: \\n SELECT t.typname as \\\"name\\\",\\n pg_catalog.format_type(t.typbasetype, t.typtypmod) as \\\"attype\\\",\\n not t.typnotnull as \\\"nullable\\\",\\n t.typdefault as \\\"default\\\",\\n pg_catalog.pg_type_is_visible(t.oid) as \\\"visible\\\",\\n n.nspname as \\\"schema\\\"\\n FROM pg_catalog.pg_type t\\n LEFT JOIN pg_catalog.pg_namespace n ON n.oid = t.typnamespace\\n WHERE t.typtype = 'd'\\n ]\\n(Background on this error at: http://sqlalche.me/e/14/e3q8)\"}\n","stream":"stderr","time":"2022-04-04T06:50:17.141336642Z"}
{"log":"[2022-04-04 06:50:17,140: ERROR/ForkPoolWorker-2] Error in metadata prefetch.\n","stream":"stderr","time":"2022-04-04T06:50:17.141533862Z"}
{"log":"Traceback (most recent call last):\n","stream":"stderr","time":"2022-04-04T06:50:17.141543748Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py\", line 1770, in _execute_context\n","stream":"stderr","time":"2022-04-04T06:50:17.141547734Z"}
{"log":" self.dialect.do_execute(\n","stream":"stderr","time":"2022-04-04T06:50:17.141552351Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py\", line 717, in do_execute\n","stream":"stderr","time":"2022-04-04T06:50:17.141556126Z"}
{"log":" cursor.execute(statement, parameters)\n","stream":"stderr","time":"2022-04-04T06:50:17.141560305Z"}
{"log":"psycopg2.OperationalError: SSL SYSCALL error: EOF detected\n","stream":"stderr","time":"2022-04-04T06:50:17.141563804Z"}
{"log":"\n","stream":"stderr","time":"2022-04-04T06:50:17.141567412Z"}
{"log":"\n","stream":"stderr","time":"2022-04-04T06:50:17.141571224Z"}
{"log":"The above exception was the direct cause of the following exception:\n","stream":"stderr","time":"2022-04-04T06:50:17.141575141Z"}
{"log":"\n","stream":"stderr","time":"2022-04-04T06:50:17.141579137Z"}
{"log":"Traceback (most recent call last):\n","stream":"stderr","time":"2022-04-04T06:50:17.141582877Z"}
{"log":" File \"/usr/src/app/chaos_genius/controllers/data_source_metadata_controller.py\", line 146, in run_metadata_prefetch\n","stream":"stderr","time":"2022-04-04T06:50:17.141586651Z"}
{"log":" _ = scan_db_and_save_table_info(\n","stream":"stderr","time":"2022-04-04T06:50:17.141590788Z"}
{"log":" File \"/usr/src/app/chaos_genius/controllers/data_source_metadata_controller.py\", line 208, in scan_db_and_save_table_info\n","stream":"stderr","time":"2022-04-04T06:50:17.14159453Z"}
{"log":" table_info = get_table_info(\n","stream":"stderr","time":"2022-04-04T06:50:17.141598702Z"}
{"log":" File \"/usr/src/app/chaos_genius/connectors/__init__.py\", line 80, in get_table_info\n","stream":"stderr","time":"2022-04-04T06:50:17.141602506Z"}
{"log":" table_info[\"columns\"] = db_connection.get_columns(table_name, schema)\n","stream":"stderr","time":"2022-04-04T06:50:17.141606394Z"}
{"log":" File \"/usr/src/app/chaos_genius/connectors/base_db.py\", line 114, in get_columns\n","stream":"stderr","time":"2022-04-04T06:50:17.141610367Z"}
{"log":" db_columns = self.inspector.get_columns(table_name=use_table, schema=use_schema)\n","stream":"stderr","time":"2022-04-04T06:50:17.14161443Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/reflection.py\", line 497, in get_columns\n","stream":"stderr","time":"2022-04-04T06:50:17.141618523Z"}
{"log":" col_defs = self.dialect.get_columns(\n","stream":"stderr","time":"2022-04-04T06:50:17.141629696Z"}
{"log":" File \"\u003cstring\u003e\", line 2, in get_columns\n","stream":"stderr","time":"2022-04-04T06:50:17.141633527Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/reflection.py\", line 55, in cache\n","stream":"stderr","time":"2022-04-04T06:50:17.141637588Z"}
{"log":" ret = fn(self, con, *args, **kw)\n","stream":"stderr","time":"2022-04-04T06:50:17.141641448Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/dialects/postgresql/base.py\", line 3652, in get_columns\n","stream":"stderr","time":"2022-04-04T06:50:17.141645024Z"}
{"log":" domains = self._load_domains(connection)\n","stream":"stderr","time":"2022-04-04T06:50:17.141648894Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/dialects/postgresql/base.py\", line 4420, in _load_domains\n","stream":"stderr","time":"2022-04-04T06:50:17.14165258Z"}
{"log":" c = connection.execution_options(future_result=True).execute(s)\n","stream":"stderr","time":"2022-04-04T06:50:17.141656908Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py\", line 1262, in execute\n","stream":"stderr","time":"2022-04-04T06:50:17.14166099Z"}
{"log":" return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)\n","stream":"stderr","time":"2022-04-04T06:50:17.141665103Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py\", line 329, in _execute_on_connection\n","stream":"stderr","time":"2022-04-04T06:50:17.141669292Z"}
{"log":" return connection._execute_clauseelement(\n","stream":"stderr","time":"2022-04-04T06:50:17.141673477Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py\", line 1451, in _execute_clauseelement\n","stream":"stderr","time":"2022-04-04T06:50:17.141677314Z"}
{"log":" ret = self._execute_context(\n","stream":"stderr","time":"2022-04-04T06:50:17.141681283Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py\", line 1813, in _execute_context\n","stream":"stderr","time":"2022-04-04T06:50:17.141685087Z"}
{"log":" self._handle_dbapi_exception(\n","stream":"stderr","time":"2022-04-04T06:50:17.14168916Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py\", line 1994, in _handle_dbapi_exception\n","stream":"stderr","time":"2022-04-04T06:50:17.141692846Z"}
{"log":" util.raise_(\n","stream":"stderr","time":"2022-04-04T06:50:17.141697078Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py\", line 211, in raise_\n","stream":"stderr","time":"2022-04-04T06:50:17.141700527Z"}
{"log":" raise exception\n","stream":"stderr","time":"2022-04-04T06:50:17.14170438Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py\", line 1770, in _execute_context\n","stream":"stderr","time":"2022-04-04T06:50:17.14170792Z"}
{"log":" self.dialect.do_execute(\n","stream":"stderr","time":"2022-04-04T06:50:17.141712173Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py\", line 717, in do_execute\n","stream":"stderr","time":"2022-04-04T06:50:17.141716237Z"}
{"log":" cursor.execute(statement, parameters)\n","stream":"stderr","time":"2022-04-04T06:50:17.141720096Z"}
{"log":"sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) SSL SYSCALL error: EOF detected\n","stream":"stderr","time":"2022-04-04T06:50:17.141723897Z"}
{"log":"\n","stream":"stderr","time":"2022-04-04T06:50:17.14172794Z"}
{"log":"[SQL: \n","stream":"stderr","time":"2022-04-04T06:50:17.141731442Z"}
{"log":" SELECT t.typname as \"name\",\n","stream":"stderr","time":"2022-04-04T06:50:17.141735092Z"}
{"log":" pg_catalog.format_type(t.typbasetype, t.typtypmod) as \"attype\",\n","stream":"stderr","time":"2022-04-04T06:50:17.141738919Z"}
{"log":" not t.typnotnull as \"nullable\",\n","stream":"stderr","time":"2022-04-04T06:50:17.141743131Z"}
{"log":" t.typdefault as \"default\",\n","stream":"stderr","time":"2022-04-04T06:50:17.141751647Z"}
{"log":" pg_catalog.pg_type_is_visible(t.oid) as \"visible\",\n","stream":"stderr","time":"2022-04-04T06:50:17.141755847Z"}
{"log":" n.nspname as \"schema\"\n","stream":"stderr","time":"2022-04-04T06:50:17.141759879Z"}
{"log":" FROM pg_catalog.pg_type t\n","stream":"stderr","time":"2022-04-04T06:50:17.14176358Z"}
{"log":" LEFT JOIN pg_catalog.pg_namespace n ON n.oid = t.typnamespace\n","stream":"stderr","time":"2022-04-04T06:50:17.141767302Z"}
{"log":" WHERE t.typtype = 'd'\n","stream":"stderr","time":"2022-04-04T06:50:17.141771547Z"}
{"log":" ]\n","stream":"stderr","time":"2022-04-04T06:50:17.141775504Z"}
{"log":"(Background on this error at: http://sqlalche.me/e/14/e3q8)\n","stream":"stderr","time":"2022-04-04T06:50:17.141779035Z"}
{"log":"[2022-04-04 06:50:17,145: INFO/ForkPoolWorker-2] Task chaos_genius.jobs.metadata_prefetch.fetch_data_source_schema[c22d705d-20dc-4b43-9d1a-39671670cfe8] succeeded in 66375.81781294598s: False\n","stream":"stderr","time":"2022-04-04T06:50:17.146113272Z"}
{"log":"[2022-04-04 06:50:43,925: INFO/MainProcess] Task chaos_genius.jobs.alert_tasks.alert_digest_daily_scheduler[a7530551-fe8c-4f26-971e-32a4ed8bd942] received\n","stream":"stderr","time":"2022-04-04T06:50:43.926092968Z"}
{"log":"[2022-04-04 06:50:43,928: INFO/ForkPoolWorker-1] Task chaos_genius.jobs.alert_tasks.alert_digest_daily_scheduler[a7530551-fe8c-4f26-971e-32a4ed8bd942] succeeded in 0.0022775530233047903s: None\n","stream":"stderr","time":"2022-04-04T06:50:43.929034972Z"}
the database system is in recovery mode
{"log":"{\"asctime\": \"2022-04-05 02:33:59,754\", \"levelname\": \"ERROR\", \"name\": \"chaos_genius.controllers.data_source_metadata_controller\", \"message\": \"Error in metadata prefetch.\", \"lineno\": 160, \"funcName\": \"run_metadata_prefetch\", \"filename\": \"data_source_metadata_controller.py\", \"exc_info\": \"Traceback (most recent call last):\\n File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py\\\", line 3212, in _wrap_pool_connect\\n return fn()\\n File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py\\\", line 301, in connect\\n return _ConnectionFairy._checkout(self)\\n File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py\\\", line 761, in _checkout\\n fairy = _ConnectionRecord.checkout(pool)\\n File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py\\\", line 419, in checkout\\n rec = pool._do_get()\\n File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/impl.py\\\", line 145, in _do_get\\n self._dec_overflow()\\n File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py\\\", line 70, in __exit__\\n compat.raise_(\\n File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py\\\", line 211, in raise_\\n raise exception\\n File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/impl.py\\\", line 142, in _do_get\\n return self._create_connection()\\n File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py\\\", line 247, in _create_connection\\n return _ConnectionRecord(self)\\n File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py\\\", line 362, in __init__\\n self.__connect()\\n File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py\\\", line 605, in __connect\\n pool.logger.debug(\\\"Error on connect(): %s\\\", e)\\n File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py\\\", line 70, in __exit__\\n compat.raise_(\\n File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py\\\", line 211, in raise_\\n raise exception\\n File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py\\\", line 599, in __connect\\n connection = pool._invoke_creator(self)\\n File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/create.py\\\", line 578, in connect\\n return dialect.connect(*cargs, **cparams)\\n File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py\\\", line 584, in connect\\n return self.dbapi.connect(*cargs, **cparams)\\n File \\\"/usr/local/lib/python3.8/dist-packages/psycopg2/__init__.py\\\", line 127, in connect\\n conn = _connect(dsn, connection_factory=connection_factory, **kwasync)\\npsycopg2.OperationalError: FATAL: the database system is in recovery mode\\nFATAL: the database system is in recovery mode\\n\\n\\nThe above exception was the direct cause of the following exception:\\n\\nTraceback (most recent call last):\\n File \\\"/usr/src/app/chaos_genius/controllers/data_source_metadata_controller.py\\\", line 138, in run_metadata_prefetch\\n schema_list, old_schemas_list = scan_db_and_save_schema_list(\\n File \\\"/usr/src/app/chaos_genius/controllers/data_source_metadata_controller.py\\\", line 172, in scan_db_and_save_schema_list\\n schema_list = get_schema_names({}, from_db_conn=True, db_conn=db_connection)\\n File \\\"/usr/src/app/chaos_genius/connectors/__init__.py\\\", line 93, in get_schema_names\\n db_connection.init_inspector()\\n File \\\"/usr/src/app/chaos_genius/connectors/base_db.py\\\", line 33, in init_inspector\\n self.inspector = SQLAlchemy().inspect(self.engine)\\n File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/inspection.py\\\", line 64, in inspect\\n ret = reg(subject)\\n File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/reflection.py\\\", line 182, in _engine_insp\\n return Inspector._construct(Inspector._init_engine, bind)\\n File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/reflection.py\\\", line 117, in _construct\\n init(self, bind)\\n File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/reflection.py\\\", line 128, in _init_engine\\n engine.connect().close()\\n File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py\\\", line 3166, in connect\\n return self._connection_cls(self, close_with_result=close_with_result)\\n File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py\\\", line 96, in __init__\\n else engine.raw_connection()\\n File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py\\\", line 3245, in raw_connection\\n return self._wrap_pool_connect(self.pool.connect, _connection)\\n File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py\\\", line 3215, in _wrap_pool_connect\\n Connection._handle_dbapi_exception_noconnection(\\n File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py\\\", line 2068, in _handle_dbapi_exception_noconnection\\n util.raise_(\\n File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py\\\", line 211, in raise_\\n raise exception\\n File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py\\\", line 3212, in _wrap_pool_connect\\n return fn()\\n File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py\\\", line 301, in connect\\n return _ConnectionFairy._checkout(self)\\n File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py\\\", line 761, in _checkout\\n fairy = _ConnectionRecord.checkout(pool)\\n File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py\\\", line 419, in checkout\\n rec = pool._do_get()\\n File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/impl.py\\\", line 145, in _do_get\\n self._dec_overflow()\\n File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py\\\", line 70, in __exit__\\n compat.raise_(\\n File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py\\\", line 211, in raise_\\n raise exception\\n File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/impl.py\\\", line 142, in _do_get\\n return self._create_connection()\\n File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py\\\", line 247, in _create_connection\\n return _ConnectionRecord(self)\\n File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py\\\", line 362, in __init__\\n self.__connect()\\n File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py\\\", line 605, in __connect\\n pool.logger.debug(\\\"Error on connect(): %s\\\", e)\\n File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py\\\", line 70, in __exit__\\n compat.raise_(\\n File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py\\\", line 211, in raise_\\n raise exception\\n File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py\\\", line 599, in __connect\\n connection = pool._invoke_creator(self)\\n File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/create.py\\\", line 578, in connect\\n return dialect.connect(*cargs, **cparams)\\n File \\\"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py\\\", line 584, in connect\\n return self.dbapi.connect(*cargs, **cparams)\\n File \\\"/usr/local/lib/python3.8/dist-packages/psycopg2/__init__.py\\\", line 127, in connect\\n conn = _connect(dsn, connection_factory=connection_factory, **kwasync)\\nsqlalchemy.exc.OperationalError: (psycopg2.OperationalError) FATAL: the database system is in recovery mode\\nFATAL: the database system is in recovery mode\\n\\n(Background on this error at: http://sqlalche.me/e/14/e3q8)\"}\n","stream":"stderr","time":"2022-04-05T02:33:59.757577703Z"}
{"log":"[2022-04-05 02:33:59,754: ERROR/ForkPoolWorker-1] Error in metadata prefetch.\n","stream":"stderr","time":"2022-04-05T02:33:59.757919714Z"}
{"log":"Traceback (most recent call last):\n","stream":"stderr","time":"2022-04-05T02:33:59.757927652Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py\", line 3212, in _wrap_pool_connect\n","stream":"stderr","time":"2022-04-05T02:33:59.757930992Z"}
{"log":" return fn()\n","stream":"stderr","time":"2022-04-05T02:33:59.757934198Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py\", line 301, in connect\n","stream":"stderr","time":"2022-04-05T02:33:59.757937062Z"}
{"log":" return _ConnectionFairy._checkout(self)\n","stream":"stderr","time":"2022-04-05T02:33:59.757940085Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py\", line 761, in _checkout\n","stream":"stderr","time":"2022-04-05T02:33:59.757942904Z"}
{"log":" fairy = _ConnectionRecord.checkout(pool)\n","stream":"stderr","time":"2022-04-05T02:33:59.757946236Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py\", line 419, in checkout\n","stream":"stderr","time":"2022-04-05T02:33:59.75794905Z"}
{"log":" rec = pool._do_get()\n","stream":"stderr","time":"2022-04-05T02:33:59.757952031Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/impl.py\", line 145, in _do_get\n","stream":"stderr","time":"2022-04-05T02:33:59.757955043Z"}
{"log":" self._dec_overflow()\n","stream":"stderr","time":"2022-04-05T02:33:59.757964155Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py\", line 70, in __exit__\n","stream":"stderr","time":"2022-04-05T02:33:59.757968649Z"}
{"log":" compat.raise_(\n","stream":"stderr","time":"2022-04-05T02:33:59.757973145Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py\", line 211, in raise_\n","stream":"stderr","time":"2022-04-05T02:33:59.757976983Z"}
{"log":" raise exception\n","stream":"stderr","time":"2022-04-05T02:33:59.757980365Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/impl.py\", line 142, in _do_get\n","stream":"stderr","time":"2022-04-05T02:33:59.757983162Z"}
{"log":" return self._create_connection()\n","stream":"stderr","time":"2022-04-05T02:33:59.757986062Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py\", line 247, in _create_connection\n","stream":"stderr","time":"2022-04-05T02:33:59.757988908Z"}
{"log":" return _ConnectionRecord(self)\n","stream":"stderr","time":"2022-04-05T02:33:59.75799222Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py\", line 362, in __init__\n","stream":"stderr","time":"2022-04-05T02:33:59.757995013Z"}
{"log":" self.__connect()\n","stream":"stderr","time":"2022-04-05T02:33:59.757997898Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py\", line 605, in __connect\n","stream":"stderr","time":"2022-04-05T02:33:59.758000692Z"}
{"log":" pool.logger.debug(\"Error on connect(): %s\", e)\n","stream":"stderr","time":"2022-04-05T02:33:59.758003723Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py\", line 70, in __exit__\n","stream":"stderr","time":"2022-04-05T02:33:59.758006669Z"}
{"log":" compat.raise_(\n","stream":"stderr","time":"2022-04-05T02:33:59.758009598Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py\", line 211, in raise_\n","stream":"stderr","time":"2022-04-05T02:33:59.758012327Z"}
{"log":" raise exception\n","stream":"stderr","time":"2022-04-05T02:33:59.758015338Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py\", line 599, in __connect\n","stream":"stderr","time":"2022-04-05T02:33:59.758018102Z"}
{"log":" connection = pool._invoke_creator(self)\n","stream":"stderr","time":"2022-04-05T02:33:59.758021086Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/create.py\", line 578, in connect\n","stream":"stderr","time":"2022-04-05T02:33:59.758023902Z"}
{"log":" return dialect.connect(*cargs, **cparams)\n","stream":"stderr","time":"2022-04-05T02:33:59.75802687Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py\", line 584, in connect\n","stream":"stderr","time":"2022-04-05T02:33:59.758030112Z"}
{"log":" return self.dbapi.connect(*cargs, **cparams)\n","stream":"stderr","time":"2022-04-05T02:33:59.758033124Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/psycopg2/__init__.py\", line 127, in connect\n","stream":"stderr","time":"2022-04-05T02:33:59.75803641Z"}
{"log":" conn = _connect(dsn, connection_factory=connection_factory, **kwasync)\n","stream":"stderr","time":"2022-04-05T02:33:59.758040866Z"}
{"log":"psycopg2.OperationalError: FATAL: the database system is in recovery mode\n","stream":"stderr","time":"2022-04-05T02:33:59.758045144Z"}
{"log":"FATAL: the database system is in recovery mode\n","stream":"stderr","time":"2022-04-05T02:33:59.758049064Z"}
{"log":"\n","stream":"stderr","time":"2022-04-05T02:33:59.758052993Z"}
{"log":"\n","stream":"stderr","time":"2022-04-05T02:33:59.758055816Z"}
{"log":"The above exception was the direct cause of the following exception:\n","stream":"stderr","time":"2022-04-05T02:33:59.758058511Z"}
{"log":"\n","stream":"stderr","time":"2022-04-05T02:33:59.75806138Z"}
{"log":"Traceback (most recent call last):\n","stream":"stderr","time":"2022-04-05T02:33:59.758064094Z"}
{"log":" File \"/usr/src/app/chaos_genius/controllers/data_source_metadata_controller.py\", line 138, in run_metadata_prefetch\n","stream":"stderr","time":"2022-04-05T02:33:59.758070206Z"}
{"log":" schema_list, old_schemas_list = scan_db_and_save_schema_list(\n","stream":"stderr","time":"2022-04-05T02:33:59.758073296Z"}
{"log":" File \"/usr/src/app/chaos_genius/controllers/data_source_metadata_controller.py\", line 172, in scan_db_and_save_schema_list\n","stream":"stderr","time":"2022-04-05T02:33:59.75807616Z"}
{"log":" schema_list = get_schema_names({}, from_db_conn=True, db_conn=db_connection)\n","stream":"stderr","time":"2022-04-05T02:33:59.758079133Z"}
{"log":" File \"/usr/src/app/chaos_genius/connectors/__init__.py\", line 93, in get_schema_names\n","stream":"stderr","time":"2022-04-05T02:33:59.758081984Z"}
{"log":" db_connection.init_inspector()\n","stream":"stderr","time":"2022-04-05T02:33:59.758084951Z"}
{"log":" File \"/usr/src/app/chaos_genius/connectors/base_db.py\", line 33, in init_inspector\n","stream":"stderr","time":"2022-04-05T02:33:59.758087759Z"}
{"log":" self.inspector = SQLAlchemy().inspect(self.engine)\n","stream":"stderr","time":"2022-04-05T02:33:59.758090691Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/inspection.py\", line 64, in inspect\n","stream":"stderr","time":"2022-04-05T02:33:59.758093484Z"}
{"log":" ret = reg(subject)\n","stream":"stderr","time":"2022-04-05T02:33:59.758096375Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/reflection.py\", line 182, in _engine_insp\n","stream":"stderr","time":"2022-04-05T02:33:59.758099127Z"}
{"log":" return Inspector._construct(Inspector._init_engine, bind)\n","stream":"stderr","time":"2022-04-05T02:33:59.758102067Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/reflection.py\", line 117, in _construct\n","stream":"stderr","time":"2022-04-05T02:33:59.758104928Z"}
{"log":" init(self, bind)\n","stream":"stderr","time":"2022-04-05T02:33:59.75810786Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/reflection.py\", line 128, in _init_engine\n","stream":"stderr","time":"2022-04-05T02:33:59.758110619Z"}
{"log":" engine.connect().close()\n","stream":"stderr","time":"2022-04-05T02:33:59.758113545Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py\", line 3166, in connect\n","stream":"stderr","time":"2022-04-05T02:33:59.758117612Z"}
{"log":" return self._connection_cls(self, close_with_result=close_with_result)\n","stream":"stderr","time":"2022-04-05T02:33:59.758121796Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py\", line 96, in __init__\n","stream":"stderr","time":"2022-04-05T02:33:59.758126164Z"}
{"log":" else engine.raw_connection()\n","stream":"stderr","time":"2022-04-05T02:33:59.758130367Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py\", line 3245, in raw_connection\n","stream":"stderr","time":"2022-04-05T02:33:59.758133728Z"}
{"log":" return self._wrap_pool_connect(self.pool.connect, _connection)\n","stream":"stderr","time":"2022-04-05T02:33:59.75813682Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py\", line 3215, in _wrap_pool_connect\n","stream":"stderr","time":"2022-04-05T02:33:59.75813971Z"}
{"log":" Connection._handle_dbapi_exception_noconnection(\n","stream":"stderr","time":"2022-04-05T02:33:59.758142747Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py\", line 2068, in _handle_dbapi_exception_noconnection\n","stream":"stderr","time":"2022-04-05T02:33:59.758145573Z"}
{"log":" util.raise_(\n","stream":"stderr","time":"2022-04-05T02:33:59.75814856Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py\", line 211, in raise_\n","stream":"stderr","time":"2022-04-05T02:33:59.758151349Z"}
{"log":" raise exception\n","stream":"stderr","time":"2022-04-05T02:33:59.758154354Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py\", line 3212, in _wrap_pool_connect\n","stream":"stderr","time":"2022-04-05T02:33:59.758159798Z"}
{"log":" return fn()\n","stream":"stderr","time":"2022-04-05T02:33:59.758162761Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py\", line 301, in connect\n","stream":"stderr","time":"2022-04-05T02:33:59.75816554Z"}
{"log":" return _ConnectionFairy._checkout(self)\n","stream":"stderr","time":"2022-04-05T02:33:59.758168467Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py\", line 761, in _checkout\n","stream":"stderr","time":"2022-04-05T02:33:59.758171275Z"}
{"log":" fairy = _ConnectionRecord.checkout(pool)\n","stream":"stderr","time":"2022-04-05T02:33:59.758174324Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py\", line 419, in checkout\n","stream":"stderr","time":"2022-04-05T02:33:59.758177091Z"}
{"log":" rec = pool._do_get()\n","stream":"stderr","time":"2022-04-05T02:33:59.758179984Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/impl.py\", line 145, in _do_get\n","stream":"stderr","time":"2022-04-05T02:33:59.758182878Z"}
{"log":" self._dec_overflow()\n","stream":"stderr","time":"2022-04-05T02:33:59.758185998Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py\", line 70, in __exit__\n","stream":"stderr","time":"2022-04-05T02:33:59.75818876Z"}
{"log":" compat.raise_(\n","stream":"stderr","time":"2022-04-05T02:33:59.758191848Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py\", line 211, in raise_\n","stream":"stderr","time":"2022-04-05T02:33:59.758195902Z"}
{"log":" raise exception\n","stream":"stderr","time":"2022-04-05T02:33:59.758200105Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/impl.py\", line 142, in _do_get\n","stream":"stderr","time":"2022-04-05T02:33:59.758204233Z"}
{"log":" return self._create_connection()\n","stream":"stderr","time":"2022-04-05T02:33:59.758208351Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py\", line 247, in _create_connection\n","stream":"stderr","time":"2022-04-05T02:33:59.758211184Z"}
{"log":" return _ConnectionRecord(self)\n","stream":"stderr","time":"2022-04-05T02:33:59.758214101Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py\", line 362, in __init__\n","stream":"stderr","time":"2022-04-05T02:33:59.758216897Z"}
{"log":" self.__connect()\n","stream":"stderr","time":"2022-04-05T02:33:59.758219851Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py\", line 605, in __connect\n","stream":"stderr","time":"2022-04-05T02:33:59.758222595Z"}
{"log":" pool.logger.debug(\"Error on connect(): %s\", e)\n","stream":"stderr","time":"2022-04-05T02:33:59.758225505Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py\", line 70, in __exit__\n","stream":"stderr","time":"2022-04-05T02:33:59.758228674Z"}
{"log":" compat.raise_(\n","stream":"stderr","time":"2022-04-05T02:33:59.758231678Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py\", line 211, in raise_\n","stream":"stderr","time":"2022-04-05T02:33:59.758234418Z"}
{"log":" raise exception\n","stream":"stderr","time":"2022-04-05T02:33:59.758237351Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py\", line 599, in __connect\n","stream":"stderr","time":"2022-04-05T02:33:59.758240095Z"}
{"log":" connection = pool._invoke_creator(self)\n","stream":"stderr","time":"2022-04-05T02:33:59.758243021Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/create.py\", line 578, in connect\n","stream":"stderr","time":"2022-04-05T02:33:59.758245851Z"}
{"log":" return dialect.connect(*cargs, **cparams)\n","stream":"stderr","time":"2022-04-05T02:33:59.758248816Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py\", line 584, in connect\n","stream":"stderr","time":"2022-04-05T02:33:59.758255426Z"}
{"log":" return self.dbapi.connect(*cargs, **cparams)\n","stream":"stderr","time":"2022-04-05T02:33:59.758258416Z"}
{"log":" File \"/usr/local/lib/python3.8/dist-packages/psycopg2/__init__.py\", line 127, in connect\n","stream":"stderr","time":"2022-04-05T02:33:59.75826123Z"}
{"log":" conn = _connect(dsn, connection_factory=connection_factory, **kwasync)\n","stream":"stderr","time":"2022-04-05T02:33:59.75826422Z"}
{"log":"sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) FATAL: the database system is in recovery mode\n","stream":"stderr","time":"2022-04-05T02:33:59.75826708Z"}
{"log":"FATAL: the database system is in recovery mode\n","stream":"stderr","time":"2022-04-05T02:33:59.758270332Z"}
{"log":"\n","stream":"stderr","time":"2022-04-05T02:33:59.758274442Z"}
{"log":"(Background on this error at: http://sqlalche.me/e/14/e3q8)\n","stream":"stderr","time":"2022-04-05T02:33:59.758278241Z"}
{"log":"[2022-04-05 02:33:59,760: INFO/ForkPoolWorker-1] Task chaos_genius.jobs.metadata_prefetch.fetch_data_source_schema[4ceeb7a0-aaa8-40c5-8a08-972f2a1a5ae0] succeeded in 0.035913983010686934s: False\n","stream":"stderr","time":"2022-04-05T02:33:59.760373818Z"}
The text was updated successfully, but these errors were encountered:
Describe the bug
Based on Worker logs we found that Sync status for some Postgres datasource went into "Error" state.
Explain the environment
Current behavior
Expected behavior
Screenshots
Additional context
Logs
SSL SYSCALL error: EOF detected
the database system is in recovery mode
The text was updated successfully, but these errors were encountered: