BaseDestinationV1V2Migrator Timeout errors

[fkody1]

In all of my syncs towards a Redshift destination, I run into backend timeout errors when the BaseDestinationV1V2Migrator is running its checks to see if an individual stream needs to be migrated. All of my destinations are already working with V2Destinations.

Airbyte: OSS Version 0.52.1
Source: Occurs with several source connectors
Destination: Redshift 2.4.3

Here is an example of the error I see from Redshift:

2024-05-03 06:07:08 destination > WARN type-and-dedupe c.z.h.p.ProxyConnection(checkException):178 HikariPool-1 - Connection com.amazon.redshift.jdbc.RedshiftConnectionImpl@69e86114 marked as broken because of SQLSTATE(08006), ErrorCode(0) com.amazon.redshift.util.RedshiftException: An I/O error occurred while sending to the backend.

Here are some log lines showcasing when the timeout occurs; always at the same stream in the run; where it just gives a INFO pool-3-thread message every minute for 1 hour, until it officially times out :

2024-05-03 04:58:13 destination > INFO type-and-dedupe i.a.i.b.d.t.BaseDestinationV1V2Migrator(migrateIfNecessary):27 Assessing whether migration is necessary for stream sst_sms_message_log
2024-05-03 04:58:13 destination > INFO type-and-dedupe i.a.i.b.d.t.BaseDestinationV1V2Migrator(shouldMigrate):46 Checking whether v1 raw table _airbyte_raw_sst_sms_message_log in dataset schemaname exists
2024-05-03 04:58:13 destination > INFO type-and-dedupe i.a.c.i.d.j.t.JdbcDestinationHandler(lambda$findExistingTable$11):271 Retrieving existing columns for database.airbyte_internal.schema_raw__stream_sst_sms_message_log
2024-05-03 04:58:14 destination > INFO type-and-dedupe i.a.c.i.d.j.t.JdbcDestinationHandler(lambda$findExistingTable$11):271 Retrieving existing columns for database.schema._airbyte_raw_sst_sms_message_log
2024-05-03 04:58:14 destination > INFO type-and-dedupe i.a.i.b.d.t.BaseDestinationV1V2Migrator(shouldMigrate):50 Migration Info: Required for Sync mode: true, No existing v2 raw tables: false, A v1 raw table exists: true
2024-05-03 04:58:14 destination > INFO type-and-dedupe i.a.i.b.d.t.BaseDestinationV1V2Migrator(migrateIfNecessary):33 No Migration Required for stream: sst_sms_message_log
2024-05-03 04:58:55 destination > INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):106 [ASYNC QUEUE INFO] Global: max: 1.02 GB, allocated: 10 MB (10.0 MB), % used: 0.009621224424604074 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.000000
2024-05-03 04:58:55 destination > INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):130 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2024-05-03 04:59:55 destination > INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):106 [ASYNC QUEUE INFO] Global: max: 1.02 GB, allocated: 10 MB (10.0 MB), % used: 0.009621224424604074 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.000000
2024-05-03 04:59:55 destination > INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):130 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2024-05-03 05:00:55 destination > INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):106 [ASYNC QUEUE INFO] Global: max: 1.02 GB, allocated: 10 MB (10.0 MB), % used: 0.009621224424604074 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.000000
2024-05-03 05:00:55 destination > INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):130 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2024-05-03 05:01:55 destination > INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):106 [ASYNC QUEUE INFO] Global: max: 1.02 GB, allocated: 10 MB (10.0 MB), % used: 0.009621224424604074 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.000000```

Any help would be very helpful. Thank you!

[fkody1]

Redshift Destination connector version 2.5.0