Airbyte creating too many attempts and not terminating old ones #38187
Labels
area/platform
issues related to the platform
community
team/platform-move
type/bug
Something isn't working
Helm Chart Version
0.44.1
What step the error happened?
During the Sync
Relevant information
In a managed kubernetes deployment managed by another team, we deployed airbyte and created some pipelines (csv to a clickhouse database and postresql to clickhouse), the sync was running everyday for couples of days but started failing, after troubleshooting, one of the
orchestrator-repl-job-50-attempt-x
that is responsible for writing data to clickhouse has insufficient cpu:we could resolve it by adding more k8s nodes or free some resources, but we found out many pods starting with names (
orchestrator-repl-job-50-attempt-X
,destination-clickhouse-check-48-X-yzgdz
,n-clickhouse-check-1dd1ea2d-a22d-4b9a-bc6d-828
,rce-mysql-discover-09f290d4-c311-426e-bdc8-53f88f4059f1-0-eqymi
, etc.) are not being deleted by airbyte. Seems that Airbyte is forcing the attempts one after another:Checking the documentation about configuring jobs parameters, to force the number of attempts to be 2 for example like the
SYNC_JOB_MAX_ATTEMPTS
, but can't find where to configure it, is it by updating the configmapairbyte-env
? or in which section invalues.yml
? need a confirmation about it for experimentation reason.I'm new to airbyte and the main question why airbyte doesn't delete old pods when it is trying many attempts? is it a bug?
Thanks,
Marwane.
Relevant log output
The text was updated successfully, but these errors were encountered: