Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

KubernetesPodOperator with multiple containers hangs if container other than base container is still running #39693

Open
2 tasks done
jonathan-ostrander opened this issue May 17, 2024 · 3 comments · May be fixed by #39694
Open
2 tasks done
Assignees
Labels
area:providers kind:bug This is a clearly a bug provider:cncf-kubernetes Kubernetes provider related issues

Comments

@jonathan-ostrander
Copy link

Apache Airflow version

main (development)

If "Other Airflow 2 version" selected, which one?

No response

What happened?

A KubernetesPodOperator with the following full_pod_spec:

apiVersion: v1
kind: Pod
metadata:
  name: multi-container-pod
spec:
  restartPolicy: Never
  containers:
  - name: base
    image: busybox
    command: ["sh", "-c", "echo base will exit after 30 seconds; sleep 30"]
  - name: sidecar
    image: busybox
    command: ["sh", "-c", "echo sidecar running indefinitely; while true; do sleep 3600; done"]

will not mark the task as successful after 30 seconds because the sidecar will continue to run after the base container has succeeded. This happens because the pod_manager gets stuck waiting for pod completion. This if statement returns False when istio is not enabled on the pod.

What you think should happen instead?

The pod should be considered complete when the base container succeeds regardless of whether or not any other containers on the pod are still running.

How to reproduce

Create a KubernetesPodOperator with the full_pod_spec provided.

Operating System

MacOS 14.4.1

Versions of Apache Airflow Providers

apache-airflow-providers-cncf-kubernetes==8.0.1

Deployment

Other

Deployment details

Kubernetes on Google Kubernetes Engine. Kubernetes executors and worker pods all run on the same cluster.

Anything else?

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

@jonathan-ostrander jonathan-ostrander added area:core kind:bug This is a clearly a bug needs-triage label for new issues that we didn't triage yet labels May 17, 2024
Copy link

boring-cyborg bot commented May 17, 2024

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

@Taragolis Taragolis added area:providers provider:cncf-kubernetes Kubernetes provider related issues and removed area:core needs-triage label for new issues that we didn't triage yet labels May 17, 2024
@Taragolis
Copy link
Contributor

Feel free to fix this behaiviour

jonathan-ostrander added a commit to jonathan-ostrander/airflow that referenced this issue May 17, 2024
Fixes apache#39693.

Special logic was added to `KubernetesPodOperator`'s lifecycle to handle the case where an istio proxy sidecar is running and preventing the pod from completing, but this logic should have been applied more generally to handle when multiple containers are ran in the pod. The new behavior considers the pod completed if the container matching the base name has completed.
@dirrao
Copy link
Collaborator

dirrao commented May 19, 2024

related: #39625

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:providers kind:bug This is a clearly a bug provider:cncf-kubernetes Kubernetes provider related issues
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants