`kubectl get` should have a way to filter for advanced pods status #49387

simonswine · 2017-07-21T14:57:37Z

What happened:

I'd like to have a simple command to check for pods that are currently not ready

What you expected to happen:

I can see a couple of options:

There is some magic flag I am not aware of
Having a flag for kubectl get to filter the output using go/jsonpath
Distinguish between Pod Phase Running&Ready and Running
Flag to filter on ready status

How to get that currently:

kubectl get pods --all-namespaces -o json  | jq -r '.items[] | select(.status.phase != "Running" or ([ .status.conditions[] | select(.type == "Ready" and .state == false) ] | length ) == 1 ) | .metadata.namespace + "/" + .metadata.name'

The text was updated successfully, but these errors were encountered:

simonswine · 2017-07-21T14:57:50Z

/kind feature

simonswine · 2017-07-21T14:58:58Z

/sig cli

EtienneDeneuve · 2017-08-28T12:39:08Z

Same here, It sound incredible to use a complex syntax to only list non running container...

jackzampolin · 2017-10-18T21:37:04Z

Ideally I would be able to say something like:

kubectl get pods --namespace foo -l status=pending

carlossg · 2017-11-23T09:54:31Z

I had to make a small modification to .status == "False" to get it to work

kubectl get pods -a --all-namespaces -o json  | jq -r '.items[] | select(.status.phase != "Running" or ([ .status.conditions[] | select(.type == "Ready" and .status == "False") ] | length ) == 1 ) | .metadata.namespace + "/" + .metadata.name'

dixudx · 2017-11-24T05:30:13Z

#50140 provides a new flag --field-selector to filter these pods now.

$ kubectl get pods --field-selector=status.phase!=Running

/close

asarkar · 2017-12-13T03:44:46Z

@dixudx

kubectl version
Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.4", GitCommit:"9befc2b8928a9426501d3bf62f72849d5cbcd5a3", GitTreeState:"clean", BuildDate:"2017-11-20T19:11:02Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0", GitCommit:"0b9efaeb34a2fc51ff8e4d34ad9bc6375459c4a4", GitTreeState:"clean", BuildDate:"2017-11-29T22:43:34Z", GoVersion:"go1.9.1", Compiler:"gc", Platform:"linux/amd64"}

kubectl get po --field-selector=status.phase==Running -l app=k8s-watcher
Error: unknown flag: --field-selector

dixudx · 2017-12-13T05:06:42Z

@asarkar --field-selector is targeted for v1.9, which is coming out soon.

simonswine · 2018-01-15T19:41:53Z

@dixudx thanks for the PR for the field-selector. But I think this is not what I had in mind. I wanted to be able figure out pods that have one or more container that are not passing the readiness checks.

Given that I have non ready pod (kubectl v1.9.1) READY 0/1:

$ kubectl get pods                                       
NAME          READY     STATUS    RESTARTS   AGE
pod-unready   0/1       Running   0          50s

This pod is still in Phase running, so I can't get it using your proposed filter:

$ kubectl get pods --field-selector=status.phase!=Running
No resources found.

simonswine · 2018-01-15T19:42:24Z

/reopen

af732p · 2018-01-18T15:04:54Z

Got the same issue.
I would be glad to have something like:
kubectl get pods --field-selector=status.ready!=True

artemyarulin · 2018-02-21T13:44:51Z

Hm, can I use it for getting nested array items? Like I want to do

kubectl get pods --field-selector=status.containerStatuses.restartCount!=0

But it returns error, tried status.containerStatuses..restartCount, but it also doesn't work and returns same error Error from server (BadRequest): Unable to find "pods" that match label selector "", field selector "status.containerStatuses..restartCount==0": field label not supported: status.containerStatuses..restartCount

jackzampolin · 2018-02-21T23:47:33Z

@artemyarulin try status.containerStatuses[*].restartCount==0

artemyarulin · 2018-02-22T08:33:17Z

Thanks, just tried with kubectl v1.9.3/cluster v1.9.2 and it returns same error - Error from server (BadRequest): Unable to find "pods" that match label selector "", field selector "status.containerStatuses[*].restartCount!=0": field label not supported: status.containerStatuses[*].restartCount. Am I doing something wrong? Does it work for you?

Due the test-flake I discovered that the termination helper didn't work as expected, and the status.phase is not represent at all (kubernetes/kubernetes#49387) Issue: ``` vagrant@k8s1:~$ kubectl delete pod testds-w7prl pod "testds-w7prl" deleted vagrant@k8s1:~$ kubectl get pods NAME READY STATUS RESTARTS AGE netcatds-bhxv4 1/1 Running 0 5m netcatds-zpzzl 1/1 Running 0 5m testclient-8qx59 1/1 Running 0 1m testclient-r9xmm 1/1 Running 0 1m testds-fwss5 1/1 Running 0 32s testds-w7prl 0/1 Terminating 0 1m vagrant@k8s1:~$ kubectl get pods -o "jsonpath='{.items[*].status.phase}'" 'Running Running Running Running^C vagrant@k8s1:~$ kubectl get pods NAME READY STATUS RESTARTS AGE netcatds-bhxv4 1/1 Running 0 5m netcatds-zpzzl 1/1 Running 0 5m testclient-8qx59 1/1 Running 0 1m testclient-r9xmm 1/1 Running 0 1m testds-fwss5 1/1 Running 0 40s testds-w7prl 0/1 Terminating 0 1m vagrant@k8s1:~$ ``` Signed-off-by: Eloy Coto <eloy.coto@gmail.com>

migueleliasweb · 2018-04-19T13:14:35Z

Sadly, the same thing happens for v1.9.4:

What I'm trying to do here is to get all pods with a given parent uid...

$ kubectl get pod --field-selector='metadata.ownerReferences[*].uid=d83a23e1-37ba-11e8-bccf-0a5d7950f698'
Error from server (BadRequest): Unable to find "pods" that match label selector "", field selector "ownerReferences[*].uid=d83a23e1-37ba-11e8-bccf-0a5d7950f698": field label not supported: ownerReferences[*].uid

Waiting anxiously for this feature •ᴗ•

dixudx · 2018-04-19T14:44:13Z

--field-selector='metadata.ownerReferences[].uid=d83a23e1-37ba-11e8-bccf-0a5d7950f698'
field label not supported: ownerReferences[].uid

This filter string is not supported.

For pods, only "metadata.name", "metadata.namespace", "spec.nodeName", "spec.restartPolicy", "spec.schedulerName", status.phase", "status.podIP", "status.nominatedNodeName", "sepc.nodeName" are supported.

@migueleliasweb If you want to filer out the pod in your case, you can use jq.

$ kubectl get pod -o json | jq '.items | map(select(.metadata.ownerReferences[] | .uid=="d83a23e1-37ba-11e8-bccf-0a5d7950f698"))'

Also you can use JSONPath Support of kubectl.

migueleliasweb · 2018-04-19T23:44:33Z

Thanks @dixudx . But let me understand a litle bit better. If I'm running this query in a cluster with a few thousand pods:

Does the APIServer fetch all of them from ETCD and them apply in-memory filtering ?
Or does my kubectl receive all pods and apply locally the filter ?
Or the filtering occours inside the ETCD ? So only the filtered results are returned ?

dixudx · 2018-04-20T01:25:59Z

Does the APIServer will fetch all of them from ETCD and them apply in-memory filtering ?
Or does my kubectl will receive all pods and apply locally the filter ?
Or the filtering occours inside the ETCD ? So only the filtered results are returned ?

@migueleliasweb If --field-selector is issued when using kubectl, the filtering is in a cache of apiserver. APIServer will have a single watch open to etcd, watching all the objects (of a given type) without any filtering. The changes delivered from etcd will then be stored in a cache of apiserver.

For --sort-by, the filtering is on kubectl client side.

kvs · 2018-04-26T09:22:49Z

This work great for me with kubectl get, but it would also be nice if it could apply to delete and describe

fejta-bot · 2018-07-25T09:48:54Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

randywatson1979 · 2021-11-17T14:11:49Z

In my case I have rolling update deployment in which the new replicaset is marked as Running and the old replicaset Terminating after kubectl rollout returned success.

Waiting for deployment "some-deployment-app" rollout to finish: 1 old replicas are pending termination...
deployment "some-deployment-app" successfully rolled out

kubectl get pods -n some-namespace --selector=app=some-app 

NAME                        READY   STATUS        RESTARTS   AGE
some-deployment-app-p5pzw   1/1     Terminating   0          8m44s
some-deployment-app-j3xwk   1/1     Running       0          91s

However, I have randomly encountered that the -o json output shows both replicasets are running with spec: .items[0].status.phase: "Running" and .items[0].status.containerStatuses[0].state.running
.items[1].status.phase: "Running" and .items[1].status.containerStatuses[0].state.running

But after one second:
.items[0].status.phase: "Running" and .items[0].status.containerStatuses[0].state.terminated
.items[1].status.phase: "Running" and .items[1].status.containerStatuses[0].state.running

Then when I use the go-lang method, I get the correct results:
kubectl get pod -n some-namespace --selector=app=some-app -o go-template --template='{{range .items}}{{$metadataName := .metadata.name}}{{range .status.containerStatuses}}{{if .state.running}}{{$metadataName}}{{end}}{{end}}{{end}}')

So as you noticed the field selector will not work because both will always return status.phase: "Running".
It seems like (assumption) that there is a delay when Kubernetes marks the replicaset as Terminating (Sigterm) but the scheduler needs to perform this action with the pod.
So, where is Terminating based from with kubectl get pods command? It's certainly not from the specs of the replicasets.

This behaviour is causing confusion.

I could set a fixed delay of 3 seconds, but that is a very bad solution.
Another way is to set terminationGracePeriodSeconds: 0 for the deployment which does remedie the issue, but this effectively disables the termination grace period which defaults to 30 seconds. I'm not sure if it is bad practice.

Further inspections seems that during the termination grace period sigterm and sigkill, the container specs aren't updated yet. When it does get a sigkill or the container has shutdown itself within the grace period, then it will update the terminating container specs after the fact:

...
"status": {
               "containerStatuses": [
                        {
                        "state": {
                                        "terminated": {
                                            "containerID": "docker://88f2d0e303bc9a253e66002523c5be662cd756206d6ba68e",
                                            "exitCode": 0,
                                            "finishedAt": "2021-11-17T14:43:04Z",
                                            "reason": "Completed",
                                            "startedAt": "2021-11-17T13:47:15Z"
                                        }
                                    }

mnpenner · 2021-11-18T22:51:18Z

You can do something like this to find pods that have a not-ready container:

kubectl get pods -oyaml | yq e '.items[] | select(.status.containerStatuses[].ready | not) | .metadata.name' -

Doesn't work so well on crons and one-off jobs but that's just to give you an idea if you're looking for a more powerful query selector.

yq being https://github.com/mikefarah/yq. Alternatively you can use -ojson with jq instead.

kubectl get pods -ojson | jq '.items[] | select(.status.containerStatuses[].started | not) | [{name: .metadata.name, statuses: .status.containerStatuses}]'

Or maybe:

kubectl get pods -ojson | jq '.items[] | select(.status.containerStatuses[] | ((.ready|not) and .state.terminated.exitCode!=0)) | [{name: .metadata.name, statuses: .status.containerStatuses}]'

pothitos · 2021-12-17T00:03:20Z

Hello! A small fix to the above (very useful) comment.

Here's something I use to find all pods that are not fully running (eg some containers are failing)
kubectl get po --all-namespaces | gawk 'match($3, /([0-9])+\/([0-9])+/, a) {if (a[1] < a[2] && $4 != "Completed") print $0}'

The + should be inside the parentheses of the matching patterns, like

kubectl get po --all-namespaces | gawk 'match($3, /([0-9]+)\/([0-9]+)/, a) {if (a[1] < a[2] && $4 != "Completed") print}'

P.S. I omitted $0 as it's not necessary.

You may also want to check this related script.

k8s-triage-robot · 2022-03-24T21:58:45Z

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle stale
Mark this issue or PR as rotten with /lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

mindw · 2022-03-25T06:14:58Z

/remove-lifecycle stale

vikas027 · 2022-05-30T00:44:06Z

For anyone who is looking to filter Completed pods, they are referred to as Succeeded.

❯ k get pod --field-selector=status.phase=Succeeded

snukone · 2022-07-27T14:44:31Z

In case someone want to use it in an shell if-statement (example with tekton pipelinerun ressource, same behaviour with any other ressource depending on json):
if [ $(kubectl get pipelinerun --selector='notification-provider' -o jsonpath='{.items[?(@.status.conditions[*].reason=="Running")].metadata.name}' | wc -c) -gt 0 ]; then echo Found running pipelines!; fi

k8s-triage-robot · 2022-10-25T15:39:22Z

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle stale
Mark this issue or PR as rotten with /lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

invidian · 2022-10-25T15:52:12Z

/remove-lifecycle stale

k8s-triage-robot · 2023-01-23T15:54:21Z

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle stale
Mark this issue or PR as rotten with /lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

invidian · 2023-01-23T15:57:02Z

/remove-lifecycle stale

k8s-triage-robot · 2023-04-23T16:03:39Z

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle stale
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

mindw · 2023-04-25T06:06:40Z

/remove-lifecycle stale

k8s-triage-robot · 2024-01-19T02:59:57Z

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle stale
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

mindw · 2024-01-19T12:51:43Z

/remove-lifecycle stale

k8s-triage-robot · 2024-04-18T13:19:57Z

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle stale
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

felipecrs · 2024-04-18T13:27:17Z

This would still be very nice to have.

mindw · 2024-04-19T08:02:44Z

/remove-lifecycle stale

k8s-ci-robot added the kind/feature Categorizes issue or PR as related to a new feature. label Jul 21, 2017

k8s-github-robot added the needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. label Jul 21, 2017

k8s-ci-robot added the sig/cli Categorizes an issue or PR as relevant to SIG CLI. label Jul 21, 2017

k8s-github-robot removed the needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. label Jul 21, 2017

k8s-ci-robot assigned dixudx Nov 24, 2017

k8s-ci-robot closed this as completed Nov 24, 2017

carlossg mentioned this issue Dec 3, 2017

Add option to apply caps only on alive pods jenkinsci/kubernetes-plugin#252

Merged

k8s-ci-robot reopened this Jan 15, 2018

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Mar 24, 2022

k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Mar 25, 2022

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Oct 25, 2022

k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Oct 25, 2022

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jan 23, 2023

k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jan 23, 2023

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Apr 23, 2023

k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Apr 25, 2023

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jan 19, 2024

k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jan 19, 2024

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Apr 18, 2024

k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Apr 19, 2024

kubectl get should have a way to filter for advanced pods status #49387

kubectl get should have a way to filter for advanced pods status #49387

Comments

simonswine commented Jul 21, 2017 • edited

simonswine commented Jul 21, 2017

simonswine commented Jul 21, 2017

EtienneDeneuve commented Aug 28, 2017 • edited

jackzampolin commented Oct 18, 2017

carlossg commented Nov 23, 2017 • edited

dixudx commented Nov 24, 2017

asarkar commented Dec 13, 2017

dixudx commented Dec 13, 2017

simonswine commented Jan 15, 2018

simonswine commented Jan 15, 2018

af732p commented Jan 18, 2018

artemyarulin commented Feb 21, 2018

jackzampolin commented Feb 21, 2018

artemyarulin commented Feb 22, 2018

migueleliasweb commented Apr 19, 2018

dixudx commented Apr 19, 2018

migueleliasweb commented Apr 19, 2018 • edited

dixudx commented Apr 20, 2018 • edited

kvs commented Apr 26, 2018

fejta-bot commented Jul 25, 2018

randywatson1979 commented Nov 17, 2021 • edited

mnpenner commented Nov 18, 2021 • edited

pothitos commented Dec 17, 2021 • edited

k8s-triage-robot commented Mar 24, 2022

mindw commented Mar 25, 2022

vikas027 commented May 30, 2022

snukone commented Jul 27, 2022 • edited

k8s-triage-robot commented Oct 25, 2022

invidian commented Oct 25, 2022

k8s-triage-robot commented Jan 23, 2023

invidian commented Jan 23, 2023

k8s-triage-robot commented Apr 23, 2023

mindw commented Apr 25, 2023

k8s-triage-robot commented Jan 19, 2024

mindw commented Jan 19, 2024

k8s-triage-robot commented Apr 18, 2024

felipecrs commented Apr 18, 2024

mindw commented Apr 19, 2024

`kubectl get` should have a way to filter for advanced pods status #49387

`kubectl get` should have a way to filter for advanced pods status #49387

simonswine commented Jul 21, 2017 •

edited

EtienneDeneuve commented Aug 28, 2017 •

edited

carlossg commented Nov 23, 2017 •

edited

migueleliasweb commented Apr 19, 2018 •

edited

dixudx commented Apr 20, 2018 •

edited

randywatson1979 commented Nov 17, 2021 •

edited

mnpenner commented Nov 18, 2021 •

edited

pothitos commented Dec 17, 2021 •

edited

snukone commented Jul 27, 2022 •

edited