Make our kubectl image more useful?

[thockin]

xref #116667 which was "resolved" with #116672
xref #73760
xref #40610

We have the "downward API" which extracts almost exclusively information which is represented as fields on "this pod" and projects either env vars or files on disk. But there's a slow but non-zero trickle of requests to extract info from other places (e.g. "this node"). This is less obviously reasonable because it could turn into a confused-deputy, exposing information that the pod is not actually authorized to know.

In #73760 (comment) (more than 2 years ago!) I suggested people use a sidecar to get and format the info. We have never actually made such a sidecar image, leaving it as an exercise for the reader.

Can / shold we do better?

The existing image (registry.k8s.io/kubernetes/kubectl) seems to serve two purposes:

1. as a portable CLI when you don't have kubectl installed
2. as this sidecars

It works for #1, but is not super useful for #2 without some sorts of tools in the image.

How would we feel about either pivoting it or adding a new kubectl-sidecar image and adding a bunch of useful shell tools? E.g.

• bash
• grep
• sed
• awk
• head
• tail
• jq
• diff
• coreutils (mv, ln, cat, sleep, ...)

In short, I think it should be possible for people to write something like:

while true; do
    kubectl get -o json node $THIS_NODE | jq -r '.status.conditions[] | [.type,.status] | @tsv' | grep Ready | cut -f2 > .ready.tmp
    mv -f file.tmp ready
    sleep 10
done

cc @dims since you touched the image before :)

[k8s-ci-robot]

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[dims]

@thockin yes of course! happy to iterate the existing image, let me find some folks who can help us take care of this image going forward (add/update docs, write up blog posts, evangelize this!)

[rayandas]

I would love to get involved in this. :)

[Ritikaa96]

/kind feature
/sig release

[Debanitrkl]

Happy to contribute to this, need more context, so we are first going to add the shell tools(do we want to list them out first?) as command to kubectl and then release a new kubectl-sidecar image?

[vatsalparekh]

I would like get involved in this too

[dims]

first step is to update the kubectl image with the shell tools listed by @thockin (No to a new image, No to new commands to kubectl for now)
i'd also request folks to write a blog with the possibilities using the kubectl image that will go with k8s 1.28

[thockin]

I'm happy to contribute to a blog on how to use this! I did think about building more of this into kubectl, but let's first try to make tools work, as that will always be more flexible. What I'd like to consider is something like an "exec-on-event" flag to `kubectl get -w` so we could watch a resource, rather than poll, and still execute a shell command for each event received. But that's a whole feature of its own.

…

On Wed, Jul 26, 2023 at 6:20 AM Davanum Srinivas ***@***.***> wrote: first step is to update the kubectl image with the shell tools listed by @thockin <https://github.com/thockin> (No to a new image, No to new commands to kubectl for now) i'd also request folks to write a blog with the possibilities using the kubectl image that will go with k8s 1.28 — Reply to this email directly, view it on GitHub <#119567 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABKWAVCGH3ACKOQYQC2ZUSTXSEKQBANCNFSM6AAAAAA2XPUR7Y> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[dipankardas011]

@thockin, I am unable to understand what changes you proposed, I have used Kubectl but not getting what features are missing

can you explain it? :)

[thockin]

For this issue I just want more tools and a shell so people can write bespoke sidecars to extract and format kube API information. LATER we can talk about new features

…

On Wed, Jul 26, 2023, 10:44 AM Dipankar Das ***@***.***> wrote: @thockin <https://github.com/thockin>, I am unable to understand what changes you proposed, I have used Kubectl but not getting what features are missing can you explain it? :) — Reply to this email directly, view it on GitHub <#119567 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABKWAVG4SRLC4M3VHZYMZW3XSFJP3ANCNFSM6AAAAAA2XPUR7Y> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[pbxqdown]

/sig cli

[BenTheElder]

This sort of image has a large "API" and needs more frequent patching.

I think we should still provide a minimal kubectl image that won't set off everyone's CVE scanners and will pull quickly.

[thockin]

What is the smaller image useful for? The only use case I can see is "I want to run kubectl in a place where I don't have a kubectl binary, but I do have a kubeconfig file". Not impossible but it seems unlikely. It's not useful in a pod except as a one-off (otherwise you need a shell to drive a loop). If it is only for CLI use, the larger image doesn't seem problematic to me.

Help me understand a user-story where this is useful other than "I want to run a sidecar which extract info from API and writes it to a file in a volume" ?

[BenTheElder]

Help me understand a user-story where this is useful other than "I want to run a sidecar which extract info from API and writes it to a file in a volume" ?

"I don't want to install kubectl when I can run a container" was a driving use-case on the original kubectl image thread IIRC, you can more readily pipe the output to other tools if you're running outside the cluster.

[xmudrii]

I left some opinions in this comment: #119592 (comment)

I think I agree with @thockin about this. My opinion is that we should optimize for user experience here.

[thockin]

"I don't want to install kubectl when I can run a container"

I do accept that use case. That said, we might disagree whether having a shell and tools in the image is important for CLI users. Even if you don't have a local kubectl binary, you probably have grep, but do you have jq? It's less common.

What's the impact of a larger (size and surface) image for CLI users? Not much, I think? Either way, we need such an image and need to keep it patched.

[rayandas]

As the revert is now merged, I've created another WIP PR to add the necessary utils and the multi arch image.
A TODO is to implement a hack to adjust the multi arch image so that the build passes.

I think one of the possible way is to add a Makefile along with the Kubectl Dockerfile, just like etcd image and force selection of the correct image.

cc: @dims @saschagrunert @xmudrii

[thockin]

Is the debian-base image not a multi-arch manifest?

[rayandas]

@thockin It is a multi-arch image, but somehow it's not picking the right image while building for ARM64. These were the failures on master-blocking build-master and master-informing Conformance-EC2-arm64-master.

[thockin]

I build multi-arch git-sync using docker buildx - docker itself thinks it is the other architecture. I don't know how this works in k/k/build any more, but here's how git-sync does it:

make all-container:

ALL_PLATFORMS := linux/amd64 linux/arm linux/arm64 linux/ppc64le linux/s390x
all-container: $(addprefix container-, $(subst /,_, $(ALL_PLATFORMS)))

calls (https://github.com/kubernetes/git-sync/blob/master/Makefile#L164-L192):

container: .container-$(DOTFILE_IMAGE) container-name
.container-$(DOTFILE_IMAGE): bin/$(OS)_$(ARCH)/$(BIN) $(LICENSES) Dockerfile.in .buildx-initialized
	sed                                  \
	    -e 's|{ARG_BIN}|$(BIN)|g'        \
	    -e 's|{ARG_ARCH}|$(ARCH)|g'      \
	    -e 's|{ARG_OS}|$(OS)|g'          \
	    -e 's|{ARG_FROM}|$(BASEIMAGE)|g' \
	    -e 's|{ARG_STAGING}|/staging|g' \
	    Dockerfile.in > .dockerfile-$(OS)_$(ARCH)
	HASH_LICENSES=$$(find $(LICENSES) -type f                    \
	    | xargs md5sum | md5sum | cut -f1 -d' ');                \
	HASH_BINARY=$$(md5sum bin/$(OS)_$(ARCH)/$(BIN)               \
	    | cut -f1 -d' ');                                        \
	FORCE=0;                                                     \
	if [ -z "$(ALLOW_STALE_APT)" ]; then FORCE=$$(date +%s); fi; \
	docker buildx build                                          \
	    --builder git-sync                                       \
	    --build-arg FORCE_REBUILD="$$FORCE"                      \
	    --build-arg HASH_LICENSES="$$HASH_LICENSES"              \
	    --build-arg HASH_BINARY="$$HASH_BINARY"                  \
	    --progress=plain                                         \
	    --load                                                   \
	    --platform "$(OS)/$(ARCH)"                               \
	    --build-arg HTTP_PROXY=$(HTTP_PROXY)                     \
	    --build-arg HTTPS_PROXY=$(HTTPS_PROXY)                   \
	    -t $(IMAGE):$(OS_ARCH_TAG)                               \
	    -f .dockerfile-$(OS)_$(ARCH)                             \
	    .
	docker images -q $(IMAGE):$(OS_ARCH_TAG) > $@

[rayandas]

Ah! alright. I should try something like this. Thanks!

[mpuckett159]

imo this should really something we have a blog on like "how to do this" rather than a solution we package and ship to people. Everyone has different threat models and security requirements and having something prescriptive like this is a sure fire way to get an unending amount of issues filed for CVEs that we are then on the hook for mitigating, but then we get into dependency management issues too, and down stream users are going to want certain versions of things, other tooling in general, etc, and really the ideal solution (in my mind) is that users have a blueprint for how to accomplish this task so that they are empowered to create what they need.

[thockin]

What is the result here? Did we decide not to do it? A PR was merged and then reverted. I'd like to go back to the problem statement.

There's a slow, but never-ending trickle of user questions that take the form "I need X information from the API in in my pod". Some of those can be satisfied by the downward API or trivial extensions thereof. But some cannot - e.g. cross-object things like "tell me the name of my topmost parent controller resource" would require kubelet to have permissions on a wide swath of types and to take responsibility for watching them all. And we would have to find ways to express all of those in downward API. In short, downward API is really only useful for things that come from the pod resource and even then, only tirivially-formatted things. Anything that requires multi-field, formatting expressions, or logic is just not workable.

The obvious answer is "do it yourself". Use your own pod's ServiceAccount and fetch the information you need. The problem is that this is a non-trivial punt, especially for 3rd party apps.

This proposal was about making a sidecar-oriented image that was appropriate for this use-case.

Are we now saying we don't care to address this?

[thockin]

https://github.com/thockin/kubectl-sidecar