Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Panic causing CrashLoopBackOff. #348

Open
GrahamDumpleton opened this issue Jan 27, 2022 · 2 comments
Open

Panic causing CrashLoopBackOff. #348

GrahamDumpleton opened this issue Jan 27, 2022 · 2 comments

Comments

@GrahamDumpleton
Copy link

Expected behavior (what you expected to happen):

No crash. :-)

Actual behavior (what actually happened):

Am still trying to narrow down the cause, but relates to using KUBERNETES_PATCH_PATH to have the Shell-operator apply changes back to the cluster. Something in a resource that is being passed back is causing the Shell-operator to crash. The error logs are:

time="2022-01-27T01:42:40Z" level=error msg="E0127 01:42:40.326099       7 runtime.go:78] Observed a panic: &errors.errorString{s:\"cannot deep copy int\"} (cannot deep copy int)\ngoroutine 9 [running]:\nk8s.io/apimachinery/pkg/util/runtime.logPanic(0x17c1a40, 0xc00070fe00)\n\t/go/pkg/mod/k8s.io/apimachinery@v0.19.11/pkg/util/runtime/runtime.go:74 +0xa6\nk8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)\n\t/go/pkg/mod/k8s.io/apimachinery@v0.19.11/pkg/util/runtime/runtime.go:48 +0x86\npanic(0x17c1a40, 0xc00070fe00)\n\t/usr/local/go/src/runtime/panic.go:965 +0x1b9\nk8s.io/apimachinery/pkg/runtime.DeepCopyJSONValue(0x177ac20, 0x267bde8, 0xc0008ae940, 0xc00054d440)\n\t/go/pkg/mod/k8s.io/apimachinery@v0.19.11/pkg/runtime/converter.go:466 +0x550\nk8s.io/apimachinery/pkg/runtime.DeepCopyJSONValue(0x17d7b40, 0xc0004ff230, 0xc0006335d0, 0x8)\n\t/go/pkg/mod/k8s.io/apimachinery@v0.19.11/pkg/runtime/converter.go:450 +0x177\nk8s.io/apimachinery/pkg/runtime.DeepCopyJSONValue(0x17d7b40, 0xc0004ff1a0, 0xc8, 0x1bc50a0)\n\t/go/pkg/mod/k8s.io/apimachinery@v0.19.11/pkg/runtime/converter.go:450 +0x177\nk8s.io/apimachinery/pkg/runtime.DeepCopyJSON(...)\n\t/go/pkg/mod/k8s.io/apimachinery@v0.19.11/pkg/runtime/converter.go:435\nk8s.io/apimachinery/pkg/apis/meta/v1/unstructured.(*Unstructured).DeepCopy(0xc000010290, 0x5)\n\t/go/pkg/mod/k8s.io/apimachinery@v0.19.11/pkg/apis/meta/v1/unstructured/unstructured.go:146 +0x75\ngithub.com/flant/shell-operator/pkg/kube/object_patch.(*ObjectPatcher).executeCreateOperation.func2(0xa93c8f, 0xc0005d5540)\n\t/app/pkg/kube/object_patch/patch.go:129 +0x3fa\nk8s.io/client-go/util/retry.OnError.func1(0x0, 0x0, 0x0)\n\t/go/pkg/mod/k8s.io/client-go@v0.19.11/util/retry/util.go:51 +0x3c\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection(0xc0008aee18, 0x100, 0x0, 0x0)\n\t/go/pkg/mod/k8s.io/apimachinery@v0.19.11/pkg/util/wait/wait.go:211 +0x69\nk8s.io/apimachinery/pkg/util/wait.ExponentialBackoff(0x989680, 0x4014000000000000, 0x3fb999999999999a, 0x4, 0x0, 0xc0008aee18, 0x0, 0x0)\n\t/go/pkg/mod/k8s.io/apimachinery@v0.19.11/pkg/util/wait/wait.go:399 +0x55\nk8s.io/client-go/util/retry.OnError(0x989680, 0x4014000000000000, 0x3fb999999999999a, 0x4, 0x0, 0x1a881d0, 0xc0008af050, 0x0, 0x0)\n\t/go/pkg/mod/k8s.io/client-go@v0.19.11/util/retry/util.go:50 +0xa6\nk8s.io/client-go/util/retry.RetryOnConflict(...)\n\t/go/pkg/mod/k8s.io/client-go@v0.19.11/util/retry/util.go:104\ngithub.com/flant/shell-operator/pkg/kube/object_patch.(*ObjectPatcher).executeCreateOperation(0xc0002c6978, 0xc000720840, 0x1bc4601, 0xc00070f030)\n\t/app/pkg/kube/object_patch/patch.go:118 +0x745\ngithub.com/flant/shell-operator/pkg/kube/object_patch.(*ObjectPatcher).ExecuteOperation(0xc0002c6978, 0x1bc38a0, 0xc000720840, 0x16, 0xc0008af170)\n\t/app/pkg/kube/object_patch/patch.go:65 +0x130\ngithub.com/flant/shell-operator/pkg/kube/object_patch.(*ObjectPatcher).ExecuteOperations(0xc0002c6978, 0xc0007d9100, 0x6, 0x8, 0x0, 0x0)\n\t/app/pkg/kube/object_patch/patch.go:49 +0x236\ngithub.com/flant/shell-operator/pkg/shell-operator.(*ShellOperator).HandleRunHook(0xc0002bd520, 0x1c12f68, 0xc000098280, 0xc000457cc0, 0xc0002cf4e4, 0x11, 0xc0006f0b10, 0xf, 0x0, 0x0, ...)\n\t/app/pkg/shell-operator/operator.go:761 +0x485\ngithub.com/flant/shell-operator/pkg/shell-operator.(*ShellOperator).TaskHandleHookRun(0xc0002bd520, 0x1c12f68, 0xc000098280, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\t/app/pkg/shell-operator/operator.go:708 +0xbe8\ngithub.com/flant/shell-operator/pkg/shell-operator.(*ShellOperator).TaskHandler(0xc0002bd520, 0x1c12f68, 0xc000098280, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\t/app/pkg/shell-operator/operator.go:538 +0x1f0\ngithub.com/flant/shell-operator/pkg/task/queue.(*TaskQueue).Start.func1(0xc000698420)\n\t/app/pkg/task/queue/task_queue.go:342 +0x3fc\ncreated by github.com/flant/shell-operator/pkg/task/queue.(*TaskQueue).Start\n\t/app/pkg/task/queue/task_queue.go:320 +0x4f\n" source=klog
panic: cannot deep copy int [recovered]
	panic: cannot deep copy int

goroutine 9 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/pkg/mod/k8s.io/apimachinery@v0.19.11/pkg/util/runtime/runtime.go:55 +0x109
panic(0x17c1a40, 0xc00070fe00)
	/usr/local/go/src/runtime/panic.go:965 +0x1b9
k8s.io/apimachinery/pkg/runtime.DeepCopyJSONValue(0x177ac20, 0x267bde8, 0xc0008ae940, 0xc00054d440)
	/go/pkg/mod/k8s.io/apimachinery@v0.19.11/pkg/runtime/converter.go:466 +0x550
k8s.io/apimachinery/pkg/runtime.DeepCopyJSONValue(0x17d7b40, 0xc0004ff230, 0xc0006335d0, 0x8)
	/go/pkg/mod/k8s.io/apimachinery@v0.19.11/pkg/runtime/converter.go:450 +0x177
k8s.io/apimachinery/pkg/runtime.DeepCopyJSONValue(0x17d7b40, 0xc0004ff1a0, 0xc8, 0x1bc50a0)
	/go/pkg/mod/k8s.io/apimachinery@v0.19.11/pkg/runtime/converter.go:450 +0x177
k8s.io/apimachinery/pkg/runtime.DeepCopyJSON(...)
	/go/pkg/mod/k8s.io/apimachinery@v0.19.11/pkg/runtime/converter.go:435
k8s.io/apimachinery/pkg/apis/meta/v1/unstructured.(*Unstructured).DeepCopy(0xc000010290, 0x5)
	/go/pkg/mod/k8s.io/apimachinery@v0.19.11/pkg/apis/meta/v1/unstructured/unstructured.go:146 +0x75
github.com/flant/shell-operator/pkg/kube/object_patch.(*ObjectPatcher).executeCreateOperation.func2(0xa93c8f, 0xc0005d5540)
	/app/pkg/kube/object_patch/patch.go:129 +0x3fa
k8s.io/client-go/util/retry.OnError.func1(0x0, 0x0, 0x0)
	/go/pkg/mod/k8s.io/client-go@v0.19.11/util/retry/util.go:51 +0x3c
k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection(0xc0008aee18, 0x100, 0x0, 0x0)
	/go/pkg/mod/k8s.io/apimachinery@v0.19.11/pkg/util/wait/wait.go:211 +0x69
k8s.io/apimachinery/pkg/util/wait.ExponentialBackoff(0x989680, 0x4014000000000000, 0x3fb999999999999a, 0x4, 0x0, 0xc0008aee18, 0x0, 0x0)
	/go/pkg/mod/k8s.io/apimachinery@v0.19.11/pkg/util/wait/wait.go:399 +0x55
k8s.io/client-go/util/retry.OnError(0x989680, 0x4014000000000000, 0x3fb999999999999a, 0x4, 0x0, 0x1a881d0, 0xc0008af050, 0x0, 0x0)
	/go/pkg/mod/k8s.io/client-go@v0.19.11/util/retry/util.go:50 +0xa6
k8s.io/client-go/util/retry.RetryOnConflict(...)
	/go/pkg/mod/k8s.io/client-go@v0.19.11/util/retry/util.go:104
github.com/flant/shell-operator/pkg/kube/object_patch.(*ObjectPatcher).executeCreateOperation(0xc0002c6978, 0xc000720840, 0x1bc4601, 0xc00070f030)
	/app/pkg/kube/object_patch/patch.go:118 +0x745
github.com/flant/shell-operator/pkg/kube/object_patch.(*ObjectPatcher).ExecuteOperation(0xc0002c6978, 0x1bc38a0, 0xc000720840, 0x16, 0xc0008af170)
	/app/pkg/kube/object_patch/patch.go:65 +0x130
github.com/flant/shell-operator/pkg/kube/object_patch.(*ObjectPatcher).ExecuteOperations(0xc0002c6978, 0xc0007d9100, 0x6, 0x8, 0x0, 0x0)
	/app/pkg/kube/object_patch/patch.go:49 +0x236
github.com/flant/shell-operator/pkg/shell-operator.(*ShellOperator).HandleRunHook(0xc0002bd520, 0x1c12f68, 0xc000098280, 0xc000457cc0, 0xc0002cf4e4, 0x11, 0xc0006f0b10, 0xf, 0x0, 0x0, ...)
	/app/pkg/shell-operator/operator.go:761 +0x485
github.com/flant/shell-operator/pkg/shell-operator.(*ShellOperator).TaskHandleHookRun(0xc0002bd520, 0x1c12f68, 0xc000098280, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/app/pkg/shell-operator/operator.go:708 +0xbe8
github.com/flant/shell-operator/pkg/shell-operator.(*ShellOperator).TaskHandler(0xc0002bd520, 0x1c12f68, 0xc000098280, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/app/pkg/shell-operator/operator.go:538 +0x1f0
github.com/flant/shell-operator/pkg/task/queue.(*TaskQueue).Start.func1(0xc000698420)
	/app/pkg/task/queue/task_queue.go:342 +0x3fc
created by github.com/flant/shell-operator/pkg/task/queue.(*TaskQueue).Start
	/app/pkg/task/queue/task_queue.go:320 +0x4f

As it crashes and brings down the pod, it restarts, but as the updates couldn't be applied to the cluster, after restart the same action gets triggered again via Synchronization rather than Event, and crashes once more.

Environment:

Shell-operator version:
flant/shell-operator:v1.0.7

Kubernetes version:

Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.4", GitCommit:"b695d79d4f967c403a96986f1750a35eb75e75f1", GitTreeState:"clean", BuildDate:"2021-11-17T15:48:33Z", GoVersion:"go1.16.10", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.1+k3s2", GitCommit:"a0cadcd3436745396c82821b4a0803ee460401e1", GitTreeState:"clean", BuildDate:"2022-01-14T08:22:21Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"}
  • Installation type (kubectl apply, helm chart, etc.):

Custom deployment using kapp.

Anything else we should know?:

I will update with more details in comments when narrow down which specific Kubernetes resource type is triggering the issue.
@GrahamDumpleton
Copy link
Author

Seems to be enough to output any resource which contains a property with an integer field and write it to KUBERNETES_PATCH_PATH. Eg.,

apiVersion: v1
kind: Service
metadata:
  name: xxx
  namespace: default
spec:
  type: ClusterIP
  ports:
    - name: http
      port: 5000
      targetPort: 5000
      protocol: TCP
  selector:
    app: xxx

Also fails with an Ingress and Deployment, thus presumably also resources such as StatefulSet and DaemonSet where ports are specified.

Previously I had only been creating Secret resources using Shell-operator so hadn't encountered the problem. This issue sort of makes KUBERNETES_PATCH_PATH unusable for more complex use cases. :-(

@GrahamDumpleton
Copy link
Author

Possibly relevant issue:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@GrahamDumpleton