shut down with error error: PANIC: reflect: reflect.Value.Set using value obtained using unexported field

[bigdave777]

Describe the bug
Upon deployment of the Emisarry helm chart, all 3 pods end up crashing whilst throwing the following error
shut down with error error: PANIC: reflect: reflect.Value.Set using value obtained using unexported field

Defaulted container "emissary-ingress" out of: emissary-ingress, wait-for-apiext (init)
time="2024-03-18 14:01:44.3956" level=info msg="Started Ambassador (Version 3.9.1)" func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.Main file="/go/cmd/entrypoint/entrypoint.go:96" CMD=entrypoint PID=1
time="2024-03-18 14:01:44.4428" level=info msg="AMBASSADOR_CLUSTER_ID=96c836c2-9289-5b89-b08c-13b204dbbe48" func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.Main file="/go/cmd/entrypoint/entrypoint.go:100" CMD=entrypoint PID=1
time="2024-03-18 14:01:44.4444" level=info msg="started command [\"diagd\" \"/ambassador/snapshots\" \"/ambassador/bootstrap-ads.json\" \"/ambassador/envoy/envoy.json\" \"--notices\" \"/ambassador/notices.json\" \"--port\" \"8004\" \"--kick\" \"kill -HUP 1\"]" func="github.com/datawire/dlib/dexec.(*Cmd).Start" file="/go/vendor/github.com/datawire/dlib/dexec/cmd.go:183" CMD=entrypoint PID=1 THREAD=/diagd dexec.pid=19
time="2024-03-18 14:01:44.4445" level=info msg="not logging input read from file \"/dev/stdin\"" func="github.com/datawire/dlib/dexec.(*Cmd).Start" file="/go/vendor/github.com/datawire/dlib/dexec/cmd.go:185" CMD=entrypoint PID=1 THREAD=/diagd dexec.pid=19 dexec.stream=stdin
time="2024-03-18 14:01:44.4445" level=info msg="not logging output written to file \"/dev/stdout\"" func="github.com/datawire/dlib/dexec.(*Cmd).Start" file="/go/vendor/github.com/datawire/dlib/dexec/cmd.go:188" CMD=entrypoint PID=1 THREAD=/diagd dexec.pid=19 dexec.stream=stdout
time="2024-03-18 14:01:44.4445" level=info msg="not logging output written to file \"/dev/stderr\"" func="github.com/datawire/dlib/dexec.(*Cmd).Start" file="/go/vendor/github.com/datawire/dlib/dexec/cmd.go:191" CMD=entrypoint PID=1 THREAD=/diagd dexec.pid=19 dexec.stream=stderr
time="2024-03-18 14:01:44.4452" level=info msg="Ambex 3.9.1 starting, snapdirPath /ambassador/snapshots" func=github.com/emissary-ingress/emissary/v3/pkg/ambex.Main file="/go/pkg/ambex/main.go:648" CMD=entrypoint PID=1 THREAD=/ambex
time="2024-03-18 14:01:44.4454" level=info msg="Wrote PID" func=github.com/emissary-ingress/emissary/v3/pkg/ambex.Main file="/go/pkg/ambex/main.go:692" CMD=entrypoint PID=1 THREAD=/ambex fields.file=ambex.pid file="/go/pkg/ambex/main.go:692" pid=1
time="2024-03-18 14:01:44.4456" level=info msg="Listening on tcp:127.0.0.1:8003" func=github.com/emissary-ingress/emissary/v3/pkg/ambex.runManagementServer file="/go/pkg/ambex/main.go:226" CMD=entrypoint PID=1 THREAD=/ambex/management-server
time="2024-03-18 14:01:44.4458" level=info msg="AMBASSADOR_RECONFIG_MAX_DELAY set to 1" func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.WatchAllTheThings file="/go/cmd/entrypoint/watcher.go:47" CMD=entrypoint PID=1 THREAD=/watcher
time="2024-03-18 14:01:44.4458" level=info msg="Saved snapshot v0" func=github.com/emissary-ingress/emissary/v3/pkg/ambex.csDump file="/go/pkg/ambex/main.go:351" CMD=entrypoint PID=1 THREAD=/ambex/main-loop
time="2024-03-18 14:01:44.4461" level=info msg="Pushing snapshot v0" func=github.com/emissary-ingress/emissary/v3/pkg/ambex.updaterWithTicker file="/go/pkg/ambex/ratelimit.go:159" CMD=entrypoint PID=1 THREAD=/ambex/updater
time="2024-03-18 14:01:44.4474" level=warning msg="Warning, unable to watch gatewayclasses.v1alpha1.networking.x-k8s.io, unknown kind." func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.GetInterestingTypes file="/go/cmd/entrypoint/interesting_types.go:156" CMD=entrypoint PID=1 THREAD=/watcher
time="2024-03-18 14:01:44.4474" level=warning msg="Warning, unable to watch gateways.v1alpha1.networking.x-k8s.io, unknown kind." func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.GetInterestingTypes file="/go/cmd/entrypoint/interesting_types.go:156" CMD=entrypoint PID=1 THREAD=/watcher
time="2024-03-18 14:01:44.4475" level=warning msg="Warning, unable to watch httproutes.v1alpha1.networking.x-k8s.io, unknown kind." func=github.com/emissary-ingress/emissary/v3/cmd/entrypoint.GetInterestingTypes file="/go/cmd/entrypoint/interesting_types.go:156" CMD=entrypoint PID=1 THREAD=/watcher
I0318 14:01:45.683439       1 request.go:697] Waited for 1.087854621s due to client-side throttling, not priority and fairness, request: GET:https://172.20.0.1:443/apis/getambassador.io/v3alpha1/tcpmappings?limit=500&resourceVersion=0
2024-03-18 14:01:45 diagd 3.9.1 [P19TMainThread] INFO: AMBASSADOR_FAST_RECONFIGURE enabled, initializing cache
2024-03-18 14:01:45 diagd 3.9.1 [P19TMainThread] INFO: WILL NOT update Mapping status
2024-03-18 14:01:45 diagd 3.9.1 [P19TMainThread] INFO: thread count 17, listening on 127.0.0.1:8004
2024-03-18 14:01:46 diagd 3.9.1 [P19TMainThread] INFO: Ambassador 3.9.1 booted
[2024-03-18 14:01:46 +0000] [19] [INFO] Starting gunicorn 21.2.0
[2024-03-18 14:01:46 +0000] [19] [INFO] Listening at: http://127.0.0.1:8004 (19)
[2024-03-18 14:01:46 +0000] [19] [INFO] Using worker: gthread
[2024-03-18 14:01:46 +0000] [21] [INFO] Booting worker with pid: 21
2024-03-18 14:01:46 diagd 3.9.1 [P21TAEW] INFO: starting Scout checker and timer logger
2024-03-18 14:01:46 diagd 3.9.1 [P21TAEW] INFO: starting event watcher
time="2024-03-18 14:01:51.7814" level=error msg="goroutine \"/watcher/loop\" exited with error: PANIC: reflect: reflect.Value.Set using value obtained using unexported field\ngithub.com/datawire/dlib/dgroup.(*Group).goWorkerCtx.func1.1\n\t/go/vendor/github.com/datawire/dlib/dgroup/group.go:369\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:914\nreflect.flag.mustBeAssignableSlow\n\t/usr/local/go/src/reflect/value.go:269\nreflect.flag.mustBeAssignable\n\t/usr/local/go/src/reflect/value.go:259\nreflect.Value.Set\n\t/usr/local/go/src/reflect/value.go:2254\nk8s.io/apimachinery/pkg/runtime.structFromUnstructured\n\t/go/vendor/k8s.io/apimachinery/pkg/runtime/converter.go:556\nk8s.io/apimachinery/pkg/runtime.fromUnstructured\n\t/go/vendor/k8s.io/apimachinery/pkg/runtime/converter.go:359\nk8s.io/apimachinery/pkg/runtime.(*unstructuredConverter).FromUnstructuredWithValidation\n\t/go/vendor/k8s.io/apimachinery/pkg/runtime/converter.go:247\nk8s.io/apimachinery/pkg/runtime.(*unstructuredConverter).FromUnstructured\n\t/go/vendor/k8s.io/apimachinery/pkg/runtime/converter.go:274\nk8s.io/apimachinery/pkg/runtime.(*Scheme).unstructuredToTyped\n\t/go/vendor/k8s.io/apimachinery/pkg/runtime/scheme.go:539\nk8s.io/apimachinery/pkg/runtime.(*Scheme).convertToVersion\n\t/go/vendor/k8s.io/apimachinery/pkg/runtime/scheme.go:453\nk8s.io/apimachinery/pkg/runtime.(*Scheme).ConvertToVersion\n\t/go/vendor/k8s.io/apimachinery/pkg/runtime/scheme.go:438\ngithub.com/emissary-ingress/emissary/v3/pkg/snapshot/v1.convertAnnotationObject\n\t/go/pkg/snapshot/v1/annotations.go:102\ngithub.com/emissary-ingress/emissary/v3/pkg/snapshot/v1.ValidateAndConvertObject\n\t/go/pkg/snapshot/v1/annotations.go:84\ngithub.com/emissary-ingress/emissary/v3/pkg/snapshot/v1.(*KubernetesSnapshot).PopulateAnnotations\n\t/go/pkg/snapshot/v1/annotations.go:46\ngithub.com/emissary-ingress/emissary/v3/cmd/entrypoint.(*SnapshotHolder).K8sUpdate.func1.2\n\t/go/cmd/entrypoint/watcher.go:448\ngithub.com/emissary-ingress/emissary/v3/pkg/debug.(*Timer).Time\n\t/go/pkg/debug/timer.go:164\ngithub.com/emissary-ingress/emissary/v3/cmd/entrypoint.(*SnapshotHolder).K8sUpdate.func1\n\t/go/cmd/entrypoint/watcher.go:447\ngithub.com/emissary-ingress/emissary/v3/cmd/entrypoint.(*SnapshotHolder).K8sUpdate\n\t/go/cmd/entrypoint/watcher.go:544\ngithub.com/emissary-ingress/emissary/v3/cmd/entrypoint.watchAllTheThingsInternal.func2\n\t/go/cmd/entrypoint/watcher.go:278\ngithub.com/datawire/dlib/dgroup.(*Group).goWorkerCtx.func1\n\t/go/vendor/github.com/datawire/dlib/dgroup/group.go:385\ngithub.com/datawire/dlib/derrgroup.(*Group).Go.func2\n\t/go/vendor/github.com/datawire/dlib/derrgroup/errgroup.go:132\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650" func="github.com/datawire/dlib/dgroup.(*Group).goWorkerCtx.func1.1" file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:380" CMD=entrypoint PID=1 THREAD=/watcher/loop
time="2024-03-18 14:01:51.7815" level=info msg="shutting down (gracefully)..." func="github.com/datawire/dlib/dgroup.(*Group).launchSupervisors.func1" file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:238" CMD=entrypoint PID=1 THREAD="/watcher:shutdown_logger"
time="2024-03-18 14:01:51.7817" level=info msg="  final goroutine statuses:" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:84" CMD=entrypoint PID=1 THREAD="/watcher:shutdown_status"
time="2024-03-18 14:01:51.7817" level=info msg="    /watcher/consul  : exited" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:95" CMD=entrypoint PID=1 THREAD="/watcher:shutdown_status"
time="2024-03-18 14:01:51.7818" level=info msg="    /watcher/loop    : exited with error" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:95" CMD=entrypoint PID=1 THREAD="/watcher:shutdown_status"
time="2024-03-18 14:01:51.7818" level=info msg="    /watcher/notifyCh: exited" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:95" CMD=entrypoint PID=1 THREAD="/watcher:shutdown_status"
time="2024-03-18 14:01:51.7820" level=error msg="goroutine \"/watcher\" exited with error: PANIC: reflect: reflect.Value.Set using value obtained using unexported field\ngithub.com/datawire/dlib/dgroup.(*Group).goWorkerCtx.func1.1\n\t/go/vendor/github.com/datawire/dlib/dgroup/group.go:369\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:914\nreflect.flag.mustBeAssignableSlow\n\t/usr/local/go/src/reflect/value.go:269\nreflect.flag.mustBeAssignable\n\t/usr/local/go/src/reflect/value.go:259\nreflect.Value.Set\n\t/usr/local/go/src/reflect/value.go:2254\nk8s.io/apimachinery/pkg/runtime.structFromUnstructured\n\t/go/vendor/k8s.io/apimachinery/pkg/runtime/converter.go:556\nk8s.io/apimachinery/pkg/runtime.fromUnstructured\n\t/go/vendor/k8s.io/apimachinery/pkg/runtime/converter.go:359\nk8s.io/apimachinery/pkg/runtime.(*unstructuredConverter).FromUnstructuredWithValidation\n\t/go/vendor/k8s.io/apimachinery/pkg/runtime/converter.go:247\nk8s.io/apimachinery/pkg/runtime.(*unstructuredConverter).FromUnstructured\n\t/go/vendor/k8s.io/apimachinery/pkg/runtime/converter.go:274\nk8s.io/apimachinery/pkg/runtime.(*Scheme).unstructuredToTyped\n\t/go/vendor/k8s.io/apimachinery/pkg/runtime/scheme.go:539\nk8s.io/apimachinery/pkg/runtime.(*Scheme).convertToVersion\n\t/go/vendor/k8s.io/apimachinery/pkg/runtime/scheme.go:453\nk8s.io/apimachinery/pkg/runtime.(*Scheme).ConvertToVersion\n\t/go/vendor/k8s.io/apimachinery/pkg/runtime/scheme.go:438\ngithub.com/emissary-ingress/emissary/v3/pkg/snapshot/v1.convertAnnotationObject\n\t/go/pkg/snapshot/v1/annotations.go:102\ngithub.com/emissary-ingress/emissary/v3/pkg/snapshot/v1.ValidateAndConvertObject\n\t/go/pkg/snapshot/v1/annotations.go:84\ngithub.com/emissary-ingress/emissary/v3/pkg/snapshot/v1.(*KubernetesSnapshot).PopulateAnnotations\n\t/go/pkg/snapshot/v1/annotations.go:46\ngithub.com/emissary-ingress/emissary/v3/cmd/entrypoint.(*SnapshotHolder).K8sUpdate.func1.2\n\t/go/cmd/entrypoint/watcher.go:448\ngithub.com/emissary-ingress/emissary/v3/pkg/debug.(*Timer).Time\n\t/go/pkg/debug/timer.go:164\ngithub.com/emissary-ingress/emissary/v3/cmd/entrypoint.(*SnapshotHolder).K8sUpdate.func1\n\t/go/cmd/entrypoint/watcher.go:447\ngithub.com/emissary-ingress/emissary/v3/cmd/entrypoint.(*SnapshotHolder).K8sUpdate\n\t/go/cmd/entrypoint/watcher.go:544\ngithub.com/emissary-ingress/emissary/v3/cmd/entrypoint.watchAllTheThingsInternal.func2\n\t/go/cmd/entrypoint/watcher.go:278\ngithub.com/datawire/dlib/dgroup.(*Group).goWorkerCtx.func1\n\t/go/vendor/github.com/datawire/dlib/dgroup/group.go:385\ngithub.com/datawire/dlib/derrgroup.(*Group).Go.func2\n\t/go/vendor/github.com/datawire/dlib/derrgroup/errgroup.go:132\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650" func="github.com/datawire/dlib/dgroup.(*Group).goWorkerCtx.func1.1" file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:380" CMD=entrypoint PID=1 THREAD=/watcher
time="2024-03-18 14:01:51.7821" level=info msg="shutting down (gracefully)..." func="github.com/datawire/dlib/dgroup.(*Group).launchSupervisors.func1" file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:238" CMD=entrypoint PID=1 THREAD="/ambex:shutdown_logger"
time="2024-03-18 14:01:51.7824" level=info msg="sending SIGINT" func="github.com/datawire/dlib/dexec.(*Cmd).Start.func1" file="/go/vendor/github.com/datawire/dlib/dexec/cmd.go:211" CMD=entrypoint PID=1 THREAD=/diagd
time="2024-03-18 14:01:51.7825" level=info msg="shutting down (gracefully)..." func="github.com/datawire/dlib/dgroup.(*Group).launchSupervisors.func1" file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:238" CMD=entrypoint PID=1 THREAD=":shutdown_logger"
[2024-03-18 14:01:51 +0000] [19] [INFO] Handling signal: int
time="2024-03-18 14:01:51.7891" level=info msg="Memory Usage 0.44Gi (37%)\n    PID 1, 0.44Gi: busyambassador entrypoint \n    PID 19, 0.04Gi: /usr/bin/python /usr/bin/diagd /ambassador/snapshots /ambassador/bootstrap-ads.json /ambassador/envoy/envoy.json --notices /ambassador/notices.json --port 8004 --kick kill -HUP 1 \n    PID 21, 0.04Gi: /usr/bin/python /usr/bin/diagd /ambassador/snapshots /ambassador/bootstrap-ads.json /ambassador/envoy/envoy.json --notices /ambassador/notices.json --port 8004 --kick kill -HUP 1 " func="github.com/emissary-ingress/emissary/v3/pkg/memory.(*MemoryUsage).Watch" file="/go/pkg/memory/memory.go:43" CMD=entrypoint PID=1 THREAD=/memory
[2024-03-18 14:01:51 +0000] [21] [INFO] Worker exiting (pid: 21)
[2024-03-18 14:01:51 +0000] [19] [INFO] Shutting down: Master
time="2024-03-18 14:01:52.0469" level=info msg="finished successfully: exit status 0" func="github.com/datawire/dlib/dexec.(*Cmd).Wait" file="/go/vendor/github.com/datawire/dlib/dexec/cmd.go:255" CMD=entrypoint PID=1 THREAD=/diagd dexec.pid=19
time="2024-03-18 14:01:52.0471" level=info msg="  final goroutine statuses:" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:84" CMD=entrypoint PID=1 THREAD=":shutdown_status"
time="2024-03-18 14:01:52.0472" level=info msg="    /ambex                   : exited" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:95" CMD=entrypoint PID=1 THREAD=":shutdown_status"
time="2024-03-18 14:01:52.0472" level=info msg="    /diagd                   : exited" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:95" CMD=entrypoint PID=1 THREAD=":shutdown_status"
time="2024-03-18 14:01:52.0473" level=info msg="    /envoy                   : exited" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:95" CMD=entrypoint PID=1 THREAD=":shutdown_status"
time="2024-03-18 14:01:52.0473" level=info msg="    /external_snapshot_server: exited" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:95" CMD=entrypoint PID=1 THREAD=":shutdown_status"
time="2024-03-18 14:01:52.0473" level=info msg="    /healthchecks            : exited" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:95" CMD=entrypoint PID=1 THREAD=":shutdown_status"
time="2024-03-18 14:01:52.0474" level=info msg="    /memory                  : exited" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:95" CMD=entrypoint PID=1 THREAD=":shutdown_status"
time="2024-03-18 14:01:52.0474" level=info msg="    /snapshot_server         : exited" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:95" CMD=entrypoint PID=1 THREAD=":shutdown_status"
time="2024-03-18 14:01:52.0475" level=info msg="    /watcher                 : exited with error" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:95" CMD=entrypoint PID=1 THREAD=":shutdown_status"
time="2024-03-18 14:01:52.0475" level=error msg="shut down with error error: PANIC: reflect: reflect.Value.Set using value obtained using unexported field" func=github.com/emissary-ingress/emissary/v3/pkg/busy.Main file="/go/pkg/busy/busy.go:87" CMD=entrypoint PID=1

To Reproduce
Deploy emissary using version: "8.9.1" and observe the logs
I've added the following config

  use_proxy_proto: true

resources:
  limits:
    memory: 1200Mi

  requests:
    memory: 600Mi

Expected behavior
Emissary pods should remain in a healthy state following the charts deployment

Versions (please complete the following information):

• Emissary 8.9.1 helm chart
• EKS 1.2.5

[gabrielb77]

Same problem here
EKS 1.29
Ambassador 3.9.1

NAME                                READY   STATUS             RESTARTS      AGE
emissary-ingress-78b8fbc589-9h9g8   0/1     CrashLoopBackOff   4 (5s ago)    105s
emissary-ingress-7bdb565474-6r7f7   0/1     CrashLoopBackOff   3 (46s ago)   105s
emissary-ingress-7f74fcb567-ld9pz   0/1     CrashLoopBackOff   3 (42s ago)   104s
emissary-ingress-c9745cddd-hgxjw    0/1     CrashLoopBackOff   4 (4s ago)    104s

time="2024-04-16 15:54:29.4007" level=error msg="goroutine \"/watcher\" exited with error: PANIC: reflect: reflect.Value.Set using value obtained using unexported field\ngithub.com/datawire/dlib/dgroup.(*Group).goWorkerCtx.func1.1\n\t/go/vendor/github.com/datawire/dlib/dgroup/group.go:369\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:914\nreflect.flag.mustBeAssignableSlow\n\t/usr/local/go/src/reflect/value.go:269\nreflect.flag.mustBeAssignable\n\t/usr/local/go/src/reflect/value.go:259\nreflect.Value.Set\n\t/usr/local/go/src/reflect/value.go:2254\nk8s.io/apimachinery/pkg/runtime.structFromUnstructured\n\t/go/vendor/k8s.io/apimachinery/pkg/runtime/converter.go:556\nk8s.io/apimachinery/pkg/runtime.fromUnstructured\n\t/go/vendor/k8s.io/apimachinery/pkg/runtime/converter.go:359\nk8s.io/apimachinery/pkg/runtime.(*unstructuredConverter).FromUnstructuredWithValidation\n\t/go/vendor/k8s.io/apimachinery/pkg/runtime/converter.go:247\nk8s.io/apimachinery/pkg/runtime.(*unstructuredConverter).FromUnstructured\n\t/go/vendor/k8s.io/apimachinery/pkg/runtime/converter.go:274\nk8s.io/apimachinery/pkg/runtime.(*Scheme).unstructuredToTyped\n\t/go/vendor/k8s.io/apimachinery/pkg/runtime/scheme.go:539\nk8s.io/apimachinery/pkg/runtime.(*Scheme).convertToVersion\n\t/go/vendor/k8s.io/apimachinery/pkg/runtime/scheme.go:453\nk8s.io/apimachinery/pkg/runtime.(*Scheme).ConvertToVersion\n\t/go/vendor/k8s.io/apimachinery/pkg/runtime/scheme.go:438\ngithub.com/emissary-ingress/emissary/v3/pkg/snapshot/v1.convertAnnotationObject\n\t/go/pkg/snapshot/v1/annotations.go:102\ngithub.com/emissary-ingress/emissary/v3/pkg/snapshot/v1.ValidateAndConvertObject\n\t/go/pkg/snapshot/v1/annotations.go:84\ngithub.com/emissary-ingress/emissary/v3/pkg/snapshot/v1.(*KubernetesSnapshot).PopulateAnnotations\n\t/go/pkg/snapshot/v1/annotations.go:46\ngithub.com/emissary-ingress/emissary/v3/cmd/entrypoint.(*SnapshotHolder).K8sUpdate.func1.2\n\t/go/cmd/entrypoint/watcher.go:448\ngithub.com/emissary-ingress/emissary/v3/pkg/debug.(*Timer).Time\n\t/go/pkg/debug/timer.go:164\ngithub.com/emissary-ingress/emissary/v3/cmd/entrypoint.(*SnapshotHolder).K8sUpdate.func1\n\t/go/cmd/entrypoint/watcher.go:447\ngithub.com/emissary-ingress/emissary/v3/cmd/entrypoint.(*SnapshotHolder).K8sUpdate\n\t/go/cmd/entrypoint/watcher.go:544\ngithub.com/emissary-ingress/emissary/v3/cmd/entrypoint.watchAllTheThingsInternal.func2\n\t/go/cmd/entrypoint/watcher.go:278\ngithub.com/datawire/dlib/dgroup.(*Group).goWorkerCtx.func1\n\t/go/vendor/github.com/datawire/dlib/dgroup/group.go:385\ngithub.com/datawire/dlib/derrgroup.(*Group).Go.func2\n\t/go/vendor/github.com/datawire/dlib/derrgroup/errgroup.go:132\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650" func="github.com/datawire/dlib/dgroup.(*Group).goWorkerCtx.func1.1" file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:380" CMD=entrypoint PID=1 THREAD=/watcher
[2024-04-16 15:54:29 +0000] [13] [INFO] Handling signal: int
[2024-04-16 15:54:29 +0000] [17] [INFO] Worker exiting (pid: 17)
[2024-04-16 15:54:29 +0000] [13] [INFO] Shutting down: Master
time="2024-04-16 15:54:29.8158" level=error msg="shut down with error error: PANIC: reflect: reflect.Value.Set using value obtained using unexported field" func=github.com/emissary-ingress/emissary/v3/pkg/busy.Main file="/go/pkg/busy/busy.go:87" CMD=entrypoint PID=1

[bigdave777]

FWIW I had an ambassador resource still running in my cluster, and once I had removed that this error stopped happening.

[gabrielb77]

FWIW I had an ambassador resource still running in my cluster, and once I had removed that this error stopped happening.

Do you remember what's resource you removed?

[martineduvaz]

FWIW I had an ambassador resource still running in my cluster, and once I had removed that this error stopped happening.

What resource did you deleted? thanks!

[bigdave777]

I believe it was a related to the auth service https://www.getambassador.io/docs/edge-stack/latest/topics/running/services/auth-service . In my particular case, it was nestled in the annotations for a service.