Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Shell-operator fails on handling something #429

Open
gunmaden opened this issue Oct 5, 2022 · 2 comments
Open

Shell-operator fails on handling something #429

gunmaden opened this issue Oct 5, 2022 · 2 comments

Comments

@gunmaden
Copy link

gunmaden commented Oct 5, 2022

Expected behavior (what you expected to happen): Periodically error occurred while shell-operator trying to handle something

Actual behavior (what actually happened): Error does not occur, have enough info about problem

Steps to reproduce:

  1. Deploy shell-operator & crd

Additional info:
Tried to enable debug log level but it doesn't help (only logs which entities are processing, each time error message meets in different places)

Environment:

  • Shell-operator version: v1.0.10
  • Kubernetes version: v1.22.12
  • Installation type (kubectl apply, helm chart, etc.): helm chart

Anything else we should know?:

Hook script
    #!/usr/bin/env bash

set -e

echoerr() {
  printf "ERROR. %s\n" "$*" >&2;
}

##### HOOK #####

if [[ $1 == "--config" ]] ; then
    cat <<EOF
configVersion: v1
onStartup: 1
kubernetes:
  - name: OnDiscoveryEvent
    apiVersion: custom.group/v1
    kind: Discovery
    executeHookOnEvent:
      - Added
      - Deleted
  - name: OnServiceEvent
    apiVersion: ""
    kind: Service
    executeHookOnEvent:
      - Added
      - Modified
      - Deleted
EOF
else
  for context in $(jq -c '.[]' "$BINDING_CONTEXT_PATH"); do
    echo "Just log"
  done
fi

Custom resource definition
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  # name must match the spec fields below, and be in the form: .
  name: discoveries.custom.group
spec:
  # group name to use for REST API: /apis//
  group: custom.group
  # either Namespaced or Cluster
  scope: Namespaced
  names:
    # plural name to be used in the URL: /apis///
    plural: discoveries
    # singular name to be used as an alias on the CLI and for display
    singular: discovery
    # kind is normally the CamelCased singular type. Your resource manifests use this.
    kind: Discovery
  # list of versions supported by this CustomResourceDefinition
  versions:
    - name: v1
      # Each version can be enabled/disabled by Served flag.
      served: true
      # One and only one version must be marked as the storage version.
      storage: true
      # The schema for validating custom objects
      schema:
        openAPIV3Schema:
          type: object
Logs
{"level":"info","msg":"shell-operator v1.0.10","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Listen on 0.0.0.0:9115","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Create metric counter discovery_operator_live_ticks","operator.component":"metricStorage","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Create metric histogram discovery_operator_tasks_queue_action_duration_seconds","operator.component":"metricsStorage","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Create metric gauge discovery_operator_tasks_queue_length","operator.component":"metricStorage","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Create metric gauge discovery_operator_kube_snapshot_objects","operator.component":"metricStorage","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Create metric gauge discovery_operator_kube_snapshot_bytes","operator.component":"metricStorage","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Create metric histogram discovery_operator_kube_jq_filter_duration_seconds","operator.component":"metricsStorage","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Create metric histogram discovery_operator_kube_event_duration_seconds","operator.component":"metricsStorage","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Create metric gauge discovery_operator_hook_enable_kubernetes_bindings_seconds","operator.component":"metricStorage","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Create metric counter discovery_operator_hook_enable_kubernetes_bindings_errors_total","operator.component":"metricStorage","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Create metric gauge discovery_operator_hook_enable_kubernetes_bindings_success","operator.component":"metricStorage","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Create metric histogram discovery_operator_hook_run_seconds","operator.component":"metricsStorage","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Create metric histogram discovery_operator_hook_run_user_cpu_seconds","operator.component":"metricsStorage","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Create metric histogram discovery_operator_hook_run_sys_cpu_seconds","operator.component":"metricsStorage","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Create metric gauge discovery_operator_hook_run_max_rss_bytes","operator.component":"metricStorage","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Create metric counter discovery_operator_hook_run_errors_total","operator.component":"metricStorage","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Create metric counter discovery_operator_hook_run_allowed_errors_total","operator.component":"metricStorage","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Create metric counter discovery_operator_hook_run_success_total","operator.component":"metricStorage","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Create metric counter discovery_operator_task_wait_in_queue_seconds_total","operator.component":"metricStorage","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Hooks dir: /hooks","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Use temporary dir: /tmp/shell-operator","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Debug endpoint listen on /var/run/shell-operator/debug.socket","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Create metric histogram discovery_operator_kubernetes_client_request_latency_seconds","operator.component":"metricsStorage","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Create metric counter discovery_operator_kubernetes_client_request_result_total","operator.component":"metricStorage","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Kubernetes client is configured successfully with 'out-of-cluster' config","operator.component":"KubernetesAPIClient","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Kubernetes client is configured successfully with 'out-of-cluster' config","operator.component":"KubernetesAPIClient","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Initialize hooks manager. Search for and load all hooks.","time":"2022-10-05T12:14:24Z"}
{"hook":"hook.sh","level":"info","msg":"Load config from '/hooks/hook.sh'","phase":"config","time":"2022-10-05T12:14:24Z"}
{"hook":"hook.sh","level":"info","msg":"Loaded config: OnStartup:1, Watch k8s kinds: 'Service', 'Discovery'","phase":"config","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"start shell-operator","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"queue task HookRun::onStartup:hook.sh with hook hook.sh","operator.component":"initMainQueue","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"queue task EnableKubernetesBindings:::hook.sh:EnableKubernetesBindings for hook hook.sh","operator.component":"initMainQueue","time":"2022-10-05T12:14:24Z"}
{"binding":"","event":"onStartup","hook":"hook.sh","level":"info","msg":"Execute hook","queue":"","task":"HookRun","time":"2022-10-05T12:14:24Z"}
{"binding":"","event":"onStartup","hook":"hook.sh","level":"info","msg":"Conext","output":"stdout","queue":"","task":"HookRun","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Create metric histogram discovery_operator_hook_run_sys_seconds","operator.component":"metricsStorage","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Create metric histogram discovery_operator_hook_run_user_seconds","operator.component":"metricsStorage","time":"2022-10-05T12:14:24Z"}
{"binding":"","event":"onStartup","hook":"hook.sh","level":"info","msg":"Hook executed successfully","queue":"","task":"HookRun","time":"2022-10-05T12:14:24Z"}
{"binding":"","hook":"hook.sh","level":"info","msg":"Enable kubernetes binding for hook","queue":"main","task":"EnableKubernetesBindings","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Create metric histogram discovery_operator_kubernetes_client_rate_limiter_latency_seconds","operator.component":"metricsStorage","time":"2022-10-05T12:14:24Z"}
{"level":"error","msg":"E1005 12:14:24.777262      10 reflector.go:127] pkg/mod/k8s.io/client-go@v0.19.11/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"I1005 12:14:25.959173      10 request.go:645] Throttling request took 1.076579751s, request: GET:https://10.96.0.1:443/apis/flowcontrol.apiserver.k8s.io/v1beta1?timeout=32s\n","source":"klog","time":"2022-10-05T12:14:25Z"}
{"level":"error","msg":"E1005 12:14:26.034804      10 reflector.go:127] pkg/mod/k8s.io/client-go@v0.19.11/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:14:26Z"}
{"level":"error","msg":"E1005 12:14:28.694095      10 reflector.go:127] pkg/mod/k8s.io/client-go@v0.19.11/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:14:28Z"}
{"level":"error","msg":"E1005 12:14:29.570104      10 reflector.go:127] pkg/mod/k8s.io/client-go@v0.19.11/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:14:29Z"}
{"binding":"","hook":"hook.sh","level":"info","msg":"Kubernetes bindings for hook are enabled successfully, 2 tasks generated","queue":"main","task":"EnableKubernetesBindings","time":"2022-10-05T12:14:29Z"}
{"binding":"OnDiscoveryEvent","event":"kubernetes","hook":"hook.sh","level":"info","msg":"Execute hook","queue":"main","task":"HookRun","time":"2022-10-05T12:14:29Z"}
{"binding":"OnDiscoveryEvent","event":"kubernetes","hook":"hook.sh","level":"info","msg":"Conext","output":"stdout","queue":"main","task":"HookRun","time":"2022-10-05T12:14:29Z"}
{"binding":"OnDiscoveryEvent","event":"kubernetes","hook":"hook.sh","level":"info","msg":"Hook executed successfully","queue":"main","task":"HookRun","time":"2022-10-05T12:14:29Z"}
{"binding":"OnDiscoveryEvent","event":"kubernetes","hook":"hook.sh","level":"info","msg":"Unlock kubernetes.Event tasks","queue":"main","task":"HookRun","time":"2022-10-05T12:14:29Z"}
{"binding":"OnServiceEvent","event":"kubernetes","hook":"hook.sh","level":"info","msg":"Execute hook","queue":"main","task":"HookRun","time":"2022-10-05T12:14:29Z"}
{"binding":"OnServiceEvent","event":"kubernetes","hook":"hook.sh","level":"info","msg":"Conext","output":"stdout","queue":"main","task":"HookRun","time":"2022-10-05T12:14:29Z"}
{"binding":"OnServiceEvent","event":"kubernetes","hook":"hook.sh","level":"info","msg":"Hook executed successfully","queue":"main","task":"HookRun","time":"2022-10-05T12:14:29Z"}
{"binding":"OnServiceEvent","event":"kubernetes","hook":"hook.sh","level":"info","msg":"Unlock kubernetes.Event tasks","queue":"main","task":"HookRun","time":"2022-10-05T12:14:29Z"}
{"level":"error","msg":"E1005 12:14:30.629997      10 reflector.go:127] pkg/mod/k8s.io/client-go@v0.19.11/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:14:30Z"}
{"level":"error","msg":"E1005 12:14:32.461660      10 reflector.go:127] pkg/mod/k8s.io/client-go@v0.19.11/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:14:32Z"}
{"level":"error","msg":"E1005 12:14:33.203631      10 reflector.go:127] pkg/mod/k8s.io/client-go@v0.19.11/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:14:33Z"}
{"level":"error","msg":"E1005 12:14:36.658349      10 reflector.go:127] pkg/mod/k8s.io/client-go@v0.19.11/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:14:36Z"}
{"level":"error","msg":"E1005 12:14:40.836229      10 reflector.go:127] pkg/mod/k8s.io/client-go@v0.19.11/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:14:40Z"}
{"level":"error","msg":"E1005 12:14:45.270768      10 reflector.go:127] pkg/mod/k8s.io/client-go@v0.19.11/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:14:45Z"}
{"level":"error","msg":"E1005 12:15:04.555854      10 reflector.go:127] pkg/mod/k8s.io/client-go@v0.19.11/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:15:04Z"}
{"level":"error","msg":"E1005 12:15:09.467184      10 reflector.go:127] pkg/mod/k8s.io/client-go@v0.19.11/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:15:09Z"}
{"level":"error","msg":"E1005 12:15:36.220781      10 reflector.go:127] pkg/mod/k8s.io/client-go@v0.19.11/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:15:36Z"}
{"level":"error","msg":"E1005 12:15:37.375321      10 reflector.go:127] pkg/mod/k8s.io/client-go@v0.19.11/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:15:37Z"}
{"level":"error","msg":"E1005 12:16:17.142853      10 reflector.go:127] pkg/mod/k8s.io/client-go@v0.19.11/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:16:17Z"}
{"level":"error","msg":"E1005 12:16:25.203734      10 reflector.go:127] pkg/mod/k8s.io/client-go@v0.19.11/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:16:25Z"}
{"level":"error","msg":"E1005 12:17:06.718118      10 reflector.go:127] pkg/mod/k8s.io/client-go@v0.19.11/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:17:06Z"}
{"level":"error","msg":"E1005 12:17:11.263395      10 reflector.go:127] pkg/mod/k8s.io/client-go@v0.19.11/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:17:11Z"}
{"level":"error","msg":"E1005 12:17:46.645551      10 reflector.go:127] pkg/mod/k8s.io/client-go@v0.19.11/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:17:46Z"}
{"level":"error","msg":"E1005 12:17:59.075634      10 reflector.go:127] pkg/mod/k8s.io/client-go@v0.19.11/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:17:59Z"}
{"level":"error","msg":"E1005 12:18:35.905553      10 reflector.go:127] pkg/mod/k8s.io/client-go@v0.19.11/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:18:35Z"}
{"level":"error","msg":"E1005 12:18:53.018460      10 reflector.go:127] pkg/mod/k8s.io/client-go@v0.19.11/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:18:53Z"}
{"level":"error","msg":"E1005 12:19:25.314806      10 reflector.go:127] pkg/mod/k8s.io/client-go@v0.19.11/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:19:25Z"}
@diafour
Copy link
Contributor

diafour commented Dec 11, 2022

Hello! Sorry, I've missed this one for so long. Do you have proper RBAC with rules for the Discovery kind?

@gunmaden
Copy link
Author

gunmaden commented Dec 30, 2022 •
edited

I think root cause is that I've forgot to add watch rules for resources. Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@diafour @gunmaden