runtime error: invalid memory address or nil pointer dereference

[sananguliyev]

Describe the bug
I have added new volume mounts to k8s configurations and applied them. Afterwards dkron start throwing this error.

P.S. My first try was broken. One of the server was stuck even after I fixed issue and applied again. That's why I evict that pod. Maybe that was related to that.

Logs:

2023/09/18 22:25:04 [Recovery] 2023/09/18 - 22:25:04 panic recovered:
runtime error: invalid memory address or nil pointer dereference
/usr/local/go/src/runtime/panic.go:220 (0x44cb75)
/usr/local/go/src/runtime/signal_unix.go:818 (0x44cb45)
/go/pkg/mod/github.com/hashicorp/raft@v1.5.0/api.go:769 (0xa6475d)
/go/src/github.com/distribworks/dkron/dkron/agent.go:624 (0x1ad55ab)
/go/src/github.com/distribworks/dkron/dkron/ui.go:76 (0x1b038ee)
/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174 (0x1ad9e1e)
/go/src/github.com/distribworks/dkron/dkron/api.go:133 (0x1ad9e05)
/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174 (0x1ad9f52)
/go/src/github.com/distribworks/dkron/dkron/api.go:139 (0x1ad9ec4)
/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174 (0xc1bb01)

[sananguliyev]

And my job disappeared from dKron, too. How can I recover the job and dKron.

[cobolbaby]

I saw this error when I restarted Dkron.

time="2023-09-20T16:45:28+08:00" level=info msg="No valid config found: Applying default values." error="Config File \"dkron\" Not Found in \"[/etc/dkron /root/.dkron /opt/dkron/config]\""

2023/09/20 16:45:31 [Recovery] 2023/09/20 - 16:45:31 panic recovered:
runtime error: invalid memory address or nil pointer dereference
/usr/local/go/src/runtime/panic.go:220 (0x44cb55)
/usr/local/go/src/runtime/signal_unix.go:818 (0x44cb25)
/go/pkg/mod/github.com/hashicorp/raft@v1.5.0/api.go:769 (0xa5b65d)
/go/src/github.com/distribworks/dkron/dkron/agent.go:624 (0x1aa160b)
/go/src/github.com/distribworks/dkron/dkron/api.go:445 (0x1aa8345)
/go/pkg/mod/github.com/gin-gonic/gin@v1.9.0/context.go:174 (0x1aa5e7e)
/go/src/github.com/distribworks/dkron/dkron/api.go:133 (0x1aa5e65)
/go/pkg/mod/github.com/gin-gonic/gin@v1.9.0/context.go:174 (0x1aa5fb2)
/go/src/github.com/distribworks/dkron/dkron/api.go:139 (0x1aa5f24)
/go/pkg/mod/github.com/gin-gonic/gin@v1.9.0/context.go:174 (0xbe7d61)
/go/pkg/mod/github.com/gin-gonic/gin@v1.9.0/recovery.go:102 (0xbe7d4c)
/go/pkg/mod/github.com/gin-gonic/gin@v1.9.0/context.go:174 (0xbe6e46)
/go/pkg/mod/github.com/gin-gonic/gin@v1.9.0/logger.go:240 (0xbe6e29)
/go/pkg/mod/github.com/gin-gonic/gin@v1.9.0/context.go:174 (0xbe5ed0)
/go/pkg/mod/github.com/gin-gonic/gin@v1.9.0/gin.go:620 (0xbe5b38)
/go/pkg/mod/github.com/gin-gonic/gin@v1.9.0/gin.go:576 (0xbe567c)
/usr/local/go/src/net/http/server.go:2916 (0x7023ba)
/usr/local/go/src/net/http/server.go:1966 (0x6fd3b6)
/usr/local/go/src/runtime/asm_amd64.s:1571 (0x4675e0)


...

[cobolbaby]

And my job disappeared from dKron, too. How can I recover the job and dKron.

If the raft data directory is reset due to pod reconstruction, it won't be possible to recover the task unless you have regular backups.

runtime error: invalid memory address or nil pointer dereference

After reviewing the code, I found that the error happens because Serf did not elect a leader. This issue can arise during the service startup process. If the leader cannot be elected for a long time, there would be something wrong with the configuration or network.

[sananguliyev]

First of all thanks for your answer. I am using persistent volume, and I do not understand why data is gone or corrupted. Regarding the no leader problem, configuration and infrastructure is same. it always happens when I restart or apply new version. I always cross finger before I restart. I do not know exact problem since it works arbitrarily since configuration is always same and network issue is very unlikely (at least that often, otherwise other services would not work on k8s)

[cobolbaby]

I am using persistent volume, and I do not understand why data is gone or corrupted.

Could you share the dkron yaml?

[sananguliyev]

I am using persistent volume, and I do not understand why data is gone or corrupted.

Could you share the dkron yaml?

Yes sure. I mount the dkron config files as volume from config map and you can find them below, too.

---
apiVersion: v1
kind: ServiceAccount
metadata:
    name: dkron
    namespace: dkron

---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
    namespace: dkron
    name: dkron-discovery-manager
rules:
    -   apiGroups: [""]
        resources: ["pods"]
        verbs: ["list"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
    name: dkron-cluster-discovery
    namespace: dkron
subjects:
    -   kind: ServiceAccount
        name: dkron
        namespace: dkron
roleRef:
    kind: Role
    name: dkron-discovery-manager
    apiGroup: rbac.authorization.k8s.io

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: dkron-agent
  namespace: dkron
  labels:
      app: dkron
      component: agent
spec:
  replicas: 1
  selector:
    matchLabels:
        app: dkron
        component: agent
  template:
    metadata:
        name: dkron-agent
        labels:
          app: dkron
          component: agent
    spec:
      serviceAccountName: dkron
      terminationGracePeriodSeconds: 30
      affinity: 
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: node.kubernetes.io/instance-type
                operator: In
                values:
                - cpx21
      containers:
      - name: dkron
        image: "dkron/dkron:3.2.6"
        imagePullPolicy: Always
        envFrom:
          - configMapRef:
              name: dkron-env
        command: 
          - /opt/local/dkron/dkron
        args:
          - "agent"
          - "--config=/etc/dkron/dkron-agent.yml"
          - "--tag=\"type=agent\""
        volumeMounts: 
          - name: config
            mountPath: /etc/dkron/
            readOnly: true
          - name: feedless-config
            mountPath: /app/config/
            readOnly: true
        ports:
          - name: serf
            containerPort: 8946
          - name: grpc
            containerPort: 6868
        resources:
          limits: 
            memory: 1Gi
            cpu: 500m
      volumes:
        - name: config
          configMap:
            name: dkron-config
        - name: feedless-config
          configMap:
            name: feedless-config

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: dkron-server
  namespace: dkron
  labels:
    app: dkron
    component: server
spec:
  replicas: 3 
  serviceName: dkron-server
  selector:
    matchLabels:
      app: dkron
      component: server
  template:
    metadata:
      labels: 
        app: dkron
        component: server
    spec:
      serviceAccountName: dkron
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: node.kubernetes.io/instance-type
                operator: In
                values:
                - cpx21
      containers:
        - name: dkron-server
          image: dkron/dkron:3.2.6
          imagePullPolicy: Always
          envFrom:
            - configMapRef:
                name: dkron-env
          ports:
            - name: http
              containerPort: 8080
            - name: serf
              containerPort: 8946
            - name: grpc
              containerPort: 6868
          command: 
           - /opt/local/dkron/dkron
          args:
              - "agent"
              - "--server"
              - "--config=/etc/dkron/dkron-server.yml"
          volumeMounts: 
            - name: data
              mountPath: /dkron/data
            - name: config
              mountPath: /etc/dkron/
              readOnly: true
            - name: feedless-config
              mountPath: /app/config/
              readOnly: true
          # lifecycle:
          #   preStop:
          #     exec:
          #       command: ["dkron", "leave"]
          startupProbe:
            tcpSocket:
              port: serf
            initialDelaySeconds: 30
            periodSeconds: 10
          readinessProbe:
            httpGet:
              path: "/health"
              port: 8080
            failureThreshold: 5
            periodSeconds: 10
            initialDelaySeconds: 10
            successThreshold: 1
            timeoutSeconds: 5
          livenessProbe:
            httpGet:
              path: "/health"
              port: 8080
            failureThreshold: 2
            periodSeconds: 10
            initialDelaySeconds: 5
            successThreshold: 1
            timeoutSeconds: 5
          resources:
            limits: 
              memory: 1Gi
              cpu: 500m
      volumes:
        - name: config
          configMap:
            name: dkron-config
        - name: feedless-config
          configMap:
            name: feedless-config
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: [ "ReadWriteOnce" ]
        resources:
          requests:
            storage: 10Gi

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: dkron-config
  namespace: dkron
data:
  dkron-agent.yml: |+
    data-dir: /dkron/data
    retry-join: ["provider=k8s namespace=\"dkron\" label_selector=\"app=dkron\""]
    tags:
      component: agent
    log-level: info
    serf-reconnect-timeout: 5s
    disable-usage-stats: true
  dkron-server.yml: |+
    server: true
    bootstrap-expect: 2
    data-dir: /dkron/data
    retry-join: ["provider=k8s namespace=\"dkron\" label_selector=\"app=dkron\""]
    tags:
      component: server
    log-level: info
    serf-reconnect-timeout: 5s
    disable-usage-stats: true