[Helm Template] Creating Cron Jobs and Queue Workers

[rlcurrall]

The project I have been working on has a lot of background processes that we run to do various data manipulation and processing. Currently, we are running 8 different crons and have 6 different queue transports that we have dedicated consumers for.

My question is, how have others approached the problem of defining the cronjobs and queue worker deployments, and is there a recommended way?

---

I'll share what I've built in my system to make it a bit easier for us to define these deployments.

CronJob

We've created a CronJob template file that allows you to define your crons as an array in your values.yaml file like so:

cronjob:
  # defaults

  crons:
    - name: some-cron
      command: ./bin/console my:command --env=prod --no-debug --no-interaction --no-ansi -vv
      schedule: "0 * * * *"

templates/cron.yaml

{{- $fullName := include "fullname" . -}}
{{- $name := "cronjob" -}}
{{- $data := dict "name" $name "Chart" $.Chart "Release" $.Release "Values" $.Values -}}
{{- range $job, $val := .Values.cronjob.crons }}
apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: {{ $fullName }}-{{ $name }}-{{ .name }}
  labels:
    {{- include "labels" $data | nindent 4 }}
spec:
  schedule: {{ .schedule | quote }}
  concurrencyPolicy: {{ .concurrencyPolicy | default $.Values.cronjob.concurrencyPolicy }}
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  startingDeadlineSeconds: {{ .startingDeadlineSeconds | default $.Values.cronjob.startingDeadlineSeconds }}
  jobTemplate:
    spec:
      backoffLimit: {{ .backoffLimit | default $.Values.cronjob.backoffLimit }}
      activeDeadlineSeconds: {{ .activeDeadlineSeconds | default $.Values.cronjob.activeDeadlineSeconds}}
      template:
        metadata:
          labels:
            {{- include "labels" $data | nindent 12 }}
        spec:
          {{- with $.Values.cronjob.imagePullSecrets }}
          imagePullSecrets:
            {{- toYaml . | nindent 12 }}
          {{- end }}
          restartPolicy: {{ $.Values.cronjob.restartPolicy }}
          containers:
            - name: {{ $.Chart.Name }}-{{ $name }}
              image: "{{ $.Values.cronjob.image.repository }}:{{ $.Values.cronjob.image.tag }}"
              imagePullPolicy: {{ $.Values.cronjob.image.pullPolicy }}
              args:
                - sh
                - -c
                - |
                  {{ .command }}
              ports:
                - containerPort: {{ $.Values.cronjob.service.internalPort }}
              env: [] # Your env vars here
              volumeMounts: [] # Your mounts here
          volumes: [] # Your volumes here
 ---
{{- end}}

Queue Workers

We've created a Deployment template that allows you to define your queue workers in an array in your values.yaml file like so:

queue:
  # defaults

  workers:
    - name: default
      transports: async

templates/worker.yaml

{{- $fullName := include "fullname" . -}}
{{- $name := "worker" -}}
{{- $data := dict "name" $name "Chart" $.Chart "Release" $.Release "Values" $.Values -}}
{{- range $job, $val := .Values.queue.workers }}
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ $fullName }}-{{ $name }}-{{ .name }}
  labels:
    {{- include "labels" $data | nindent 4 }}
spec:
  selector:
    matchLabels:
      {{- include "labels" $data | nindent 6 }}
  template:
    metadata:
      labels:
        {{- include "labels" $data | nindent 8 }}
    spec:
      {{- with $.Values.nodeSelector }}
      nodeSelector:
        {{ toYaml . | nindent 8 }}
      {{- end }}
      {{- with $.Values.affinity }}
      affinity:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      {{- with $.Values.tolerations }}
      tolerations:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      terminationGracePeriodSeconds: {{ $.Values.queue.terminationGracePeriodSeconds }}
      {{- with $.Values.queue.imagePullSecrets }}
      imagePullSecrets:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      containers:
        - name: {{ $.Chart.Name }}-{{ $name }}
          image: "{{ $.Values.queue.image.repository }}:{{ $.Values.queue.image.tag }}"
          imagePullPolicy: {{ $.Values.queue.image.pullPolicy }}
          command:
            - /bin/sh
            - -c
            - >
              sleep 3;
              rm -rf var/cache/*;
              bin/console cache:clear;
              until [ -f /tmp/stop-worker ]; do
                bin/console messenger:consume {{ .transports }} \
                  -vv --env=prod --no-ansi --no-interaction \
                  --memory-limit={{ .memoryLimit | default "128M" }} \
                  --sleep={{ .sleep | default 1 }}
              done;
              rm /tmp/stop-worker;
          ports:
            - containerPort: {{ $.Values.queue.service.internalPort }}
          lifecycle:
            preStop:
              exec:
                command:
                  - /bin/sh
                  - -c
                  - >
                    touch /tmp/stop-worker;
                    until [ ! -f /tmp/stop-worker ]; do
                      echo "Waiting for worker to stop";
                      sleep 5;
                    done;
                    echo "Exiting";
          readinessProbe:
            exec:
              command: ["/bin/sh", "-c", "/bin/ps -ef | grep messenger:consume | grep -v grep"]
            initialDelaySeconds: 10
            periodSeconds: 3
          resources:
            {{- if .resources }} {{ toYaml .resources | nindent 12 }}
            {{- else }} {{ toYaml $.Values.queue.resources | nindent 12 }}
            {{- end }}
          env: [] # Your env vars here
          volumeMounts: [] # Your mounts here
      volumes: [] # Your volumes here
 ---
{{- end }}

---

Disclaimer: I am by no means an expert on Helm, and the charts I've added have gone through many different iterations and are probably not the ideal solution, but they have been working for me.

I was just curious what other solutions people have been using to solve these problems.