Feature Request: tolerations and nodeAffinity

[eljeffeg]

Is your feature request related to a problem? Please describe.
I've been given a dedicated node cluster with sufficient resources for AssemblyLine, but I need to define it using the tolerations and nodeAffinity. Currently, core components allow you to define the nodeAffinity, but not tolerations. Services don't allow you to define either nodeAffinity or tolerations as it's managed by the Scaler.

Describe the solution you'd like
I'd like to be able to define the tolerations and nodeAffinity for core and services in the Helm file.

Describe alternatives you've considered
I've made the changes to allow core to use the tolerations, but it doesn't apply to services. When services spin up, they run out of resources. It seems the service change is currently beyond the helm charts.

[cccs-rs]

Would the scaler.linux_node_selector configuration work in this case as you would just need to specify the selector (whether it be based on fields or labels) that correspond to the node?

https://cybercentrecanada.github.io/assemblyline4_docs/odm/models/config/#scaler

[eljeffeg]

From what I can tell, Field and Label selectors cannot be used to directly dictate scheduling behaviors such as tolerations or node affinity.

[cccs-rs]

Looks like the configuration for the linux_node_selector should translate to a nodeAffinity for the service pods:
https://github.com/CybercentreCanada/assemblyline-core/blob/14bfcd276ac904882724c4eb9075fe0350cf9f3f/assemblyline_core/scaler/controllers/kubernetes_ctl.py#L140

So we would just need to add a configuration for tolerations.

[cccs-rs]

You can test with this development release to make sure you can configure the system as necessary when we merge this into stable (along with the helm-chart changes).

[eljeffeg]

@cccs-rs I pulled the latest dev images and updated my helm charts, but wasn't able to get the nodeAffinity and tolerations on Services. They did show up on all the core services. Here are my respective value.yaml config.

# An affinity to be applied to all core (non-service) pods not provided by imported charts.
# https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#NodeAffinity
nodeAffinity:
  requiredDuringSchedulingIgnoredDuringExecution:
    nodeSelectorTerms:
    - matchExpressions:
      - key: dedicated
        operator: In
        values:
        - assemblyline

# The tolerations to be applied to all core (non-service) pods not provided by imported charts.
# https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#tolerations
tolerations:
  - effect: NoSchedule
    key: dedicated
    operator: Equal
    value: assemblyline
    
configuration:
  core:
    scaler:
      linux_node_selector:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: dedicated
              operator: In
              values:
              - assemblyline