feat: refactor kubernetes helm chart

[JuniorJPDJ]

• ConfigMap -> Secret (we are holding passwords and hmac_key here)
• Ingress support
• standard labels
• better object naming - to name objects invidious instead of invidious-invidious if helm release is called invidious
• ability to disable internal postgres and use external one
• smaller image for pg_isready use (wait_for_postgres initContainer)
• envFrom instead of env with config (simpler syntax)
• more options to specify securityPolicy (and podSecurityPolicy separatly)
• tolerations, affinity and nodeSelector support
• sort the config with original config order

[perennialtech]

Wouldn't it better to leave INVIDIOUS_CONFIG as a configmap, with secrets being templated in as needed? Having a separate secrets.yaml would also sensitive info to actually be encrypted (K8s Secret resources aren't actually encrypted by default), for example by using the helm-secrets plugin, without making changes to INVIDIOUS_CONFIG needlessly difficult.

For example:

# secrets.yaml
postgresql:
  auth:
    username: kemal
    password: kemal
  primary:
    initdb:
      username: kemal
      password: kemal

config:
  db:
    user: kemal
    password: kemal
  hmac_key: hmac_key

# values.yaml
postgresql:
  enabled: true
  image:
    tag: 16
  auth:
    # username: SECRET
    # password: SECRET
    database: invidious
  primary:
    initdb:
      # username: SECRET
      # password: SECRET
      scriptsConfigMap: invidious-postgresql-init

config:
  db:
    # user: kemal
    # password: kemal
    host: invidious-postgresql
    port: 5432
    dbname: invidious
  # hmac_key: SECRET

[unixfox]

I do agree on that.

[JuniorJPDJ]

Well, I wanted to do the same but I lack the idea how to do this properly. It would be easier if config options would be a separate env vars, but this is one env var and we are unable to concatenate it in helm chart before pushing it to the pod/container.

[JuniorJPDJ]

Also the solution here allows the usage of helm-secret without any difficulty - it's an object, not a string in the values, so you can split the secret parts to other values file and encrypt it with helm-secrets using SOPS or provide it externally using vals.
Exactly the same solution as you proposed with snippets will work now without any changes.

[JuniorJPDJ]

Secrets aren't encrypted by default, but ConfigMaps cannot be encrypted at all.
If we have an idea how we can split part of config to go to ConfigMap and Secret we can then roll it back to ConfigMap and selectively move parts of config values to Secret. If we have no idea how to do this, I'd personally prefer to have whole INVIDIOUS_CONFIG env var in the Secret due to sensitive values inside.

Encryption of values being input to helm chart doesn't change anything here, as those values will be stored unencrypted in etcd even when someone enabled Secret encryption in the cluster due to being just a ConfigMap. Also it will be able to be accessed by applications running in the cluster with ConfigMap access, but without Secret access.