HostTailes / fileTailers: ability to mount additional hostPaths more easily

[PlayMTL]

Is your feature request related to a problem? Please describe.

The Rancher Kubernetes Engine (RKE) creates a directory "/var/lib/rancher/rke/log" from which log files under "/var/lib/docker/containers" are then pointed to via symlink. The Problem is, that the path /var/lib/docker/containers is not mounted, because the defined path in the fileTailers of the HostTailers is "/var/lib/rancher/rke/log". So the target file cannot be found in the fluentbit container because the host path has not been mounted.

Describe the solution you'd like
It would be cool if you could simply define additional "mount paths" in the fileTailers without configuring additional volume mounts.

Describe alternatives you've considered
My current workaround looks like this:

apiVersion: logging-extensions.banzaicloud.io/v1alpha1
kind: HostTailer
metadata:
  name: rke-logs
spec:
  fileTailers:
    - containerOverrides:
        volumeMounts:
          - mountPath: /var/lib/docker/containers
            name: host-docker-path
            readOnly: true
          - mountPath: /var/pos
            name: positions
          - mountPath: /var/lib/rancher/rke/log
            name: var-lib-rancher-rke-log
      name: kublet
      path: /var/lib/rancher/rke/log/kubelet_*.log
  workloadOverrides:
    volumes:
      - hostPath:
          path: /var/lib/docker/containers
          type: Directory
        name: host-docker-path
      - hostPath:
          path: /var/pos
          type: ''
        name: positions
      - hostPath:
          path: /var/lib/rancher/rke/log
          type: ''
        name: var-lib-rancher-rke-log

Additional context

Is a little bit related to: #984

[OverOrion]

Hey @PlayMTL, thanks for opening this issue!

Could you please share some more details?

• Your Logging operator version?
• Can you upgrade to a recent one (4.4 is the latest)?

You mentioned that it is a symlinked directory. Would it be possible to mount the original folder instead?
We have also checked it and it seems that on rke2 that path is no longer a symlink, so mounting it should work on rke2.

[PlayMTL]

Hey @OverOrion,

thanks for your reply.

• Your Logging operator version? I think i tested it on 4.3.0 or 4.2.2
• Can you upgrade to a recent one (4.4 is the latest)? I will do this, but need some time therefor ;)

You mentioned that it is a symlinked directory. Would it be possible to mount the original folder instead?
I think this will not work, because RKE maps an Logfile from an "unkown" container (for example kubelet) to this file. I have no chance to determine which log file from the docker directory i need to mount.

$ ll /var/lib/rancher/rke/log
kubelet_XXXXXXX.log -> /var/lib/docker/containers/XXXXXXX/XXXXXXX-json.log

We have also checked it and it seems that on rke2 that path is no longer a symlink, so mounting it should work on rke2.
--> That should be right. I think its a specific rke(1) case.

[pepov]

is RKE1 something that will stick, or do you expect to upgrade from it one day?

anyone else having the same issue? just would like to understand how much benefit we would have if we fix this on the long run...

[aslafy-z]

I have the same issue with some production clusters. There is no upgrade path given by the vendor and no plans for EOL it from now on.

[pepov]

@PlayMTL you can also reach out over our project discord in case you want to have a chat: https://discord.gg/sXFdDKEKK8