Log receivers should be from source namespace only

[edwardwu]

Describe the bug
When choose the source from e.g. default namespace,
the filelog receciver should be use /var/log/pods/default_*/*/*.log instead of /var/log/pods/*/*/*.log
receivers:
filelog:
exclude:
- /var/log/pods/kube-system_*/*/*.log
include:
- /var/log/pods/default_*/*/*.log
include_file_name: false
include_file_path: true

To Reproduce
Steps to reproduce the behavior:

1. install v1.0.17
2. select source from all pods from default namespace
3. select log destination to use loki
4. I will see the logs from all namespaces

Expected behavior
the logs should be from default namespace only

Desktop (please complete the following information):

• OS: Linux
• Version: Ubuntu 22.04

[edwardwu]

the same issue on v1.0.19 - get the logs from all pods (/var/log/pods/*/*/*.log )
manually updated
filelog receciver to use /var/log/pods/default_*/*/*.log

[edeNFed]

Hi @edwardwu
Thanks for reporting! this is indeed a known bug in the current implementation.
If you would like to open a PR to fix it, we will be happy to accept it.
As a general idea, I think the easiest way to solve this would be to write a new processor for the collector that filters the logs according to the selected namespaces. Happy to elaborate more.

[edwardwu]

Hi @edeNFed
Thanks for the suggestion. I can try to write a new processor for the collector that that filters the logs according to the selected namespaces. Could you provide additional insights or further elaboration on this matter?
I also like to have a Clickhouse gateway too.

[edeNFed]

Hi @edwardwu
Sure, you can see an example of a custom processor here:
https://github.com/keyval-dev/odigos/tree/main/collector/processors/odigosresourcenameprocessor

After creating a new Go module and writing the processor, you will also need to update this YAML file for the collector to be aware of this new processor:
https://github.com/keyval-dev/odigos/blob/0ba231fd5f7cb79e3c7a998babb5831cea56c3b5/collector/builder-config.yaml#L68

Hope that helps, please let me know if you have any more questions.

[rauno56]

@edwardwu, I'm looking into making this a better experience.

Current solution is clearly sub-optimal so the first idea is to narrow down the sources to only ones that have a specific label in k8s. That closes the flood and makes it usable.

Second step in my mind would be to make it configurable in the UI or by some other means, but happy to hear your thoughts.

[rauno56]

There's now a PR open to fix the pressing issue: #1080

After #1080 only instrumented apps are collected for logs.