Breaking change in version 1.16 related to annotation_match

[amanmahajan26]

Describe the bug

We have fluentd configured as daemonset running on our k8s cluster. In the configuration, we have an annotation: logs.westwing.eu/application based on which fluentd filters out the logs.

Up until version 1.15, in order to extract the value from the annotation, we had to replace . with _ for some reason which I don't remember now, like this:

<record>
     application ${record.dig("kubernetes", "annotations", "logs_westwing_eu/application")
 </record>

But in version 1.16, this breaks and logs don't get ingested and works only when we skip replacing . with _, like this:

<record>
     application ${record.dig("kubernetes", "annotations", "logs.westwing.eu/application")
 </record>

I could not find anything in the change logs related to this, hence creating this bug. Would be good to know what's causing this unpredictable behaviour.

To Reproduce

pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: counter
  labels:
    fluentd/loggable: "true"
  annotations:
    logs.westwing.eu/application: counter
spec:
  containers:
  - name: counter
    image: busybox
    args: [/bin/sh, -c,
            'i=0; while true; do echo "$i: $(date)"; i=$((i+1)); sleep 1; done']

Expected behavior

Logs should be ingested since the annotation logs.westwing.eu/application: counter is present. Works with version 1.15 but not from 1.16

Your Environment

- Fluentd version: 1.16
- TD Agent version:
- Operating system: 
- Kernel version:

Your Configuration

<source>
  @type tail
  read_from_head true
  tag kubernetes.*
  path /var/log/containers/*.log
  pos_file /var/log/fluentd-containers.log.pos
  exclude_path ["/var/log/containers/fluent*"]
  <parse>
    @type kubernetes
    @type "json"
    time_format %Y-%m-%dT%H:%M:%S.%NZ
  </parse>
</source>

<filter kubernetes.**>
  @type kubernetes_metadata
  @id filter_kube_metadata
  skip_labels 'false'
  skip_container_metadata "false"
  annotation_match ["logs.westwing.eu\/.*"]
</filter>

<filter kubernetes.**>
  @type record_transformer
  enable_ruby
  <record>
    application ${record.dig("kubernetes", "annotations", "logs_westwing_eu/application")
    fluentd/loggable ${record.dig("kubernetes", "labels", "fluentd/loggable")}
  </record>
  remove_keys docker, $.kubernetes.labels
</filter>

<filter kubernetes.**>
  @type grep
    <regexp>
      key fluentd/loggable
      pattern /true/
    </regexp>
    <exclude>
      key application
      pattern /^$/
    </exclude>
</filter>

<match kubernetes.**>
  @type file
  path /tmp/file-test.log
</match>

Your Error Log

No error in the logs.

Additional context

No response

[ashie]

Are you using fluentd-kurbernetes-daemonset?
If so, probably it's same issue with fluent/fluentd-kubernetes-daemonset#1442
We updated fluent-plugin-kubernetes_metadata_filter from 2.13.0 to 3.2.0.
This version drops dedot feature.

[amanmahajan26]

Yes, using fluentd-kurbernetes-daemonset. Thanks for the update.

[ashie]

Thanks.
We should tackle on it at fluent/fluentd-kubernetes-daemonset#1442