label semantics with parsers changed in v3.0.0

[jameshartig]

Describe the bug
We have logs like so:

~ DEBUG -- pinging client -- addr="172.28.128.1:51438" user="1"

In Loki v2.9 and earlier regexp "^~ (?P<level>[^\\s]+)\\s+--\\s+(?P<message>.*?)(?:\\s+--\\s+(?P<kv>.*))?$" would replace level with DEBUG but now in v3.0.0 it does not and the level is info. If I change ?P<level> to ?P<level2> then it does pull out level2 so I'm inclined to believe the regex is working.

We were not using structured metadata prior to switching to v3.0.0 so I'm not sure if structured metadata broke it or v3.0.0 broke it.

To Reproduce
Steps to reproduce the behavior:

1. {...} | line_format "INFO" | label_format foo="bar" | regexp "^(?P<foo>[^\\s]+)$"
2. Observe "foo: barand notfoo: INFO`

Expected behavior
I expected regexp to continue to replace labels like it did in versions prior to v3.0.0 that are in the regex. I don't see any mention of this in the CHANGELOG.

Environment:

• Infrastructure: GCP VMs
• Deployment tool: Internal

Screenshots, Promtail config, or terminal output
N/A

[jameshartig]

v3.0.0

curl 'http://127.0.0.1:3100/loki/api/v1/query_range?limit=1&query=%7Bhostname%3D%22gecko%22%7D%20%7C~%20%22pinging%22%20%7C%20regexp%20%22%5E~%20(%3FP%3Clevel%3E%5B%5E%5C%5Cs%5D%2B)%5C%5Cs%2B--%5C%5Cs%2B(%3FP%3Cmessage%3E.*%3F)(%3F%3A%5C%5Cs%2B--%5C%5Cs%2B(%3FP%3Ckv%3E.*))%3F%24%22' | jq .data.result

[
  {
    "stream": {
      "datacenter": "gce-us-east1",
      "environment": "dev",
      "hostname": "gecko",
      "job": "systemd-journal",
      "kv": "addr=\"172.28.128.1:51438\" user=\"1\"",
      "level": "info",
      "message": "pinging client",
      "service_name": "llama-server",
      "unit": "llama-server.service"
    },
    "values": [
      [
        "1714495115039096000",
        "~ DEBUG -- pinging client -- addr=\"172.28.128.1:51438\" user=\"1\""
      ]
    ]
  }
]

v3.0.0 with X-Loki-Response-Encoding-Flags:categorize-labels

curl 'http://127.0.0.1:3100/loki/api/v1/query_range?limit=1&query=%7Bhostname%3D%22gecko%22%7D%20%7C~%20%22pinging%22%20%7C%20regexp%20%22%5E~%20(%3FP%3Clevel%3E%5B%5E%5C%5Cs%5D%2B)%5C%5Cs%2B--%5C%5Cs%2B(%3FP%3Cmessage%3E.*%3F)(%3F%3A%5C%5Cs%2B--%5C%5Cs%2B(%3FP%3Ckv%3E.*))%3F%24%22'  -H 'X-Loki-Response-Encoding-Flags:categorize-labels' | jq .data.result
             
[
  {
    "stream": {
      "datacenter": "gce-us-east1",
      "environment": "dev",
      "hostname": "gecko",
      "job": "systemd-journal",
      "service_name": "llama-server",
      "unit": "llama-server.service"
    },
    "values": [
      [
        "1714495165039172000",
        "~ DEBUG -- pinging client -- addr=\"172.28.128.1:51438\" user=\"1\"",
        {
          "structuredMetadata": {
            "level": "info"
          },
          "parsed": {
            "kv": "addr=\"172.28.128.1:51438\" user=\"1\"",
            "message": "pinging client"
          }
        }
      ]
    ]
  }
]

v2.9.0

curl 'http://127.0.0.1:3100/loki/api/v1/query_range?limit=1&query=%7Bhostname%3D%22gecko%22%7D%20%7C~%20%22pinging%22%20%7C%20regexp%20%22%5E~%20(%3FP%3Clevel%3E%5B%5E%5C%5Cs%5D%2B)%5C%5Cs%2B--%5C%5Cs%2B(%3FP%3Cmessage%3E.*%3F)(%3F%3A%5C%5Cs%2B--%5C%5Cs%2B(%3FP%3Ckv%3E.*))%3F%24%22' | jq .data.result

[
  {
    "stream": {
      "datacenter": "gce-us-east1",
      "environment": "dev",
      "hostname": "gecko",
      "job": "systemd-journal",
      "kv": "addr=\"172.28.128.1:51438\" user=\"1\"",
      "level": "DEBUG",
      "message": "pinging client",
      "service_name": "llama-server",
      "unit": "llama-server.service"
    },
    "values": [
      [
        "1714495375889219000",
        "~ DEBUG -- pinging client -- addr=\"172.28.128.1:51438\" user=\"1\""
      ]
    ]
  }
]

[jameshartig]

I believe this was broken in #11587.

[jameshartig]

I also believe this is related to metric queries not using the resulting label values either:

sum by (level) (count_over_time({hostname=~".+"} | label_format level="FATAL" [1m]))

The above query in v3.0.0 doesn't have any aggregations including FATAL but in v2.9.0 the only aggregation is {level="FATAL"}.