Logging Operator should check semantics as well

[OverOrion]

Bugs should be filed for issues encountered whilst operating logging-operator.
You should first attempt to resolve your issues through the community support
channels, e.g. Slack, in order to rule out individual configuration errors. #logging-operator
Please provide as much detail as possible.

Describe the bug:
If there is a syslog-ng configuration change which is syntactically correct, but semantically not (i.e., unknown destination/output) then the logging-operator's configcheck does not catch it.

Expected behaviour:
The configcheck should check for semantics as well during the validation.

Steps to reproduce the bug:
Follow the official quick start guide but the output should refer to something undefined.

Additional context:

Environment details:

• Kubernetes version (e.g. v1.15.2):
• Cloud-provider/provisioner (e.g. AKS, GKE, EKS, PKE etc):
• logging-operator version (e.g. 2.1.1): master
• Install method (e.g. helm or static manifests):
• Logs from the misbehaving component (and any other relevant logs):

[2023-09-05T13:56:24.988862] Setting current version as config version; version='4.3'                                                                                                                                   [2023-09-05T13:56:25.063798] Error resolving reference; content='destination', name='output_quickstart_httpUnknown', location='/etc/syslog-ng/config/syslog-ng.conf:30:5'

• Resource definition (possibly in YAML format) that caused the issue, without sensitive data:

kubectl --namespace quickstart apply -f - <<"EOF"
apiVersion: logging.banzaicloud.io/v1beta1
kind: SyslogNGFlow
metadata:
  name: log-generator
spec:
  match:
    regexp:
      value: "json#kubernetes#labels#app.kubernetes.io/instance"
      pattern: log-generator
      type: string
  localOutputRefs:
    - httpUnknown
---
apiVersion: logging.banzaicloud.io/v1beta1
kind: SyslogNGOutput
metadata:
  name: http
spec:
  http:
    url: http://logging-operator-test-receiver:8080
    headers:
      - "Content-Type: application/json"
    disk_buffer:
      dir: /buffers
      disk_buf_size: 512000000 # 512 MB
      reliable: true
EOF

/kind bug

[OverOrion]

Somewhat related issue: #1432.

[pepov]

Yes, the intent is the same!

[pepov]

We could however add validation in the operator itself so that it catches output references that do not exist in the config. The fluentd config builder does this already.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions!

[pepov]

this can only be implemented for syslog-ng currently