Wouldn't it be a good idea to introduce a syslog-ng destination for promtail that uses http (promtail push api to Loki) ?

[itheodoridis]

Hello all.
I would like to initiate a discussion about whether it would be a good idea to introduce a syslog-ng destination for promtail that uses http. I had the intention of investigating the possibility to gather Cisco network devices logs and get them over to Loki. Peter Czanik had written a blog post about storing such logs in files using json format:
https://www.syslog-ng.com/community/b/blog/posts/parsing-cisco-logs-in-syslog-ng

I used part of that configuration and after a lot of search and help all around I got to send those logs over to Loki using http by targeting the Promtail push API that can forward logs to Loki in the desired Loki format. This doesn't touch the filesystem obviously.

This what the syslog-ng config looks for this.

source s_net {
    default-network-drivers(flags(store-raw-message));
};

destination d_http {
    http(
        url("http://localhost:3500/loki/api/v1/push")
        method("POST")
        headers("Content-Type: application/json")
        user_agent("syslog-ng User Agent")
        body-suffix("\n")
        body('{ "streams": [ { "stream": { "host": "${HOST_FROM}", "source": "syslog", "severity": "${SEVERITY}", "priority":"${PRIORITY}", "facility": "${FACILITY}" }, "values": [ [ "${UNIXTIME}000000", "${MESSAGE}", "${ISODATE}" ] ] } ] }')
    );
};

log {
    source(s_net);
    destination(d_http);
};

After a suggestion from Peter, I tried with changing "${UNIXTIME}000000" to "${USEC}". Still works.

This is what the Promtail config looks for this.

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://127.0.0.1:3100/loki/api/v1/push

scrape_configs:
- job_name: push1
  loki_push_api:
    server:
      http_listen_port: 3500
      grpc_listen_port: 3600
    labels:
      pushserver: push1
      job: syslog

  pipeline_stages:
    - json:
        expressions:
          host: HOST
          priority: PRIORITY
          severity: severity
          message: MESSAGE
          timestamp: ISODATE
    - timestamp:
        format: RFC3339
        source: timestamp
    - labels:
        severity: 
        host: 
        priority:
    - output:
        source: message

I think it may be beneficial for the community to construct a syslog-ng destination for promtail that can replicate this in a cleaner way or maybe even optimize it.
I am in no way a log expert or even medium level log connoisseur.. So I am pretty sure a lot of people can do a better job with this than I did. Maybe though this can help people use syslog-ng in more ways then it is possible now.
I plan to transfer this setup over to docker containers soon (in the next couple of days). I don't expect any trouble.

[czanik]

@itheodoridis Thank you for bringing this up and the initial configuration! I plan to replicate your setup locally (using https://www.syslog-ng.com/community/b/blog/posts/sending-logs-from-syslog-ng-to-grafana-loki to refresh my memories about Loki / Grafana :-) ) and then create a configuration block out of it (using /usr/share/syslog-ng/include/scl/elasticsearch/elastic-http.conf as an example).

As a first step I'll use the current template, then I'll check how one use format-json instead of the MESSAGE macro.

[itheodoridis]

Thank you for picking this up! Can I help with providing you the rest of the config for Grafana/Loki/Promtail/Syslog-ng on docker? Or maybe just the Grafana/Loki part on docker? Anything you need just say the word.
Happy to be part of the community.

[bazsi]

Thanks a lot for the configuration, the only issue I can see is that we would need to add batching of some form.

[czanik]

Update. I have now a loki-http() configuration block with batching, largely based on elasticsearch-http(). It seems to work. There is a single catch: I have no idea about the Loki side :-) So, next up: installing Grafana to have a GUI instead of just logcli and reading a bit Loki documentation to understand both the Loki configuration and the content of the body() template better.

source s_net {
    default-network-drivers(flags(store-raw-message));
};

block destination loki-http(
  url()
  workers(4)
  batch_lines(100)
  timeout(10)
  headers("Content-Type: application/json")
  body_suffix("\n")
  body('{ "streams": [ { "stream": { "host": "${HOST_FROM}", "source": "syslog", "severity": "${SEVERITY}", "priority":"${PRIORITY}", "facility": "${FACILITY}" }, "values": [ ["${UNIXTIME}${USEC}", "${MESSAGE}", "${ISODATE}" ] ] } ] }')
  ...)
{

@requires http "The loki-http() driver depends on the syslog-ng http module, please install the syslog-ng-mod-http (Debian & derivatives) or the syslog-ng-http (RHEL & co) package"

    http(
        url(`url`)
        method("POST")
        headers(`headers`)
        workers(`workers`)
        batch_lines(`batch_lines`)
        timeout(`timeout`)
        body_suffix(`body_suffix`)
        body(`body`)
        `__VARARGS__`
    );
};

destination d_loki {
  loki-http (
    url("http://localhost:3500/loki/api/v1/push")
  );
};


log {
    source(s_sys);
    source(s_net);
    destination(d_loki);
};

[itheodoridis]

I can help on the Loki and Grafana side. Is Loki and Grafana on docker? what are your capabilities? Same VM for everything? Docker host? Let me know and we can find a way I can help probably.

[czanik]

Right now I have loki/promtail/logcli in a single directory together with their configurations:

[root@localhost loki]# ls -la
total 197768
drwxr-xr-x   2 root root      135 May  5 13:38 .
dr-xr-x---. 11 root root     4096 May  5 13:38 ..
-rwxr-xr-x   1 root root 52432896 May  3 13:15 logcli-linux-amd64
-rwxr-xr-x   1 root root 59424768 May  3 13:14 loki-linux-amd64
-rw-r--r--   1 root root     1294 May  4 17:51 loki-local-config.yaml
-rwxr-xr-x   1 root root 90640576 May  3 13:16 promtail-linux-amd64
-rw-r--r--   1 root root      700 May  4 17:44 promtail.yaml

I have yet to install Grafana, and that will be most likely using Docker.

OK. I checked the format expected by the Loki API: https://grafana.com/docs/loki/latest/api/#push-log-entries-to-loki This is something that we cannot produce using format-json. This is most likely where I gave up checking the HTTP API last time :-)

{
  "streams": [
    {
      "stream": {
        "label": "value"
      },
      "values": [
          [ "<unix epoch in nanoseconds>", "<log line>" ],
          [ "<unix epoch in nanoseconds>", "<log line>" ]
      ]
    }
  ]
}

What your original template produces is a JSON structure with a single log message. Each connection sends just one of these structures.

What my configuration does, is putting many of these single log message JSON structures on a single line. I'm not sure if it conforms with the official API definition, but it seems to work :-) Still, this seems to be the way to go, as the labels on each message can be different.

Also PRIORITY, SEVERITY and LEVEL are the same, aren't they? So, forwarding one or the other should be enough. How promtail.yaml should be changed?

The ISODATE macro within "values" does not seem to do anything according to the API definition (it only expect unix time and the log message). I removed it and still seems to work :-)

This is my current template:

  body('{ "streams": [ { "stream": { "host": "${HOST_FROM}", "source": "syslog", "severity": "${SEVERITY}", "facility": "${FACILITY}" }, "values": [ ["${UNIXTIME}${USEC}", "${MESSAGE}" ] ] } ] }')

cc: @bazsi

[itheodoridis]

This is a docker-compose.yml you can use for both Loki and Grafana (this should work just fine as long as you create your own self-signed certificates for the Grafana server. Also this is setup on an Ubuntu 22.04.2 LTS with docker version 23.0.4 and docker compose versin 2.3.3. Just put it in an directory where you also create an ssl folder and put those certs in there. In this case the reference is an absolute path so the folder is at /opt/loki/ssl ):

version: '3.8'

networks:
  monitor-net:
    driver: bridge

volumes:
    grafana_data: {}
    loki_data: {}

services:
  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    volumes:
      - grafana_data:/var/lib/grafana
      - /opt/loki/ssl:/etc/grafana/ssl
    #  - ./grafana/etc/grafana.ini:/etc/grafana/grafana.ini
    #  - ./grafana/etc/ldap.toml:/etc/grafana/ldap.toml
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro

    environment:
      - GF_SERVER_ROOT_URL=https://grafanaserverhostname:3000/
      - GF_SERVER_PROTOCOL=https
      - GF_SERVER_DOMAIN=domain.com
      - GF_SECURITY_ADMIN_USER=${ADMIN_USER:-admin}
      - GF_SECURITY_ADMIN_PASSWORD=${ADMIN_PASSWORD:-admin}
      - GF_USERS_ALLOW_SIGN_UP=false
      - GF_INSTALL_PLUGINS=magnesium-wordcloud-panel
      - GF_SERVER_CERT_FILE=/etc/grafana/ssl/grafanaserverhostname.crt
      - GF_SERVER_CERT_KEY=/etc/grafana/ssl/grafanaserverhostname.pem
      - GF_ALERTING_ENABLED=true
      - GF_ALERTING_EXECUTE_ALERTS=true
      - GF_AUTH.LDAP_ENABLED=true
      - GF_AUTH.LDAP_CONFIG_FILE=/etc/grafana/ldap.toml
      - GF_AUTH.LDAP_ALLOW_SIGN_UP=true
      - GF_FEATURE_TOGGLES_ENABLE=ngalert
      - GF_PLUGINS_PLUGIN_ADMIN_ENABLED=true

    restart: unless-stopped
    expose:
      - 3000
    ports:
      - 3000:3000
    networks:
      - monitor-net

  loki:
    image: grafana/loki:2.8.2
    container_name: loki
    restart: unless-stopped
    volumes:
      - loki_data:/loki
    #  - ./loki/config-loki.yml:/etc/loki/local-config.yml
    expose:
      - 3100
    ports:
      - 3100:3100
    networks:
      - monitor-net

I have commented out the bind volume entries where I define the config files externally so I can edit them in the docker host before launching the server. If I were you I would comment out the LDAP related env variable entries as well (we are using active directory). If you do want to define them then the ldap.toml file needs to be modified and the same goes if you want to allow Grafana to send emails (so you would need to set things up in the grafana.ini file.
PRIORITY and SEVERITY are different in your original config where you do the json export. Here they have the same value. The LEVEL value is not available in this case (it was in your config with local storage), it would have been an integer. I don't have an opinion on which to keep PRIORITY or SEVERITY but I might prefer to keep priority in case you want to have common dashboards and dashboard variables with other kinds of hosts than Cisco Devices.
The Promtail config is easy to adjust, either remove the severity lines or the priority lines from the expressions and labels. For example (putting up just the scrape configs key):

scrape_configs:
- job_name: push1
  loki_push_api:
    server:
      http_listen_port: 3500
      grpc_listen_port: 3600
    labels:
      pushserver: push1
      job: syslog

  pipeline_stages:
    - json:
        expressions:
          host: HOST
          severity: severity
          message: MESSAGE
          timestamp: ISODATE
    - timestamp:
        format: RFC3339
        source: timestamp
    - labels:
        severity: 
        host: 
    - output:
        source: message

Of course this affects the labels detected by Loki and as such, the variables you need to declare in the dashboards in Grafana, so you would need to adjust those there.
I noticed you kept "${UNIXTIME} ${USEC}" there. Is that correct? I thought I had to replace one with the other, I probably understood wrong.
I had a feeling the "ISODATE" field did nothing in the value list. I wasn't aware the value list was about log lines , I though you could just forward a list of values there (as in contrast with key/value pairs). Are you sure though that this doesn't make it to promtail? If yes, then maybe that line also needs to be removed from expressions.
It's possible that the "source": "syslog", part can also be omitted, as the promtail config marks all logs with label job: syslog.
Thanks for the help.
Do you still feel a promtail destination is relevant?
Let me know if you need any help with Grafana.
I would try the explore mode first (there is a label browser).
If you get comfortable you can declare data sources, create a dashboard, define variables and use them to filter out logs in the dashboards, or even go as far as to define alerts. But one step at a time.

[itheodoridis]

You know I was thinking about two things:

• If you want this to support a general way to get logs into Loki using http instead of syslog (always through promtail) you may want to keep priority instead of severity or allow for both. Severity seems to be specific for Cisco and possibly more network vendors (I really have no idea about other vendors). I would try to allow for both and let the one who is actually getting the content into Loki choose which labels to use.
• if one needs to get parts of the inner structure of the cisco log message like the MNEMONIC key, than perhaps a way to process the logs before the stage of the destination needs to be created. I would think of an alternative parser to fill that role. Perhaps if I read enough of the spec and Peter's blog series I can make something in Python, although that would probably hinder performance a bit. As log as it's an optional component bringing more labels in the first section of the json message I would see no problem with that either. So a way to integrate a series of labels in the destination regardless of the actual names of each label would probably go a long way for this case and would probably cover more use cases in the long run in a generic way, specifically for loki (through the promtail push api).

[czanik]

Just a heads up: I did not forget about this. I have Grafana up and running in docker. I started to read the Loki documentation, but had to stop, as my temperature is going up sharply right now.

Yes, I still find a loki-http() destination a good idea. However, it seems that this is not so generic, as simply sending RFC 5424 logs to promtail. One needs both a syslog-ng configuration and a matching promtail configuration.

The current promtail.yaml and my config are good starting points. Two things need to be documented in a few sentences (most likely in /usr/share/syslog-ng/include/scl/loki-http/plugin.conf):

• what is included in promtail.yaml
• how to extend it with additional labels (name-value pairs) by editing both promtail.yaml and the body() template

[itheodoridis]

I am in no way in a hurry for this, so please take good care of yourself before anything else. I am eager to help through, so anytime you think there is something I can do, let me know.
A matching config for the push API at least, is definitely needed on the Promtail side. I guess more can be defined if one wants to do any kind of processing (I believe that's also necessary to define the labels).
I will try and come up with a text and run it by you to see if it's any help.

[bazsi]

Just wanted to add that I intend to work on this once #3966 is merged. In that PR I have preparations for partitioning the message stream based on the value of specific name value pairs and with that we could start using message templates in the body-header() and body-footer() options, thereby implementing an improved batching that would let us populate the header in an improved form.

[itheodoridis]

This sounds very interesting! I am eager to see the impact it will have in detecting the fields in this use case.