log-vm: To centralize logs using grafana promtail & loki

[vunnyso]

Description of changes

Checklist for things done

• Summary of the proposed changes in the PR description
• More detailed description in the commit message(s)
• Commits are squashed into relevant entities - avoid a lot of minimal dev time commits in the PR
• Contribution guidelines followed
• Ghaf documentation updated with the commit - https://tiiuae.github.io/ghaf/
• PR linked to architecture documentation and requirement(s) (ticket id)
• Test procedure described (or includes tests). Select one or more:
  • Tested on Lenovo X1 x86_64
  • Tested on Jetson Orin NX or AGX aarch64
  • Tested on Polarfire riscv64
• Author has run nix flake check --accept-flake-config and it passes
• All automatic Github Action checks pass - see actions
• Author has added reviewers and removed PR draft status

Testing

To view logs in locally on ghaf target machine

ssh into log-vm and use logcli to view systemd journal logs

[ghaf@ghaf-host:~]$ ssh 192.168.101.66

[ghaf@log-vm:~]$ logcli query '{job="systemd-journal"}'

[ghaf@log-vm:~]$ logcli query '{hostname="ghaf-host"}'

To view logs in grafana GUI on remote server/ dev machine.

1. Move to dev machine / machine where you want to install grafana
   To install grafana on NixOS you can follow mentioned guide here

Example:

services.grafana = {
  enable   = true;
  settings = {
    server = {
      http_addr = "<your_local_network_grafana_instance_ip_address_here>";
      domain = "localdomain";
      http_port = 3000;
      protocol = "http";
    };
  };

  dataDir  = "/var/lib/grafana";
};

2. Make sure you have connected to WIFI using nm-launcher on ghaf target machine.

3. Run the below command to add loki data source, replace with netvm-wifi-ip-address with yours IP.
   [ghaf@log-vm:~]$ curl 'http://admin:admin@<grafana_ip_in_your_local_network>:3000/api/datasources'
   -X POST -H 'Content-Type: application/json;charset=UTF-8' --data-binary '{"name":"loki","type":"loki","url":"http://<net-vm_wifi_ip_address>:3100","access":"proxy","isDefault":true,"database":"tsdb"}'

4. Go to browser where you want to view the logs and type following address http://<grafana_ip_in_your_local_network>:3000/explore
   Follow below steps in order to view logs/download logs.

Also there is way to filter out logs based on hostName as in example below:

Next Steps

1. Build Grafana alloy nix package
2. Change from promtail agent to alloy

[vilvo]

Better link testing instructions for grafana installation to https://nixos.wiki/wiki/Grafana instead of https://community.grafana.com/t/installing-on-nixos/6712
The latter link is from 2018 and many nixos grafana settings have been changed - for example

trace: warning: The option `services.grafana.domain' defined in `/nix/store/mzgmjayar7pm38x46yxhf27q4jv11zmp-source/hosts/nixos/carrie/configuration.nix' has been renamed to `services.grafana.settings.server.domain'.
trace: warning: The option `services.grafana.port' defined in `/nix/store/mzgmjayar7pm38x46yxhf27q4jv11zmp-source/hosts/nixos/carrie/configuration.nix' has been renamed to `services.grafana.settings.server.http_port'.
trace: warning: The option `services.grafana.protocol' defined in `/nix/store/mzgmjayar7pm38x46yxhf27q4jv11zmp-source/hosts/nixos/carrie/configuration.nix' has been renamed to `services.grafana.settings.server.protocol'.

I got grafana running with:

networking.firewall.allowedTCPPorts = [ 3000 ];
services.grafana = {
  enable   = true;
  settings = {
    server = {
      http_addr = "<your_local_network_grafana_instance_ip_address_here>";
      domain = "localdomain";
      http_port = 3000;
      protocol = "http";
    };
  };

  dataDir  = "/var/lib/grafana";
};

[vunnyso]

Better link testing instructions for grafana installation to https://nixos.wiki/wiki/Grafana instead of https://community.grafana.com/t/installing-on-nixos/6712 The latter link is from 2018 and many nixos grafana settings have been changed - for example

trace: warning: The option `services.grafana.domain' defined in `/nix/store/mzgmjayar7pm38x46yxhf27q4jv11zmp-source/hosts/nixos/carrie/configuration.nix' has been renamed to `services.grafana.settings.server.domain'.
trace: warning: The option `services.grafana.port' defined in `/nix/store/mzgmjayar7pm38x46yxhf27q4jv11zmp-source/hosts/nixos/carrie/configuration.nix' has been renamed to `services.grafana.settings.server.http_port'.
trace: warning: The option `services.grafana.protocol' defined in `/nix/store/mzgmjayar7pm38x46yxhf27q4jv11zmp-source/hosts/nixos/carrie/configuration.nix' has been renamed to `services.grafana.settings.server.protocol'.

I got grafana running with:

networking.firewall.allowedTCPPorts = [ 3000 ];
services.grafana = {
  enable   = true;
  settings = {
    server = {
      http_addr = "<your_local_network_grafana_instance_ip_address_here>";
      domain = "localdomain";
      http_port = 3000;
      protocol = "http";
    };
  };

  dataDir  = "/var/lib/grafana";
};

Thanks @vilvo for details, I have updated Testing section accordingly.

[vilvo]

Also, please change step 3. in Testing section instructions to:

[ghaf@log-vm:~]$ curl 'http://admin:admin@<grafana_ip_in_your_local_network>:3000/api/datasources'
-X POST -H 'Content-Type: application/json;charset=UTF-8' --data-binary '{"name":"loki","type":"loki","url":"http://<net-vm_wifi_ip_address>:3100","access":"proxy","isDefault":true,"database":"tsdb"}'

As of now, it's unclear where the tester is expected to run the command. Also, it's better use grafana instance ip address instead of localhost. Tested it to work that way. Also, the ghaf logs filtering based on the ghaf hostname works fine 👍🏻