Add netbox services discovery

[7840vz]

https://demo.netbox.dev/static/docs/core-functionality/services/

I think one of the great use cases could be if netbox services could be discovered as well.

Critical services can be defined in the netbox, to be checked by blackbox exporter later on.

[jeffbaier]

This would be very useful to us as well.

[FlxPeters]

We already have service support included. I use it to configure node exporter for example and filter hosts on this.

Anything specific you have demand on?

[jeffbaier]

The only service support I see is the __meta_netbox_services label on Virtual Machines and Devices. That is just the text string of the Service name, I don't see how that is useful. Can you explain your setup and how it works? For example how would you do a http_2xx probe with the blackbox_exporter to an https Netbox Service?

I have created a fork with a /api/plugins/prometheus-sd/services/ endpoint that gives us what we need. Maybe we don't need that if you can explain to me how to use the existing service support.

[FlxPeters]

Use keep and drop rules to filter relevant targets by service.

You can find an example here: #97 (comment)

But may be I should add some more examples. Feel free to make am merge request of your solution with an example prometheus config, so we can discuss if it's also an option. I'm open to multiple solutions as long as it works and tests are available.

[jeffbaier]

So if I understand your example, for it to work you have to make the Virtual Machine name a DNS or IP address because the prometheus config uses that as the __address__? And the node exporter port is hardcoded?

I added a /services endpoint in my fork because I need the extra info (primary IP, service IP, ports, custom fields, etc). I'll make another comment soon and show examples of my API and prometheus config using a real world example.

[jeffbaier]

This config has two jobs. The first using blackbox to check a URL for a 200 OK response. There is no IP defined for the service because its not applicable. I have added a custom field for the Heath Check URL for blackbox to probe. For some web services we can hit the root URL, others have a specific endpoint we need to access.

The second job is node-exporter. In this case the Virtual Machine has a primary IP of 111.222.333.5, but node-exporter is only listening on the private IP 111.222.333.135 and the non-standard port 9111. The prometheus config isn't hardcoded to address or port and uses the appropriate labels from Netbox.

---

prometheus.yaml

global:
  scrape_interval: 15s

scrape_configs:
- job_name: blackbox-http
  metrics_path: /probe
  params:
    module: [http_2xx]
  http_sd_configs:
      - url: https://XXXXX/api/plugins/prometheus-sd/services/?tag=scrape-http
        refresh_interval: 15s
        authorization:
          type: "Token"
          credentials: "abcdef123456"
  relabel_configs:
    - regex: __meta_netbox_(.+)
      replacement: netbox_$1
      action: labelmap
    - source_labels: [__meta_netbox_custom_field_health_url]
      target_label: __param_target
    - source_labels: [__meta_netbox_name]
      target_label: instance
    - target_label: __address__
      replacement: blackbox-address:9115

- job_name: node_exporter
  http_sd_configs:
      - url: https://XXXXX/api/plugins/prometheus-sd/services/?tag=scrape-node
        refresh_interval: 15s
        authorization:
          type: "Token"
          credentials: "abcdef123456"
  relabel_configs:
    - regex: __meta_netbox_(.+)
      replacement: netbox_$1
      action: labelmap
    - source_labels: [__meta_netbox_ipaddresses, __meta_netbox_ports]
      separator: ":"
      target_label: "__address__"
      action: "replace"

API output of the http_sd URLs used in the above config:

GET /api/plugins/prometheus-sd/services/?tag=scrape-http

HTTP 200 OK
Allow: GET, POST, HEAD, OPTIONS
Content-Type: application/json
Vary: Accept

[
    {
        "targets": [
            "blog"
        ],
        "labels": {
            "__meta_netbox_id": "35",
            "__meta_netbox_name": "blog",
            "__meta_netbox_display": "blog (TCP/80, 443)",
            "__meta_netbox_ports": "80,443",
            "__meta_netbox_tags": "Scrape HTTP",
            "__meta_netbox_tag_slugs": "scrape-http",
            "__meta_netbox_parent": "PHP-MYSQL-2",
            "__meta_netbox_primary_ip": "111.222.333.5",
            "__meta_netbox_primary_ip4": "111.222.333.5",
            "__meta_netbox_cluster": "2B",
            "__meta_netbox_cluster_type": "AvailabilityZone",
            "__meta_netbox_site": "US-WEST-2",
            "__meta_netbox_site_slug": "us-west-2",
            "__meta_netbox_custom_field_health_url": "https://blog.xxx.zzz/"
        }
    },
    {
        "targets": [
            "contact"
        ],
        "labels": {
            "__meta_netbox_id": "36",
            "__meta_netbox_name": "contact",
            "__meta_netbox_display": "contact (TCP/80, 443)",
            "__meta_netbox_ports": "80,443",
            "__meta_netbox_tags": "Scrape HTTP",
            "__meta_netbox_tag_slugs": "scrape-http",
            "__meta_netbox_parent": "PHP-MYSQL-2",
            "__meta_netbox_primary_ip": "111.222.333.5",
            "__meta_netbox_primary_ip4": "111.222.333.5",
            "__meta_netbox_cluster": "2B",
            "__meta_netbox_cluster_type": "AvailabilityZone",
            "__meta_netbox_site": "US-WEST-2",
            "__meta_netbox_site_slug": "us-west-2",
            "__meta_netbox_custom_field_health_url": "https://loremipsum.xxxx.zzz/health"
        }
    }
]

GET /api/plugins/prometheus-sd/services/?tag=scrape-node

HTTP 200 OK
Allow: GET, POST, HEAD, OPTIONS
Content-Type: application/json
Vary: Accept

[
    {
        "targets": [
            "node-exporter"
        ],
        "labels": {
            "__meta_netbox_id": "37",
            "__meta_netbox_name": "node-exporter",
            "__meta_netbox_display": "node-exporter (TCP/9111)",
            "__meta_netbox_ipaddresses": "111.222.333.135",
            "__meta_netbox_ports": "9111",
            "__meta_netbox_tags": "Scrape Node",
            "__meta_netbox_tag_slugs": "scrape-node",
            "__meta_netbox_parent": "PHP-MYSQL-2",
            "__meta_netbox_primary_ip": "111.222.333.5",
            "__meta_netbox_primary_ip4": "111.222.333.5",
            "__meta_netbox_cluster": "2B",
            "__meta_netbox_cluster_type": "AvailabilityZone",
            "__meta_netbox_site": "US-WEST-2",
            "__meta_netbox_site_slug": "us-west-2",
            "__meta_netbox_custom_field_health_url": "None"
        }
    }
]

[FlxPeters]

Sounds good to me. More powerful than my current solution. Looking forward to the PR!

[pdenessen]

Hi, interested as well and would love to see this services endpoint as well.

[jeffbaier]

I have created PR #122 for this issue. I have been running this code for the last week in my production Netbox.

[k0ste]

@jeffbaier is possible to add filter for parent devices? Your PR looks good for us, but useless, because we can't filter services by device status or device_cf 🫠

We filter devices like this

/api/plugins/prometheus-sd/devices/?status=active&has_primary_ip=true&cf_prometheus_server=<prometheus server name, that request for targets>

With current /services implementation is impossible to put device to maintenance ('maintenance' or 'offline' device status, i.e. not 'active'), because we get alerts for services. It will be nice if plugin support something like:

/api/plugins/prometheus-sd/services/?{parent|device}_status=active&{parent|device}_has_primary_ip=true&{parent|device}_cf_prometheus_server=<prometheus server name, that request for targets>

Thanks