src_port, dst_port, protocol details

[vishnubraj]

Hi,

Is it possible to add src_port, dst_port, protocol labels also to the metrics?

[bswinnerton]

Hi @vishnubraj,

It's definitely possible, but not something that I would recommend. By adding source and destination ports, it would result in very high cardinality metrics (you'd have a unique combination of every label type). This not only would slow down queries but would significantly increase the amount of data stored in Prometheus.

If you were to do this, you would need to first export the data from pmacct, with something like this in your pmacctd.conf

aggregate: src_host, dst_host, src_port, dst_port, src_as, dst_as, label, src_port, dst_port

(other options are laid out here)

And then the following sections of the Kafka consumer would need to be updated:

flow-exporter/internal/consumer/consumer.go

Lines 17 to 33 in e8e2114

	var (
	flowReceiveBytesTotal = promauto.NewCounterVec(
	prometheus.CounterOpts{
	Name: "flow_receive_bytes_total",
	Help: "Bytes received.",
	},
	[]string{"source_as", "source_as_name", "destination_as", "destination_as_name", "hostname"},
	)
	flowTransmitBytesTotal = promauto.NewCounterVec(
	prometheus.CounterOpts{
	Name: "flow_transmit_bytes_total",
	Help: "Bytes transferred.",
	},
	[]string{"source_as", "source_as_name", "destination_as", "destination_as_name", "hostname"},
	)
	)

flow-exporter/internal/consumer/consumer.go

Lines 35 to 42 in e8e2114

	type flow struct {
	SourceAS int `json:"as_src"`
	DestinationAS int `json:"as_dst"`
	SourceIP string `json:"ip_dst"`
	DestinationIP string `json:"ip_src"`
	Bytes int `json:"bytes"`
	Hostname string `json:"label"`
	}

flow-exporter/internal/consumer/consumer.go

Lines 127 to 147 in e8e2114

	if f.SourceAS == asn {
	flowTransmitBytesTotal.With(
	prometheus.Labels{
	"source_as": strconv.Itoa(f.SourceAS),
	"source_as_name": asns[f.SourceAS],
	"destination_as": strconv.Itoa(f.DestinationAS),
	"destination_as_name": asns[f.DestinationAS],
	"hostname": f.Hostname,
	},
	).Add(float64(f.Bytes))
	} else if f.DestinationAS == asn {
	flowReceiveBytesTotal.With(
	prometheus.Labels{
	"source_as": strconv.Itoa(f.SourceAS),
	"source_as_name": asns[f.SourceAS],
	"destination_as": strconv.Itoa(f.DestinationAS),
	"destination_as_name": asns[f.DestinationAS],
	"hostname": f.Hostname,
	},
	).Add(float64(f.Bytes))
	}

[vishnubraj]

Thanks for the clear explanation @bswinnerton
I agree with the increase in the data storage part. I will check if I can use it without any performance degradation.

[tf3t]

It would also be a nice addition (I'm testing it) to add tags like iface_in, iface_out - this would open up more granularity of distinguishing what provider the traffic is coming from (e.g. peering vs transit) I'm looking into if the tags can be used in pmacct.
and also multi-as, in my use case I'm monitoring two different AS'es. I solved it by running two exporters - but I'm going to try to modify so multiple AS'es are supported. (perhaps a special case).