Enriching flow data with Model-Driven Telemetry data

[hyberdk]

Description
Im trying to setup Model Driven Telemetry in some of our Cisco IOS-XE devices.. ultimately I want to push this to Kafka (like our netflow via nfacctd)

I have this config on my 1100 router:

telemetry ietf subscription 100
 encoding encode-kvgpb
 filter xpath /device-hardware-data
 stream yang-push
 update-policy periodic 6000
 receiver ip address 10.8.8.14 1620 protocol grpc-tcp

This is my pmtelemetryd.conf

! Telemetry thread configuration
 telemetry_daemon: true
!
telemetry_daemon_port_tcp: 1620
telemetry_daemon_decoder: cisco_v1
!
! telemetry_dump_amqp_routing_key: telemetry-dump

!telemetry_dump_kafka_topic: telemetry-dump

telemetry_dump_file: /telemetry-dump-$peer_src_ip-%Y%m%d-%H%M.txt
telemetry_dump_latest_file: /telemetry-dump-$peer_src_ip.latest

!
! [Optional] librdkafka config for buffer tuning, ssl config, etc. 
! telemetry_dump_kafka_config_file: /path/to/librdkafka.conf
!
telemetry_dump_refresh_time: 60

I run it in the "latest" container like this:
docker run --name mdt -v pwd/pmtelemetryd.conf:/etc/pmacct/pmtelemetryd.conf -p 1620:1620 pmacct/pmtelemetryd

here is the version running in the container:

root@5b9d2f500c8e:/# pmtelemetryd -V       
Streaming Network Telemetry Daemon, pmtelemetryd 1.7.8-git [20221231-1 (723b0cb2)]

Arguments:
 '--enable-mysql' '--enable-pgsql' '--enable-sqlite3' '--enable-kafka' '--enable-geoipv2' '--enable-jansson' '--enable-rabbitmq' '--enable-nflog' '--enable-ndpi' '--enable-zmq' '--enable-avro' '--enable-serdes' '--enable-redis' '--enable-gnutls' 'AVRO_CFLAGS=-I/usr/local/avro/include' 'AVRO_LIBS=-L/usr/local/avro/lib -lavro' '--enable-l2' '--enable-traffic-bins' '--enable-bgp-bins' '--enable-bmp-bins' '--enable-st-bins'

Libs:
cdada 0.4.0
libpcap version 1.8.1
MariaDB 10.3.36
PostgreSQL 110018
sqlite3 3.27.2
rabbimq-c 0.11.0
rdkafka 2.0.0
jansson 2.14
MaxmindDB 1.7.1
ZeroMQ 4.3.2
Redis 1.1.1
GnuTLS 3.6.7
avro-c
serdes
nDPI 4.2.0
netfilter_log

System:
Linux 5.4.0-92-generic #103-Ubuntu SMP Fri Nov 26 16:13:00 UTC 2021 x86_64

Compiler:
gcc 8.3.0

For suggestions, critics, bugs, contact me: Paolo Lucente <paolo@pmacct.net>.
root@5b9d2f500c8e:/# 

My problem is that the data that are returning looks like this, eg. its not decoding the data:

{"event_type": "dump", "seq": 5530, "timestamp": "2023-06-20 09:59:00", "telemetry_node": "10.12.19.49", "telemetry_port": 57325, "serialization": "unknown"}
{"event_type": "dump", "seq": 5530, "timestamp": "2023-06-20 09:59:00", "telemetry_node": "10.12.19.49", "telemetry_port": 57325, "serialization": "unknown"}
{"event_type": "dump", "seq": 5530, "timestamp": "2023-06-20 09:59:00", "telemetry_node": "10.12.19.49", "telemetry_port": 57325, "serialization": "unknown"}
{"event_type": "dump", "seq": 5530, "timestamp": "2023-06-20 09:59:00", "telemetry_node": "10.12.19.49", "telemetry_port": 57325, "serialization": "unknown"}
{"event_type": "dump", "seq": 5530, "timestamp": "2023-06-20 09:59:00", "telemetry_node": "10.12.19.49", "telemetry_port": 57325, "serialization": "unknown"}
{"event_type": "dump", "seq": 5530, "timestamp": "2023-06-20 09:59:00", "telemetry_node": "10.12.19.49", "telemetry_port": 57325, "serialization": "unknown"}
{"event_type": "dump", "seq": 5530, "timestamp": "2023-06-20 09:59:00", "telemetry_node": "10.12.19.49", "telemetry_port": 57325, "serialization": "unknown"}
{"event_type": "dump", "seq": 5530, "timestamp": "2023-06-20 09:59:00", "telemetry_node": "10.12.19.49", "telemetry_port": 57325, "serialization": "unknown"}
{"event_type": "dump", "seq": 5530, "timestamp": "2023-06-20 09:59:00", "telemetry_node": "10.12.19.49", "telemetry_port": 57325, "serialization": "unknown"}
{"event_type": "dump", "seq": 5530, "timestamp": "2023-06-20 09:59:00", "telemetry_node": "10.12.19.49", "telemetry_port": 57325, "serialization": "unknown"}

I have also attached a capture of the traffic if that helps.
test2.zip

Now the question is... are IOS-XE not supported or am I missing some configuration somewhere?

thanks for your help and awesome product!

Appreciation
stared/liked many years ago ;-)

[scuzzilla]

Hi @hyberdk,

recently I wrote a short how-to describing the main steps to enable gRPC dial-out on pmtelemetryd.

To get a general idea about gRPC dial-out @pmtelemetryd you could also refer to these paragraphs.

Please, let me know if the steps outlined in the guide are clear and if the process is working for you. I would be more than happy to help you further.

Salvatore.

[hyberdk]

Hi @scuzzilla,

thanks for your reply.. I did actually notice the project yesterday after I did this question. Im a little confused to be honest.. It looks like there have been multiple implementations of streaming telemetry over the years in pmacct and mdt-dialout-collector is the latest implementation, is that a fair statement?

From what I can read in the docs, you can run mdt-dialout-collector independently or you can choose to use it as a plugin in pmtelemetryd?

Not entirely sure when to do one or the other.. We are currently using nfacct for netflow and ideally I could stream the interface names etc and enrich the netflow data with MDT data, even before we shoot it into kafka, but today we have an external system for that.

If I just want to stream mdt into kafka, is there a need to use pmtelemetryd at all? Or am I missing some big picture here?

Esben

[scuzzilla]

Hi Esben,

Streaming telemetry is a broad field with many protocols. Here, we are focusing specifically on YANG-modeled data-streams.

Within this context, "gRPC dial-out" is one of the widely used methods. While it's not standardized, it employs protobuf, leading to a more or less uniform implementation across all relevant vendors.

You are correct in your understanding that mdt-dialout-collector can be used either standalone or integrated with pmtelemetryd. The standalone use case is probably simpler and directly focuses on gathering and streaming data to Kafka. On the other hand, integrating with pmtelemetryd creates a foundation that allows for further development in the direction of near real-time data correlation.

I hope this helps clarify the matter. If you have more questions or need further elaboration, please feel free to ask.

Salvatore.

[hyberdk]

Hi Salvatore,

Again many thanks for taking your time to explain it to me.. I think I will try and do the standalone setup now for testing, but I would like to understand more about how to do this correlation directly in pmacct, but I have not really been able to find documentation that outlines the "overall" architecture/flow and what you can do.. If you can give me pointers in the right direction it would be great.

I would really like to understand my options here, before I start migrating away from snmp ;-)

Esben

[scuzzilla]

Hi Esben,

At the moment pmacct doesn't support direct correlation with gRPC dial-out. However, this could be a feature in future developments.

For any other correlation functionalities concerning, for example, BGP/BMP & IPFIX, I recommend using the pmacct Quickstart guide.

Salvatore.

[paololucente]

Hi Esben ( @hyberdk ),

After the conversation with the great Salvatore ( @scuzzilla ), would you consider fair to rename this issue as "Enriching flow data with Model-Driven Telemetry data" and labelling as enhancement / feature request?

Paolo