Sampling rate not read from Cisco Catalyst netflow

[vincentbernat]

Discussed in #88

^{Originally posted by kwbrd August 12, 2022}
Hello!

Just got "sampling rate missing" situation.

I have couple of Cisco 9300 switches for exporting netflow v9. Also configured sampling random 1 of 100.

sampler AKVORADO_NETFLOW_SAMPLER
 mode random 1 out-of 100

flow exporter NETFLOW_TO_AKVORADO
 description Export NetFlow to Akvorado analizer
 destination %Akvorado IP%
 source Vlan1123
 transport udp 2055
 template data timeout 30
 option interface-table
 option sampler-table timeout 10
 option application-table timeout 10

interface TenGigabitEthernet1/1/1
 ip flow monitor NETFLOW_TO_ANALIZER_IN sampler AKVORADO_NETFLOW_SAMPLER input
 ip flow monitor NETFLOW_TO_ANALIZER_OUT sampler AKVORADO_NETFLOW_SAMPLER output

I dumped netflow options packet with sampling information on Akvorado host ethernet interface

Frame 32: 147 bytes on wire (1176 bits), 147 bytes captured (1176 bits)
Ethernet II, Src: fe:c7:f0:88:9a:0f (fe:c7:f0:88:9a:0f), Dst: Qisda_5e:f0:40 (00:1e:21:5e:f0:40)
Internet Protocol Version 4, Src: 10.10.23.6, Dst: 10.48.205.33
User Datagram Protocol, Src Port: 54330, Dst Port: 2055
Cisco NetFlow/IPFIX
    Version: 9
    Count: 2
    SysUptime: 1596976.492000000 seconds
    Timestamp: Aug 11, 2022 17:31:58.000000000 MSK
        CurrentSecs: 1660228318
    FlowSequence: 292
    SourceId: 2
    FlowSet 1 [id=1] (Options Template): 257
        FlowSet Id: Options Template(V9) (1)
        FlowSet Length: 30
        Options Template (Id = 257) (Scope Count = 1; Data Count = 4)
            Template Id: 257
            Option Scope Length: 4
            Option Length: 16
            Field (1/1) [Scope]: System
                Scope Type: System (1)
                Length: 4
            Field (1/4): FLOW_SAMPLER_ID
                Type: FLOW_SAMPLER_ID (48)
                Length: 4
            Field (2/4): SAMPLER_NAME
                Type: SAMPLER_NAME (84)
                Length: 40
            Field (3/4): FLOW_SAMPLER_MODE
                Type: FLOW_SAMPLER_MODE (49)
                Length: 1
            Field (4/4): FLOW_SAMPLER_RANDOM_INTERVAL
                Type: FLOW_SAMPLER_RANDOM_INTERVAL (50)
                Length: 2
    FlowSet 2 [id=257] (1 flows)
        FlowSet Id: (Data) (257)
        FlowSet Length: 55
        [Template Frame: 32]
        Flow 1
            ScopeSystem: 0a0a1706
            SamplerID: 5
            SamplerName: AKVORADO_NETFLOW_SAMPLER
            SamplerMode: Random (2)
            SamplerRandomInterval: 100

Fixed this just for now with setting up 'default-sampling-rate: 100'.

Please tell me any suggestions how to make inlet core to read it from netflow

[vincentbernat]

@kwbrd I would appreciate a pcap if you can. Looking at the code, it should work.

[kwbrd]

Sure!
You can find first sampling options packet in frame 32

flow2.zip

[vincentbernat]

Thanks! I notice that the sampling rate is for sourceID 2 and there is no flow for sourceID 2. The other sources do not have a sampling rate. Maybe the capture is too short? If you have a long capture, can you check? I am using cflow.template_field_type == 50.

[kwbrd]

flow3.zip

Here is longer capture from single switch. Netflow is only traffic there, so i don't set any filter but src host.

[vincentbernat]

Applying cflow.template_field_type == 50 as a filter, I only get sourceId: 3. Applying cflow.source_id == 3 as a filter, I only get the packets with the flow sampler information. The RFC is pretty clear that the templates are scoped by the observation domain ID (the source ID). To me, it seems a bug in the Cisco implementation. Which versions of NXOS are you using?

[kwbrd]

It is IOS-XE 16.09.04 software, sorry didn't said that earlier

[vincentbernat]

Would you be able to raise this issue to Cisco TAC? I don't have anything to test on my side.

[kwbrd]

Unfortunately, I'm not sure, but I'll try. Thanks

[kwbrd]

Well, TAC is not available for us in the near future

[vincentbernat]

I have several ASR903 running IOS-XE 16.09.03, but I suppose there are several editions or licensing options, because mine does not know a thing about Netflow (the only completion I get for flow is flow-sampler-map).

[kwbrd]

On Catalysts i have

(config)#flow ?
  exporter  Define a Flow Exporter
  monitor   Define a Flow Monitor
  record    Define a Flow Record

[vincentbernat]

The workaround is easy enough to put this issue to sleep. Let's see if someone running IOS XE has the same problem and which version they are using.

[gsalisbury]

Also having same problem here with a ISR4431/K9 on 17.03.05.

[vincentbernat]

Were you able to confirm with a pcap that there is no flow for the source ID with the sampling rate? If yes, are you able to open a ticket to TAC?

[gsalisbury]

I'll confirm with a pcap later today or tomorrow. I'll also check if we can open a TAC case.

[gsalisbury]

These are conference routers, and are only used once a year for a couple of weeks, we don't have TAC support for these.

Relevant config:

flow exporter svc-conf1
 destination 192.168.0.122
 source GigabitEthernet0/0/0.991
 transport udp 2055
 template data timeout 60
 option interface-table
 option exporter-stats
 option sampler-table
 option application-table

flow monitor mk.v4
 exporter svc-conf1
 record netflow ipv4 original-input

flow monitor mk.v6
 exporter svc-conf1
 record netflow ipv6 original-input

sampler random-sampler
 mode random 1 out-of 100

interface GigabitEthernet0/0/0.993
 description lan.mk
 encapsulation dot1Q 993
 ip flow monitor mk.v4 sampler random-sampler input
 ipv6 flow monitor mk.v6 sampler random-sampler input

Pcap: netflow.pcap.gz

[vincentbernat]

Yes, that's the same issue. When filtering with cflow.source_id == 6, we see the option templates and option data (including the sampling rate), but no data at all. It's odd for such a "bug" to be present for so long. Maybe @lspgn has already seen this?

[achurak]

Try adding collect flow sampler to your flow record, then remove the default-sampling-rate from the akvorado.yaml, restart the containers to apply the updated config and wait for the option sampler-table timeout to kick in and send the mapping between the flow's sampler ID and the sampling rate to the collector/akvorado.

This field contains the ID of the flow sampler used to monitor the flow. This is useful when more than one flow sampler is being used with different sampling rates. The flow exporter option sampler-table command exports options records with mappings of the flow sampler ID to sampling rate so the collector can calculate the scaled counters for each flow.

[dstarkovn]

I tried add collect flow sampler in record, but It did not help, counters akvorado_inlet_flow_decoder_error_count{name="netflow"} and akvorado_inlet_flow_decoder_netflow_errors_count{error="template not found"} and akvorado_inlet_core_flows_errors{error="sampling rate missing"} began to increase

Try adding collect flow sampler to your flow record, then remove the default-sampling-rate from the akvorado.yaml, restart the containers to apply the updated config and wait for the option sampler-table timeout to kick in and send the mapping between the flow's sampler ID and the sampling rate to the collector/akvorado.

This field contains the ID of the flow sampler used to monitor the flow. This is useful when more than one flow sampler is being used with different sampling rates. The flow exporter option sampler-table command exports options records with mappings of the flow sampler ID to sampling rate so the collector can calculate the scaled counters for each flow.