Support for new IPFIX features support

[raghurampai]

Raising this ticket to request support for features new to IPFIX as compared to Netflow V9-

1. Support for variable length information element-
   ====== snip from RFC 7011 ========

7. Variable-Length Information Element

The IPFIX Template mechanism is optimized for fixed-length
Information Elements [RFC7012]. Where an Information Element has a
variable length, the following mechanism MUST be used to carry the
length information for both the IANA-assigned and enterprise-specific
Information Elements.

In the Template Set, the Information Element Field Length is recorded
as 65535. This reserved length value notifies the Collecting Process
that the length value of the Information Element will be carried in
the Information Element content itself.

In most cases, the length of the Information Element will be less
than 255 octets. The following length-encoding mechanism optimizes
the overhead of carrying the Information Element length in this more
common case. The length is carried in the octet before the
Information Element, as shown in Figure R.

  0                   1                   2                   3
  0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 | Length (< 255)|          Information Element                  |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |                      ... continuing as needed                 |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

        Figure R: Variable-Length Information Element (IE)
                       (Length < 255 Octets)

The length may also be encoded into 3 octets before the Information
Element, allowing the length of the Information Element to be greater
than or equal to 255 octets. In this case, the first octet of the
Length field MUST be 255, and the length is carried in the second and
third octets, as shown in Figure S.

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|      255      |      Length (0 to 65535)      |       IE      |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                      ... continuing as needed                 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

        Figure S: Variable-Length Information Element (IE)
                    (Length 0 to 65535 Octets)

====================================

2. Support for enterprise specific information elements-
   ==== snip from RFC 7011 =====
   3.2. Field Specifier Format

   Vendors need the ability to define proprietary Information Elements,
   because, for example, they are delivering a pre-standards product, or
   the Information Element is in some way commercially sensitive. This
   section describes the Field Specifier format for both IANA-registered
   Information Elements [IANA-IPFIX] and enterprise-specific Information
   Elements.

The Field Specifier format is shown in Figure G.

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |E|  Information Element ident. |        Field Length           |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |                      Enterprise Number                        |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                 Figure G: Field Specifier Format

Where:

E

  Enterprise bit.  This is the first bit of the Field Specifier.  If
  this bit is zero, the Information Element identifier identifies an
  Information Element in [IANA-IPFIX], and the four-octet Enterprise
  Number field MUST NOT be present.  If this bit is one, the
  Information Element identifier identifies an enterprise-specific
  Information Element, and the Enterprise Number field MUST be
  present.

Information Element identifier

  A numeric value that represents the Information Element.  Refer to
  [IANA-IPFIX].

Field Length

  The length of the corresponding encoded Information Element, in
  octets.  Refer to [IANA-IPFIX].  The Field Length may be smaller
  than that listed in [IANA-IPFIX] if the reduced-size encoding is
  used (see Section 6.2).  The value 65535 is reserved for variable-
  length Information Elements (see Section 7).

Enterprise Number

  IANA enterprise number [IANA-PEN] of the authority defining the
  Information Element identifier in this Template Record.

========================

Saw that these features are not supported currently. Please let me know if any plans to support these.

Thanks!

[lspgn]

@raghurampai it seems very generic: while the library can be improved to support it while this may not be used when producing the protobuf abstracted flows.
So far no plan of supporting them.

What's the use case? What are the fields produced? What are the devices producing it? Do you have a pcap?

[mathiaske]

Hi just jumping in
We are using dedicated Netflow Probes like Flowmon and others. These are capable of extracting additional information such as DNS requests, HTTP information etc. This is very useful in open high performance environments where firewall, proxy etc. are no option.
This data is then transmitted in IPFIX like any other network traffic information.
Using the corresponding Enterprise ID (IANA is 0) and Field IDs configurations, this allows IPFIX to be used as a widely supported, very flexible data transmission container.
Ideally the enterprise field definitions would not be hard coded, but configurable.
The CESNET IPFIXcol implements such features.

Cheers.

[lspgn]

@mathiaske Got it, thank you for the details. I believe this is beyond the scope of GoFlow which is meant to produce a unique flow format for network samples. The decoding library could definitely benefit from this though. Could you send some samples?
Actually, sFlow also supports things like DNS and HTTP, in which case, it may be interesting to have in GoFlow.

[raghurampai]

@lspgn, sorry for the delay in response. I don't have the pcap currently. The use case at high level is similar to what @mathiaske mentioned. To describe further, the monitoring of BNG (broadband subscriber services) will have some enterprise specific information elements, which basically needs to be processed-
https://www.juniper.net/documentation/en_US/junos/topics/concept/ipfix-mediation-bng.html

Thanks!

[lspgn]

@raghurampai thank you for the details!
I will definitely look into it. Do you have a spec detailing the enterprise specifics fields?

[mathiaske]

We currently use some Flowmon Probes -> CESNET, Flowmon (former Inveatec) fields
We also intend to have our own fields in the future for some use cases
-> it would be greate if this would be configurable (but please not in xml)
Example: https://github.com/CESNET/ipfixcol/blob/master/base/config/ipfix-elements.xml

[raghurampai]

Below link has the enterprise specific fields-
https://www.juniper.net/documentation/en_US/junos/topics/reference/general/logging-dictionary-template-types-tdf.html

Yes, it would be great to have option of some sort of configurable file for goflow decoder to be able to understand the enterprise specific fields. The decoder could then decode these additional fields.