The plugin converts IPFIX flow records into UniRec format and sends them as UniRec records via a TRAP output interface. Structure of the output record is defined using a UniRec template that consists of one or more UniRec fields.
Each flow record is converted individually. For each IPFIX field the plugin tries to find mapping to a particular UniRec field of the output record and performs data conversion. It is possible that the original IPFIX record doesn't contain all required fields. In this case, a user could choose to drop the whole record or fill undefined UniRec fields with default values. Converted records are sent over a unix socket/tcp/tcp-tls for further processing or stored as files.
An instance of the plugin can use only one TRAP output interface and one UniRec template at the same time. However, it is possible to create multiple instances of this plugin with different configuration. For example, one instance can convert all flows using a UniRec template that consists of common fields available in all flow records and another instance converts only flow records that consist of HTTP fields.
By default, the plugin is not distributed with IPFIXcol due to extra dependencies. To build the plugin, IPFIXcol (and its header files) and the following dependencies must be installed on your system:
- Nemea Framework (or Nemea project)
Finally, compile and install the plugin:
$ mkdir build && cd build && cmake ..
$ make
# make install
The following output instance configuration describes all supported types of TRAP output interfaces, however, only one can be used. Therefore, remove all interfaces in <trapIfcSpec>
except the one you would like to use.
<output>
<name>UniRec plugin</name>
<plugin>unirec</plugin>
<params>
<uniRecFormat>TIME_FIRST,TIME_LAST,SRC_IP,DST_IP,PROTOCOL,?SRC_PORT,?DST_PORT,?TCP_FLAGS,PACKETS,BYTES</uniRecFormat>
<splitBiflow>true</splitBiflow>
<trapIfcCommon>
<timeout>NO_WAIT</timeout>
<buffer>true</buffer>
<autoflush>500000</autoflush>
</trapIfcCommon>
<trapIfcSpec>
<!-- Only ONE of the following output interface! -->
<tcp>
<port>8000</port>
<maxClients>64</maxClients>
</tcp>
<tcp-tls>
<port>8000</port>
<maxClients>64</maxClients>
<keyFile>...</keyFile>
<certFile>...</certFile>
<caFile>...</caFile>
</tcp-tls>
<unix>
<name>ipfixcol-output</name>
<maxClients>64</maxClients>
</unix>
<file>
<name>/tmp/nemea/trapdata</name>
<mode>write</mode>
</file>
</trapIfcSpec>
</params>
</output>
uniRecFormat
Comma separated list of UniRec fields. All fields are mandatory be default, therefore, if a flow record to convert doesn't contain all mandatory fields, it is dropped. However, UniRec fields that start with '?' are optional and if they are not present in the record (e.g. TCP_FLAGS) default value (typically zero) is used. List of all supported UniRec fields is defined in unirec-element.txt file. Example value: "DST_IP,SRC_IP,BYTES,DST_PORT,?TCP_FLAGS,SRC_PORT,PROTOCOL".
splitBiflow
In case of Biflow records, split the record to two unidirectional flow records. Non-biflow records are unaffected. [values: true/false, default: true]
trapIfcCommon
The following parameters can be used with any type of a TRAP interface. There are parameters of the interface that are normally let default. However, it is possible to override them by user:
timeout
Time in microseconds that the output interface can block waiting for message to be send. There are also special values: "WAIT" (block indefinitely), "NO_WAIT" (don't block), "HALF_WAIT" (block only if some client is connected). Be very careful, inappropriate configuration can significantly slowdown the collector and lead to loss of data. [default: "HALF_WAIT"]
buffer
Enable buffering of data and sending in larger bulks (increases throughput) [default: true]
autoflush
Automatically flush data even if the output buffer is not full every X microseconds. If the automatic flush is disabled (value 0), data are not send until the buffer is full. [default: 500000]
trapIfcSpec
Specification of interface type and its parameters. For more details, see section "Output interface types".
mappingFile
Path to configuration file with mapping IPFIX fields to UniRec fields. If the parameter is not defined, the default configuration file is used. See section "UniRec configuration file".
Exactly one of the following output type must be defined in the instance configuration of this plugin.
unix
Communicates through a UNIX socket. The output interface creates a socket and listens, input interface connects to it. There may be more than one input interfaces connected to the output interface, every input interface will get the same data. Parameters:
name
Socket name i.e. any string usable as a file name. The name MUST not include colon character.
maxClients
Maximal number of connected clients (input interfaces). [default: 64]
tcp
Communicates through a TCP socket. The output interface listens on a given port, input interface connects to it. There may be more than one input interfaces connected to the output interface, every input interface will get the same data. Parameters:
port
Local port number
maxClients
Maximal number of connected clients (input interfaces). [default: 64]
tcp-tls
Communicates through a TCP socket after establishing encrypted connection. You have to provide a certificate, a private key and a CA chain file with trusted CAs. Otherwise, same as TCP: The output interface listens on a given port, input interface connects to it. There may be more than one input interfaces connected to the output interface, every input interface will get the same data. Paths to files MUST not include colon character. Parameters:
port
Local port number
maxClients
Maximal number of connected clients (input interfaces). [default: 64]
keyFile
Path to a file of a private key in PEM format.
certFile
Path to a file of certificate chain in PEM format.
caFile
Path to a file of trusted CA certificates in PEM format.
file
Stores UniRec records into a file. The interface allows to split data into multiple files after a specified time or a size of the file. If both options are enabled at the same time, the data are split primarily by time, and only if a file of one time interval exceeds the size limit, it is further split. The index of size-split file is appended after the time. Parameters:
name
Name of the output file. The name MUST not include colon character.
mode
Output mode:
write
/append
. If the specified file exists, modewrite
overwrites it, mode append creates a new file with an integer suffix. [default:write
]time
If the parameter is non-zero, the output interface will split captured data to individual files as often, as value of this parameter (in minutes) indicates. The output interface creates unique file name for each file according to current timestamp in format: "filename.YYYYmmddHHMM". [default: 0]
size
If the parameter is non-zero, the output interface will split captured data into individual files after a size of a current file (in MB) exceeds given threshold. Numeric suffix is added to the original file name for each file in ascending order starting with 0. [default: 0]
Conversion from IPFIX fields to UniRec fields is defined in the configuration file unirec-element.txt. The file is distributed and installed together with the plugin and it is usually placed in the same directory as the default IPFIXcol startup configuration (see ipfixcol2 -h
for help).
The structure of the file is simple. Every line corresponds to one UniRec field and consists of three mandatory parameters (name, type, list of IPFIX Information Elements). For example, a line: "BYTES uint64 e0id1"
:
- First parameter specifies an UniRec name. This name is used in a plugin configuration in the
<uniRecFormat>
element. - Second parameter specifies a data type of the UniRec field. List of all supported types is available in UniRec documentation. The plugin also supports additional virtual type "string_trimmed" that converts data to UniRec string type, but only characters up to the first occurrence of '\0' (excluding) are copied.
- The third parameter is comma separated list of corresponding IPFIX Information Elements (IEs). In this case, "e0id1" means IPFIX IE with Private Enterprise ID 0 and IE ID 1 (which is "octetDeltaCount"). Instead of numeric identification an IE name in "<scope>:<name>" format can be also used, for example,
"BYTES uint64 iana:octetDeltaCount"
.
First of all, make sure that definitions of IPFIX Information Elements to convert are known to the IPFIXcol. The plugin needs to know the definitions to find appropriate conversion functions from IPFIX to UniRec data types. All definitions of IEs are provided by libfds library and if any definition is missing, it is possible to add it manually to configuration. See the project website for help.
Now you can create a new entry in the configuration file. All three parameters, i.e. UniRec field name, UniRec type and list of IPFIX IEs, must be defined. If the mapping is configured correctly the new UniRec field can be used in the template specification i.e. <uniRecFormat>
.
Be aware that data types of IPFIX IEs and corresponding UniRec fields could be slightly different. When a value of an IPFIX field cannot fit into a UniRec field (e.g. conversion from 64b to 32b unsigned integer), the converted value is saturated (the maximum/minimum possible value is used) or extra bits are discarded. To distinguish these situation, the data semantic of used IPFIX IE is used. If its semantic is flags
or identifier
, extra bits are discarded. Otherwise, the value is saturated.
Note: If saturation is applied and a negative integer is converted from signed to unsigned integer type, the result value is zero.
To see conversion warnings, add the UniRec field to <uniRecFormat>
and run the collector with increased verbosity level i.e. ipfixcol2 -v
.
When multiple IPFIX Information Elements are mapped to the same UniRec field and those IPFIX fields are present in an IPFIX record, the last field occurrence (in the appropriate IPFIX Template) is converted to the UniRec field.
Due to historical reasons, special UniRec fields LINK_BIT_FIELD
and DIR_BIT_FIELD
are implemented using internal conversion functions. LINK_BIT_FIELD
represents Observation Domain ID (ODID) of a flow record as bit field. For example, ODID 5 is binary stored as ...00100000 (i.e. 32). Keep on mind that, maximum size of the UniRec field is 2^64, therefore modulo 64 is applied to ODID before the conversion. The purpose of DIR_BIT_FIELD
is to distinguish ingress and egress flows. For this reason, the lowest bit of IPFIX field iana:ingressInterface
(PEN: 0, ID: 10) is copied to the particular UniRec field (Nemea modules expects that 1 = ingress flow and 0 = egress flow).
Other UniRec specific fields available are ODID
and EXPORTER_IP
. The information for these fields is extracted from the IPFIX message session. The ODID
contains raw Observation Domain ID of the exporter and EXPORTER_IP
carries exporter IP address to identify which probe the flow was observed on.