There are two main ways of deploying Packetbeat:
-
On dedicated servers, getting the traffic from mirror ports or tap devices.
-
On your existing application servers.
The first option has the big advantage that there is no overhead of any kind on your application servers. But it requires dedicated networking gear, which is generally not available on cloud setups.
In both cases, the sniffing performance (reading packets passively from the network) is very important. In the case of a dedicated server, better sniffing performance means that less hardware is required. When Packetbeat is installed on an existing application server, better sniffing performance means less overhead.
Currently Packetbeat has several options for traffic capturing:
-
pcap, which uses the libpcap library and works on most platforms, but it’s not the fastest option. -
af_packet, which uses memory mapped sniffing. This option is faster than libpcap and doesn’t require a kernel module, but it’s Linux-specific.
The af_packet option, also known as "memory-mapped sniffing," makes use of a
Linux-specific
feature.
This could be the optimal sniffing mode for both the dedicated server and
when Packetbeat is deployed on an existing application server.
The way it works is that both the kernel and the user space program map the same memory zone, and a simple circular buffer is organized in this memory zone. The kernel writes packets into the circular buffer, and the user space program reads from it. The poll system call is used for getting a notification for the first packet available, but the remaining available packets can be simply read via memory access.
The af_packet sniffer can be further tuned to use more memory in exchange for
better performance. The larger the size of the circular buffer, the fewer
system calls are needed, which means that fewer CPU cycles are consumed. The default size
of the buffer is 30 MB, but you can increase it like this:
packetbeat.interfaces.device: eth0
packetbeat.interfaces.type: af_packet
packetbeat.interfaces.buffer_size_mb: 100On Windows {beatname_uc} requires an Npcap DLL installation. This is provided by {beatname_uc}
for users of the Elastic Licenced version. In some cases users may wish to use
their own installed version. In order to do this the packetbeat.npcap.never_install
option can be used. Setting this option to true will not attempt to install the
bundled Npcap library on start-up unless no Npcap is already installed.
packetbeat.npcap.never_install: trueYou can specify the following options in the packetbeat.interfaces section
of the {beatname_lc}.yml config file. Here is an example configuration:
packetbeat.interfaces.device: any
packetbeat.interfaces.snaplen: 1514
packetbeat.interfaces.type: af_packet
packetbeat.interfaces.buffer_size_mb: 100The network device to capture traffic from. The specified device is set automatically to promiscuous mode, meaning that Packetbeat can capture traffic from other hosts on the same LAN.
Example:
packetbeat.interfaces.device: eth0On Linux, you can specify any for the device, and Packetbeat captures all
messages sent or received by the server where Packetbeat is installed.
|
Note
|
When you specify any for the device, the interfaces are not set
to promiscuous mode.
|
The device option also accepts specifying the device by its index in the list of
devices available for sniffing. To obtain the list of available devices,
run Packetbeat with the following command:
packetbeat devicesThis command returns a list that looks something like the following:
0: en0 (No description available)
1: awdl0 (No description available)
2: bridge0 (No description available)
3: fw0 (No description available)
4: en1 (No description available)
5: en2 (No description available)
6: p2p0 (No description available)
7: en4 (No description available)
8: lo0 (No description available)The following example sets up sniffing on the first interface in the list:
packetbeat.interfaces.device: 0Specifying the index is especially useful on Windows where device names can be long.
Alternatively the default_route, default_route_ipv4 or default_route_ipv6 device
may be specified. This will set the capture device to be the device that is associated
with the first default route identified on Packetbeat start-up. default_route will
select the first default route from either IPv4 or IPv6 with a preference for the IPv4
route, while default_route_ipv4 and default_route_ipv6 will only select from the
specified stack. The selected interface is not altered after it is chosen.
The maximum size of the packets to capture. The default is 65535, which is large enough for almost all networks and interface types. If you sniff on a physical network interface, the optimal setting is the MTU size. On virtual interfaces, however, it’s safer to accept the default value.
Example:
packetbeat.interfaces.device: eth0
packetbeat.interfaces.snaplen: 1514Packetbeat supports these sniffer types:
-
pcap, which uses the libpcap library and works on most platforms, but it’s not the fastest option. -
af_packet, which uses memory-mapped sniffing. This option is faster than libpcap and doesn’t require a kernel module, but it’s Linux-specific.
The default sniffer type is pcap.
Here is an example configuration that specifies
the af_packet sniffing type:
packetbeat.interfaces.device: eth0
packetbeat.interfaces.type: af_packetOn Linux, if you are trying to optimize the CPU usage of Packetbeat, we
recommend trying the af_packet option.
If you use the af_packet sniffer, you can tune its behaviour by specifying the
following options:
The maximum size of the shared memory buffer to use
between the kernel and user space. A bigger buffer usually results in lower CPU
usage, but consumes more memory. This setting is only available for the
af_packet sniffer type. The default is 30 MB.
Example:
packetbeat.interfaces.device: eth0
packetbeat.interfaces.type: af_packet
packetbeat.interfaces.buffer_size_mb: 100To scale processing across multiple Packetbeat processes, a fanout group
identifier can be specified. When fanout_group is used the Linux kernel splits
packets across Packetbeat instances in the same group by using a flow hash. It
computes the flow hash modulo with the number of Packetbeat processes in order
to consistently route flows to the same Packetbeat instance.
The value must be between 0 and 65535. By default, no value is set.
This is only available on Linux and requires using type: af_packet. Each process
must be running in same network namespace. All processes must use the same
interface settings. You must take responsibility for running multiple instances
of Packetbeat.
Example:
packetbeat.interfaces.type: af_packet
packetbeat.interfaces.fanout_group: 1Configure the metrics polling interval for supported interface types. Currently,
only af_packet is supported.
The value must be a duration string. The default is 5s (5 seconds). A value
less than or equal to zero will be set to the default value.
Example:
packetbeat.interfaces.type: af_packet
packetbeat.interfaces.metrics_interval: 5sWith auto_promisc_mode Packetbeat puts interface in promiscuous mode automatically on startup.
This option does not work with any interface device.
The default option is false and requires manual set-up of promiscuous mode.
Warning: under some circumstances (e.g beat crash) promiscuous mode
can stay enabled even after beat is shut down.
Example:
packetbeat.interfaces.device: eth0
packetbeat.interfaces.type: af_packet
packetbeat.interfaces.buffer_size_mb: 100
packetbeat.interfaces.auto_promisc_mode: truePacketbeat automatically generates a
BPF for capturing only
the traffic on ports where it expects to find known protocols.
For example, if you have configured port 80 for HTTP and port 3306 for MySQL,
Packetbeat generates the following BPF filter: "port 80 or port 3306".
However, if the traffic contains VLAN
tags, the filter that Packetbeat generates is ineffective because the
offset is moved by four bytes. To fix this, you can enable the with_vlans option, which
generates a BPF filter that looks like this: "port 80 or port 3306 or (vlan and (port 80 or port 3306))".
Packetbeat automatically generates a
BPF for capturing only
the traffic on ports where it expects to find known protocols.
For example, if you have configured port 80 for HTTP and port 3306 for MySQL,
Packetbeat generates the following BPF filter: "port 80 or port 3306".
You can use the bpf_filter setting to overwrite the generated BPF filter. For example:
packetbeat.interfaces.device: eth0
packetbeat.interfaces.bpf_filter: "net 192.168.238.0/0 and port 80 or port 3306"|
Note
|
This setting disables automatic generation of the BPF filter. If
you use this setting, it’s your responsibility to keep the BPF filters in sync with the
ports defined in the protocols section.
|
If the ignore_outgoing option is enabled, Packetbeat ignores all the
transactions initiated from the server running Packetbeat.
This is useful when two Packetbeat instances publish the same transactions. Because one Packetbeat
sees the transaction in its outgoing queue and the other sees it in its incoming
queue, you can end up with duplicate transactions. To remove the duplicates, you
can enable the packetbeat.ignore_outgoing option on one of the servers.
For example, in the following scenario, you see a 3-server architecture where a Beat is installed on each server. t1 is the transaction exchanged between Server1 and Server2, and t2 is the transaction between Server2 and Server3.
By default, each transaction is indexed twice because Beat2
sees both transactions. So you would see the following published transactions
(when ignore_outgoing is false):
-
Beat1: t1
-
Beat2: t1 and t2
-
Beat3: t2
To avoid duplicates, you can force your Beats to send only the incoming
transactions and ignore the transactions created by the local server. So you would
see the following published transactions (when ignore_outgoing is true):
-
Beat1: none
-
Beat2: t1
-
Beat3: t2
If the internal_networks option is specified, when monitoring network taps or mirror ports, Packetbeat
will attempt to classify the network directionality of traffic not intended for this host as it
relates to a network perimeter. Any CIDR block specified in internal_networks is treated internal to
the perimeter, and any IP address falling outside of these CIDR blocks is considered external.
This is useful when Packetbeat is running on an appliance that sits at a network boundary such as a firewall or VPN. Note that this only affects how the directionality of network traffic is classified.
You can configure Packetbeat to collect and report statistics on network flows. A flow is a group of packets sent over the same time period that share common properties, such as the same source and destination address and protocol. You can use this feature to analyze network traffic over specific protocols on your network.
For each flow, Packetbeat reports the number of packets and the total number of bytes sent from the source to the destination. Each flow event also contains information about the source and destination hosts, such as their IP address. For bi-directional flows, Packetbeat reports statistics for the reverse flow.
Packetbeat collects and reports statistics up to and including the transport layer. See [exported-fields-flows_event] for more info about the exported data.
Here’s an example of flow events visualized in the Flows dashboard:
To configure flows, use the packetbeat.flows option in the {beatname_lc}.yml
config file. Flows are enabled by default. If this section is missing
from the configuration file, network flows are disabled.
packetbeat.flows:
timeout: 30s
period: 10sHere’s an example of a flow information sent by Packetbeat. See [exported-fields-flows_event] for a description of each field.
{
"@timestamp": "2018-11-15T14:41:24.000Z",
"agent": {
"hostname": "host.example.com",
"name": "host.example.com",
"version": "{version}"
},
"destination": {
"bytes": 460,
"ip": "198.51.100.2",
"mac": "06-05-04-03-02-01",
"packets": 2,
"port": 80
},
"event": {
"dataset": "flow",
"duration": 3000000000,
"end": "2018-11-15T14:41:24.000Z",
"start": "2018-11-15T14:41:21.000Z"
},
"flow": {
"final": true, <1>
"id": "FQQA/wz/Dv//////Fv8BAQEBAgMEBQYGBQQDAgGrAMsAcQPGM2QC9ZdQAA",
"vlan": 171
},
"network": {
"bytes": 470,
"community_id": "1:t9T66/2c66NQyftAEsr4aMZv4Hc=",
"packets": 3,
"transport": "tcp",
"type": "ipv4"
},
"source": {
"bytes": 10,
"ip": "203.0.113.3",
"mac": "01-02-03-04-05-06",
"packets": 1,
"port": 38901
}
}-
Packetbeat sets the
flow.finalflag tofalseto indicate that the event contains an intermediate report about a flow that it’s tracking. When the flow completes, Packetbeat sends one last event withflow.finalset totrue. If you want to aggregate sums of traffic, you need to filter onfinal:true, or use some other technique, so that you get only the latest update from each flow. You can disable intermediate reports by settingperiod: -1s.
You can specify the following options in the packetbeat.flows section of
the {beatname_lc}.yml config file:
Enables flows support if set to true. Set to false to disable network flows support without having to delete or comment out the flows section. The default value is true.
Timeout configures the lifetime of a flow. If no packets have been received for a flow within the timeout time window, the flow is killed and reported. The default value is 30s.
Configure the reporting interval. All flows are reported at the very same point in time. Periodical reporting can be disabled by setting the value to -1. If disabled, flows are still reported once being timed out. The default value is 10s.
Optional fields that you can specify to add additional information to the
output. For example, you might add fields that you can use for filtering log
data. Fields can be scalar values, arrays, dictionaries, or any nested
combination of these. By default, the fields that you specify here will be
grouped under a fields sub-dictionary in the output document. To store the
custom fields as top-level fields, set the fields_under_root option to true.
If a duplicate field is declared in the general configuration, then its value
will be overwritten by the value declared here.
If this option is set to true, the custom fields
are stored as top-level fields in the output document instead of being grouped
under a fields sub-dictionary. If the custom field names conflict with other
field names added by Packetbeat, then the custom fields overwrite the other
fields.
A list of tags that will be sent with the protocol event. This setting is optional.
A list of processors to apply to the data generated by the protocol.
See [filtering-and-enhancing-data] for information about specifying processors in your config.
If this option is set to true, fields with null values will be published in
the output document. By default, keep_null is set to false.
Overrides the index that flow events are published to.
The packetbeat.protocols section of the {beatname_lc}.yml config file
contains configuration options for each supported protocol, including common
options like enabled, ports, send_request, send_response, and options
that are protocol-specific.
Currently, Packetbeat supports the following protocols:
Example configuration:
packetbeat.protocols:
- type: icmp
enabled: true
- type: dhcpv4
ports: [67, 68]
- type: dns
ports: [53]
- type: http
ports: [80, 8080, 8000, 5000, 8002]
- type: amqp
ports: [5672]
- type: cassandra
ports: [9042]
- type: memcache
ports: [11211]
- type: mysql
ports: [3306,3307]
- type: redis
ports: [6379]
- type: pgsql
ports: [5432]
- type: thrift
ports: [9090]
- type: tls
ports: [443, 993, 995, 5223, 8443, 8883, 9243]The following options are available for all protocols:
The enabled setting is a boolean setting to enable or disable protocols without having to comment out configuration sections. If set to false, the protocol is disabled.
The default value is true.
Exception: For ICMP the option enabled has to be used instead.
The ports where Packetbeat will look to capture traffic for specific protocols. Packetbeat installs a BPF filter based on the ports specified in this section. If a packet doesn’t match the filter, very little CPU is required to discard the packet. Packetbeat also uses the ports specified here to determine which parser to use for each packet.
If this option is enabled, the raw message of the request (request field) is
sent to Elasticsearch. The default is false. This option is useful when you want to
index the whole request. Note that for HTTP, the body is not included by
default, only the HTTP headers.
If this option is enabled, the raw message of the response (response field)
is sent to Elasticsearch. The default is false. This option is useful when you
want to index the whole response. Note that for HTTP, the body is not included
by default, only the HTTP headers.
The per protocol transaction timeout. Expired transactions will no longer be correlated to incoming responses, but sent to Elasticsearch immediately.
Optional fields that you can specify to add additional information to the
output. For example, you might add fields that you can use for filtering log
data. Fields can be scalar values, arrays, dictionaries, or any nested
combination of these. By default, the fields that you specify here will be
grouped under a fields sub-dictionary in the output document. To store the
custom fields as top-level fields, set the fields_under_root option to true.
If a duplicate field is declared in the general configuration, then its value
will be overwritten by the value declared here.
Overrides the index that events for the given protocol are published to.
packetbeat.protocols:
- type: http
ports: [80]
fields:
service_id: nginxIf this option is set to true, the custom fields
are stored as top-level fields in the output document instead of being grouped
under a fields sub-dictionary. If the custom field names conflict with other
field names added by Packetbeat, then the custom fields overwrite the other
fields.
A list of tags that will be sent with the transaction event. This setting is optional.
A list of processors to apply to the data generated by the protocol.
See [filtering-and-enhancing-data] for information about specifying processors in your config.
If this option is set to true, fields with null values will be published in
the output document. By default, keep_null is set to false.
The icmp section of the {beatname_lc}.yml config file specifies options for
the ICMP protocol. Here is a sample configuration section for ICMP:
packetbeat.protocols:
- type: icmp
enabled: trueAlso see Common protocol options.
The dns section of the {beatname_lc}.yml config file specifies configuration
options for the DNS protocol. The DNS protocol supports processing DNS messages
on TCP and UDP. Here is a sample configuration section for DNS:
packetbeat.protocols:
- type: dns
ports: [53]
include_authorities: true
include_additionals: trueAlso see Common protocol options.
If this option is enabled, dns.authority fields (authority resource records) are added to DNS events. The default is false.
The HTTP protocol has several specific configuration options. Here is a
sample configuration for the http section of the {beatname_lc}.yml config file:
packetbeat.protocols:
- type: http
ports: [80, 8080, 8000, 5000, 8002]
hide_keywords: ["pass", "password", "passwd"]
send_headers: ["User-Agent", "Cookie", "Set-Cookie"]
split_cookie: true
real_ip_header: "X-Forwarded-For"Also see Common protocol options.
A list of query parameters that Packetbeat will automatically censor in
the transactions that it saves. The values associated with these parameters are replaced
by 'xxxxx'. By default, no changes are made to the HTTP messages.
Packetbeat has this option because, unlike SQL traffic, which typically only contains the hashes of the passwords, HTTP traffic may contain sensitive data. To reduce security risks, you can configure this option to avoid sending the contents of certain HTTP POST parameters.
|
Warning
|
This option replaces query parameters from GET requests and top-level
parameters from POST requests. If sensitive data is encoded inside a
parameter that you don’t specify here, Packetbeat cannot censor it. Also, note that if
you configure Packetbeat to save the raw request and response fields (see the send_request
and the send_response options), sensitive data may be present in those
fields.
|
When this option is enabled, Packetbeat obscures the value of
Authorization and Proxy-Authorization HTTP headers, and censors
those strings in the response.
You should set this option to true for transactions that use Basic Authentication because they may contain the base64 unencrypted username and password.
A list of header names to capture and send to Elasticsearch. These
headers are placed under the headers dictionary in the resulting JSON.
Instead of sending a white list of headers to Elasticsearch, you can send all headers by setting this option to true. The default is false.
A list of headers to redact if present in the HTTP request. This will keep the header field present, but will redact it’s value to show the header’s presence.
The list of content types for which Packetbeat exports the full HTTP payload. The HTTP body is available under
http.request.body.content and http.response.body.content for these Content-Types.
In addition, if send_response option is enabled, then the HTTP body is exported together with the HTTP
headers under response and if
send_request enabled, then request contains the entire HTTP message including the body.
In the following example, the HTML attachments of the HTTP responses are exported under the response field and under
http.request.body.content or http.response.body.content:
packetbeat.protocols:
- type: http
ports: [80, 8080]
send_response: true
include_body_for: ["text/html"]A boolean flag that controls decoding of HTTP payload. It interprets the
Content-Encoding and Transfer-Encoding headers and uncompresses the entity
body. Supported encodings are gzip and deflate. This option is only
applicable in the cases where the HTTP payload is exported, that is, when
one of the include_*_body_for options is specified or a POST request
contains url-encoded parameters.
If the Cookie or Set-Cookie headers are sent, this option controls whether
they are split into individual values. For example, with this option set, an
HTTP response might result in the following JSON:
"response": {
"code": 200,
"headers": {
"connection": "close",
"content-language": "en",
"content-type": "text/html; charset=utf-8",
"date": "Fri, 21 Nov 2014 17:07:34 GMT",
"server": "gunicorn/19.1.1",
"set-cookie": { (1)
"csrftoken": "S9ZuJF8mvIMT5CL4T1Xqn32wkA6ZSeyf",
"expires": "Fri, 20-Nov-2015 17:07:34 GMT",
"max-age": "31449600",
"path": "/"
},
"vary": "Cookie, Accept-Language"
},
"status_phrase": "OK"
}-
Note that
set-cookieis a map containing the cookie names as keys.
The default is false.
The header field to extract the real IP from. This setting is useful when you
want to capture traffic behind a reverse proxy, but you want to get the
geo-location information. If this header is present and contains a valid IP
addresses, the information is used for the network.forwarded_ip field.
The amqp section of the {beatname_lc}.yml config file specifies configuration options for the AMQP 0.9.1
protocol. Here is a sample configuration:
packetbeat.protocols:
- type: amqp
ports: [5672]
max_body_length: 1000
parse_headers: true
parse_arguments: false
hide_connection_information: trueAlso see Common protocol options.
The maximum size in bytes of the message displayed in the request or response fields.
Messages that are bigger than the specified size are truncated. Use this option to
avoid publishing huge messages when send_request or send_request
is enabled. The default is 1000 bytes.
If set to true, Packetbeat parses the additional arguments specified in the headers field of a message. Those arguments are key-value pairs that specify information such as the content type of the message or the message priority. The default is true.
If set to true, Packetbeat parses the additional arguments specified in AMQP methods. Those arguments are key-value pairs specified by the user and can be of any length. The default is true.
The following settings are specific to the Cassandra protocol. Here is a sample
configuration for the cassandra section of the {beatname_lc}.yml config file:
packetbeat.protocols:
- type: cassandra
send_request_header: true
send_response_header: true
compressor: "snappy"
ignored_ops: ["SUPPORTED","OPTIONS"]Also see Common protocol options.
If this option is enabled, the raw message of the response (cassandra_request.request_headers field)
is sent to Elasticsearch. The default is true. enable send_request first before enable this option.
If this option is enabled, the raw message of the response (cassandra_response.response_headers field)
is included in published events. The default is true. enable send_response first before enable this option.
This option indicates which Operator/Operators captured will be ignored. currently support:
ERROR ,STARTUP ,READY ,AUTHENTICATE ,OPTIONS ,SUPPORTED ,
QUERY ,RESULT ,PREPARE ,EXECUTE ,REGISTER ,EVENT ,
BATCH ,AUTH_CHALLENGE,AUTH_RESPONSE ,AUTH_SUCCESS .
The memcache section of the {beatname_lc}.yml config file specifies configuration options for the memcache
protocol. Here is a sample configuration section for memcache:
packetbeat.protocols:
- type: memcache
ports: [11211]
parseunknown: false
maxvalues: 0
maxbytespervalue: 100
transaction_timeout: 200
udptransactiontimeout: 200Also see Common protocol options.
When this option is enabled, it forces the memcache text protocol parser to accept unknown commands.
|
Note
|
The unknown commands MUST NOT contain a data part. |
The maximum number of values to store in the message (multi-get). All values will be base64 encoded.
The possible settings for this option are:
-
maxvalue: -1, which stores all values (text based protocol multi-get) -
maxvalue: 0, which stores no values (default) -
maxvalue: N, which stores up to N values
The maximum number of bytes to be copied for each value element.
|
Note
|
Values will be base64 encoded, so the actual size in the JSON document will be 4 times the value that
you specify for maxbytespervalue.
|
The transaction timeout in milliseconds. The defaults is 10000 milliseconds.
|
Note
|
Quiet messages in UDP binary protocol get responses only if there is an error.
The memcache protocol analyzer will wait for the number of milliseconds specified by
udptransactiontimeout before publishing quiet messages. Non-quiet messages or
quiet requests with an error response are published immediately.
|
The mysql section of the {beatname_lc}.yml config file specifies configuration
options for the MySQL protocols.
packetbeat.protocols:
- type: mysql
ports: [3306,3307]Also see Common protocol options.
The maximum number of rows from the SQL message to publish to Elasticsearch. The default is 10 rows.
The pgsql sections of the {beatname_lc}.yml config file specifies configuration
options for the PgSQL protocols.
packetbeat.protocols:
- type: pgsql
ports: [5432]Also see Common protocol options.
The maximum number of rows from the SQL message to publish to Elasticsearch. The default is 10 rows.
Apache Thrift is a communication protocol and RPC framework initially created at Facebook. It is sometimes used in microservices architectures because it provides better performance when compared to the more obvious HTTP/RESTful API choice, while still supporting a wide range of programming languages and frameworks.
Packetbeat works based on a copy of the traffic, which means that you get performance management features without having to modify your services in any way and without any latency overhead. Packetbeat captures the transactions from the network and indexes them in Elasticsearch so that they can be analyzed and searched.
Packetbeat indexes the method, parameters, return value, and
exceptions of each Thrift-RPC call. You can search by and create statistics
based on any of these fields. Packetbeat automatically fills in the status
column with either OK or Error, so it’s easy to find the problematic RPC calls.
A transaction is put into the Error state if it returned an exception.
Packetbeat also indexes the event.duration field so you can get performance
analytics and find the slow RPC calls.
Here is an example performance dashboard:
Thrift supports multiple transport
and protocol types. Currently Packetbeat supports the default TSocket
transport as well as the TFramed transport. From the protocol point of view,
Packetbeat currently supports only the default TBinary protocol.
Packetbeat also has several configuration options that allow you to get the right balance between visibility, disk usage, and data protection. You can, for example, choose to obfuscate all strings or to store the requests but not the responses, while still capturing the response time for each of the RPC calls. You can also choose to limit the size of strings and lists to a given number of elements, so you can fine tune how much data you want to have stored in Elasticsearch.
The Thrift protocol has several specific configuration options. Here is an example configuration section for the Thrift protocol in the {beatname_lc}.yml config file:
packetbeat.protocols:
- type: thrift
transport_type: socket
protocol_type: binary
idl_files: ["tutorial.thrift", "shared.thrift"]
string_max_size: 200
collection_max_size: 20
capture_reply: true
obfuscate_strings: true
drop_after_n_struct_fields: 100Providing the Thrift IDL files to Packetbeat is optional. The binary Thrift messages include the called method name and enough structural information to decode the messages without needing the IDL files. However, if you provide the IDL files, Packetbeat can also resolve the service name, arguments, and exception names.
Also see Common protocol options.
The Thrift transport type. Currently this option accepts the values socket
for TSocket, which is the default Thrift transport, and framed for the TFramed Thrift
transport. The default is socket.
The Thrift protocol type. Currently the only accepted value is binary for
the TBinary protocol, which is the default Thrift protocol.
The Thrift interface description language (IDL) files for the service that Packetbeat is monitoring. Providing the IDL files is optional, because the Thrift messages contain enough information to decode them without having the IDL files. However, providing the IDL enables Packetbeat to include parameter and exception names.
The maximum length for strings in parameters or return values. If a string is longer than this value, the string is automatically truncated to this length. Packetbeat adds dots at the end of the string to mark that it was truncated. The default is 200.
The maximum number of elements in a Thrift list, set, map, or structure. If a collection
has more elements than this value, Packetbeat captures only the
specified number of elements. Packetbeat adds a fictive last element … to the end
of the collection to mark that it was truncated. The default is 15.
If this option is set to false, Packetbeat decodes the method name from the reply and simply skips the rest of the response message. This setting can be useful for performance, disk usage, or data retention reasons. The default is true.
If this option is set to true, Packetbeat replaces all strings found in method parameters,
return codes, or exception structures with the "*" string.
The maximum number of fields that a structure can have before Packetbeat ignores the whole transaction. This is a memory protection mechanism (so that Packetbeat’s memory doesn’t grow indefinitely), so you would typically set this to a relatively high value. The default is 500.
The following settings are specific to the MongoDB protocol. Here is a sample
configuration for the mongodb section of the {beatname_lc}.yml config file:
packetbeat.protocols:
- type: mongodb
send_request: true
send_response: true
max_docs: 0
max_doc_length: 0The max_docs and max_doc_length settings are useful for limiting the amount
of data Packetbeat indexes in the response fields.
Also see Common protocol options.
The maximum number of documents from the response to index in the response field. The
default is 10. You can set this to 0 to index an unlimited number of documents.
Packetbeat adds a […] line at the end to signify that there were additional documents
that weren’t saved because of this setting.
The maximum number of characters in a single document indexed in the response
field. The default is 5000. You can set this to 0 to index an unlimited number
of characters per document.
If the document is trimmed because of this setting, Packetbeat adds the string …
at the end of the document.
Note that limiting documents in this way means that they are no longer correctly formatted JSON objects.
TLS is a cryptographic protocol that provides secure communications on top of an existing application protocol, like HTTP or MySQL.
Packetbeat intercepts the initial handshake in a TLS connection and extracts useful information that helps operators diagnose problems and strengthen the security of their network and systems. It does not decrypt any information from the encapsulated protocol, nor does it reveal any sensitive information such as cryptographic keys. TLS versions 1.0 to 1.3 are supported.
It works by intercepting the client and server "hello" messages, which contain the negotiated parameters for the connection such as cryptographic ciphers and protocol versions. It can also intercept TLS alerts, which are sent by one of the parties to signal a problem with the negotiation, such as an expired certificate or a cryptographic error.
An example of indexed event:
"tls": {
"client": {
"supported_ciphers": [
"TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384",
"TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384",
"TLS_RSA_WITH_3DES_EDE_CBC_SHA",
"TLS_EMPTY_RENEGOTIATION_INFO_SCSV"
],
"ja3": "e6573e91e6eb777c0933c5b8f97f10cd",
"server_name": "example.net"
},
"server": {
"subject": "CN=www.example.org,OU=Technology,O=Internet Corporation for Assigned Names and Numbers,L=Los Angeles,ST=California,C=US",
"issuer": "CN=DigiCert SHA2 Secure Server CA,O=DigiCert Inc,C=US",
"not_before": "2018-11-28T00:00:00.000Z",
"not_after": "2020-12-02T12:00:00.000Z",
"hash": {
"sha1": "7BB698386970363D2919CC5772846984FFD4A889"
}
},
"version": "1.2",
"version_protocol": "tls",
"cipher": "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
"established": true,
"next_protocol": "h2",
"detailed": {
"server_certificate": {
"subject": {
"common_name": "www.example.org",
"country": "US",
"organization": "Internet Corporation for Assigned Names and Numbers",
"organizational_unit": "Technology",
"locality": "Los Angeles",
"province": "California"
},
"not_after": "2020-12-02T12:00:00.000Z",
"public_key_size": 2048,
"alternative_names": [
"www.example.org",
"example.com",
"example.edu",
"example.net",
"example.org",
"www.example.com",
"www.example.edu",
"www.example.net"
],
"signature_algorithm": "SHA256-RSA",
"version": 3,
"issuer": {
"organization": "DigiCert Inc",
"common_name": "DigiCert SHA2 Secure Server CA",
"country": "US"
},
"not_before": "2018-11-28T00:00:00.000Z",
"public_key_algorithm": "RSA",
"serial_number": "21020869104500376438182461249190639870"
},
"server_certificate_chain": [
{
"public_key_algorithm": "RSA",
"not_before": "2013-03-08T12:00:00.000Z",
"not_after": "2023-03-08T12:00:00.000Z",
"version": 3,
"serial_number": "2646203786665923649276728595390119057",
"issuer": {
"organizational_unit": "www.digicert.com",
"common_name": "DigiCert Global Root CA",
"country": "US",
"organization": "DigiCert Inc"
},
"subject": {
"country": "US",
"organization": "DigiCert Inc",
"common_name": "DigiCert SHA2 Secure Server CA"
},
"public_key_size": 2048,
"signature_algorithm": "SHA256-RSA"
},
{
"public_key_algorithm": "RSA",
"subject": {
"common_name": "DigiCert Global Root CA",
"country": "US",
"organization": "DigiCert Inc",
"organizational_unit": "www.digicert.com"
},
"issuer": {
"country": "US",
"organization": "DigiCert Inc",
"organizational_unit": "www.digicert.com",
"common_name": "DigiCert Global Root CA"
},
"signature_algorithm": "SHA1-RSA",
"serial_number": "10944719598952040374951832963794454346",
"not_before": "2006-11-10T00:00:00.000Z",
"not_after": "2031-11-10T00:00:00.000Z",
"public_key_size": 2048,
"version": 3
}
],
"client_certificate_requested": false,
"version": "TLS 1.2",
"client_hello": {
"version": "3.3",
"supported_compression_methods": [
"NULL"
],
"extensions": {
"ec_points_formats": [
"uncompressed"
],
"supported_groups": [
"x25519",
"secp256r1",
"secp384r1"
],
"signature_algorithms": [
"rsa_pkcs1_sha512",
"ecdsa_secp521r1_sha512",
"(unknown:0xefef)",
"rsa_pkcs1_sha384",
"ecdsa_secp384r1_sha384",
"rsa_pkcs1_sha256",
"ecdsa_secp256r1_sha256",
"(unknown:0xeeee)",
"(unknown:0xeded)",
"(unknown:0x0301)",
"(unknown:0x0303)",
"rsa_pkcs1_sha1",
"ecdsa_sha1"
],
"application_layer_protocol_negotiation": [
"h2",
"http/1.1"
],
"server_name_indication": [
"example.net"
]
}
},
"server_hello": {
"version": "3.3",
"session_id": "23bb2aed5d215e1228220b0a51d7aa220785e9e4b83b4f430229117971e9913f",
"selected_compression_method": "NULL",
"extensions": {
"application_layer_protocol_negotiation": [
"h2"
],
"_unparsed_": [
"renegotiation_info",
"server_name_indication"
],
"ec_points_formats": [
"uncompressed",
"ansiX962_compressed_prime",
"ansiX962_compressed_char2"
]
}
}
}
}The TLS events generated by {beatname_uc} follow the Elastic Common Schema (ECS) format. See {ecs-ref}/ecs-tls.html[ECS TLS fields] for a description of the populated fields.
Detailed information that is not defined in ECS is added under the
tls.detailed key. The include_detailed_fields configuration flag is used
to control whether this information is exported.
The fields under tls.detailed.client_hello contain the algorithms and
extensions supported by the client, as well as the maximum TLS version
it supports.
Fields under tls.detailed.server_hello contain the final settings for the TLS
session: The selected cipher, compression method, TLS version to use and
other extensions such as application layer protocol negotiation (ALPN).
See the [exported-fields-tls_detailed] section for more information.
The following settings are specific to the TLS protocol. Here is a sample
configuration for the tls section of the {beatname_lc}.yml config file:
packetbeat.protocols:
- type: tls
send_certificates: true
include_raw_certificates: false
include_detailed_fields: true
fingerprints: [ md5, sha1, sha256 ]The send_certificates and include_detailed_fields settings are useful for
limiting the amount of data Packetbeat indexes, as multiple certificates are
usually exchanged in a single transaction, and those can take a considerable
amount of storage.
Also see Common protocol options.
This setting causes information about the certificates presented by the
client and server to be included in the detailed fields. The server’s
certificate is indexed under tls.detailed.server_certificate and its
certification chain under tls.detailed.server_certificate_chain.
For the client, the client_certificate and client_certificate_chain fields
are used. The default is true.
You can set include_raw_certificates to include the raw certificate chains
encoded in PEM format, under the tls.server.certificate_chain and
tls.client.certificate_chain fields. The default is false.
Controls whether the [exported-fields-tls_detailed] are added to exported
documents. When set to false, only {ecs-ref}/ecs-tls.html[ECS TLS fields]
are included. The default is true.
The Redis protocol has several specific configuration options. Here is a
sample configuration for the redis section of the {beatname_lc}.yml config file:
packetbeat.protocols:
- type: redis
ports: [6379]
queue_max_bytes: 1048576
queue_max_messages: 20000Also see Common protocol options.
In order for request/response correlation to work, {beatname_uc} needs to
store requests in memory until a response is received. These settings impose
a limit on the number of bytes (queue_max_bytes) and number of requests
(queue_max_messages) that can be stored. These limits are per-connection.
The default is to queue up to 1MB or 20.000 requests per connection, which
allows to use request pipelining while at the same time limiting the amount
of memory consumed by replication sessions.
This section of the {beatname_lc}.yml config file is optional, but configuring the processes enables Packetbeat to show you not only the servers that the traffic is flowing between, but also the processes. Packetbeat can even show you the traffic between two processes running on the same host, which is particularly useful when you have many services running on the same server. By default, process enrichment is disabled.
When Packetbeat starts, and then periodically afterwards, it scans the process table for processes that match the configuration file. For each of these processes, it monitors which file descriptors it has opened. When a new packet is captured, it reads the list of active TCP and UDP connections and matches the corresponding one with the list of file descriptors.
All this information is available via system interfaces: The /proc file system
in Linux and the IP Helper API (iphlpapi.dll) on Windows, so {beatname_uc}
doesn’t need a kernel module.
|
Note
|
Process monitoring is currently only supported on Linux and Windows systems. Packetbeat automatically disables process monitoring when it detects other operating systems. |
Example configuration:
packetbeat.procs.enabled: trueWhen the process monitor is enabled, it will enrich all the events whose source
or destination is a local process. The source.process and/or
destination.process fields will be added to an event, when the server side or
client side of the connection belong to a local process, respectively.
You can specify the following process monitoring options in the monitored
section of the {beatname_lc}.yml config file to customize the name of process:
The name of the process as it will appear in the published transactions. The name doesn’t have to match the name of the executable, so feel free to choose something more descriptive (for example, "myapp" instead of "gunicorn").
The name used to identify the process at run time. When Packetbeat starts, and then
periodically afterwards, it scans the process table for
processes that match the values specified for this option. The match is done against the
process' command line as read from /proc/<pid>/cmdline.
How long Packetbeat waits on shutdown. By default, this option is disabled. Packetbeat
will wait for shutdown_timeout and then close. It will not track if all events were sent
previously.
Example configuration:
packetbeat.shutdown_timeout: 5sBy default Ingest pipelines are not updated if a pipeline with the same ID already exists. If this option is enabled {beatname_uc} overwrites pipelines every time a new Elasticsearch connection is established.
The default value is false.