investigate default capture settings for best Suricata performance

[mmguero]

We're not doing much in the way of tuning for Suricata capture, neither on Malcolm or on Hedgehog. We should go over some of these resources and make adjustments where needed. However, note that we do have a number of variables that can be set via suricata_config_populate.py so we might already have the stuff in place to handle it. We just need to examine what might be best for good defaults to work fast "out of the box."

• tuning considerations
• high performance

We should look at the defaults for any of the following:

• the af-packet section
• anything else that has to do with memory caps, hash sizes, buffer sizes, etc.

[mmguero]

See #445 as well, as splitting out eve.json by thread and enabling rotation will also help with performance.

[mmguero]

(note: all Malcolm environment variables in this document should be prepended with SURICATA_)

Tuning Considerations

Suricata setting	Malcolm variable	Default
max-pending-packets	MAX_PENDING_PACKETS	1024
mpm-algo	(not currently adjustable)	AC (Aho-Corasick)
detect.profile	(not currently adjustable)	high
detect.sgh-mpm-context	(not currently adjustable)	single
af-packet	various	various
stream.bypass	(not currently adjustable)	no

• we may want to see if Hyperscan is available/being used (I'm not sure without looking it up)

High Performance

• NIC
  • nic-capture-setup.sh is used to set NIC settings if PCAP_IFACE_TWEAK is true in pcap-capture or arkime-live containers. I should probably do this across all capture containers (I don't think it hurts anything to call it more than once, e.g., in each container's startup).
  • the value of threads can be set with the AF_PACKET_IFACE_THREADS variable (default is auto)
  • cluster-type can be set with AF_PACKET_CLUSTER_TYPE (default is cluster_flow)
    • cluster_qm could be used for "high end systems/NICs", see the Suricata documentation linked, although I don't think nic-capture-setup.sh does everything it's talking about there
• CPU affinity and NUMA
  • I'm not sure how we could do the ethtool stuff in a general sense with scripting minus just giving the user the ability to provide a script, but I feel like with that amount of specificity it might make sense to just run this external to Malcolm, as it seems quite specific to the NIC. We could potentially provide some variables for the cpu-affinity section under threading.
  • here it discusses some default values for ring-size and block-size which are significantly smaller than our defaults, we may want to examine those
• Other considerations
  • this mentions the isolcpus kernel boot parameter. There's nothign we could do about this for Docker/Kubernetes installations, but for the ISOs maybe it's something we want to look at

IP Defrag

• Settings are found here. I'm not sure if our default settings are recommended or not, although they match the example in the documentation.

Flow and Stream handling

• Flow settings are found here.
  • We may want to add options for emergency_recovery and prune_flows.
  • We are not doing anything with Flow Time-Outs
• Stream setings are found here
  • We may want to add an option for memcap-policy

Defaults

• We should look at all default values here and confirm that they make sense generally for all (or most) Malcolm users. Otherwise the user can override them.