memory usage increase issue

[ganduulgag]

hi,

is there someone who experienced a memory usage increase issue when you run rulecat? if yes, how did you tackle the issue?

in our case, we configured rulecat to run once a day in cron job as we use Security Onion. Whenever the cron job starts, memory usage increases drastically. We suppressed several rules with idstools. As a solution, we added swap space for the moment but we are concerned if this memory usage keeps increasing but we don't want to allocate more swap or memory.

[jasonish]

I just looked at rulecat with the ET/Open ruleset and it did use about 500MB. Not ideal, but it does load, parse and sort all the rules in memory. But this might not be the only thing using memory. rulecat can be configured to test the output with suricata -T. This causes suricata to load the rules in memory to check for sanity which can use even more memory. This is somewhat of a known issue, but not much can be done about it if you want to validate the rules before restarted Suricata.

[ganduulgag]

where can I configure to test output with suricata -T? Currently, I am using rulecat with ETPro rules and when I run the rulecat, memory usage increases about 3GB which is quite a lot memory for the system. is there a way I can reduce memory usage? besides, it takes roughly 3 min to finish the rulecat. Here is my console output after running rulecat:

2022-09-07 04:05:02,669 - -- Loading ./rulecat.conf.
2022-09-07 04:05:02,682 - -- Forcing Suricata version to 6.0.
2022-09-07 04:05:02,695 - -- Fetching https://urlhaus.abuse.ch/downloads/urlhaus_suricata.tar.gz.
100% - 741376/741376
2022-09-07 04:05:02,934 - -- Done.
2022-09-07 04:05:03,126 - -- Fetching https://rules.emergingthreatspro.com/0148694801847852/suricata-6.0.0/etpro.rules.tar.gz.
100% - 8922965/8922965
2022-09-07 04:05:04,836 - -- Done.
2022-09-07 04:05:05,259 - -- Fetching https://sslbl.abuse.ch/blacklist/sslipblacklist.rules.
100% - 14294/14294
2022-09-07 04:05:05,401 - -- Done.
2022-09-07 04:05:05,403 - -- Fetching https://threatfox.abuse.ch/downloads/threatfox_suricata.tar.gz.
100% - 1073155/1073155
2022-09-07 04:05:05,908 - -- Done.
2022-09-07 04:05:06,098 - -- Ignoring file rules/deleted.rules
2022-09-07 04:05:57,472 - -- Loaded 155162 rules.
2022-09-07 04:08:32,612 - -- Disabled 1233 rules.
2022-09-07 04:08:32,612 - -- Enabled 0 rules.
2022-09-07 04:08:32,612 - -- Modified 19831 rules.
2022-09-07 04:08:32,612 - -- Dropped 0 rules.
2022-09-07 04:08:35,223 - -- Enabled 184 rules for flowbit dependencies.
2022-09-07 04:09:10,379 - -- Writing rules to /home/kali/all.rules: total: 155162; enabled: 137950; added: 13435; removed 11563; modified: 1359
2022-09-07 04:09:13,878 - -- Done

[jasonish]

There is a command line option, --test-command that you can then provide a command to run the test. Something like --test-command "suricata -T" might work if using all the defaults.

If you are only using this tool for Suricata, you should really look at suricata-update which is bundled with Suricata these days. Its had some work to reduce memory usage over time, and runs suricata -T by default.