nProbe IPS: Custom Protocols and Categories not getting blocked (bug?)

[l0crian1]

I am evaluating nProbe in IPS mode and have blocking based on predefined values working correctly. I am trying to now get blocking working on custom protocol and category lists. I'm not sure if it's a bug or some quirk on how to format the files differs from other ntop documentation. Can simple examples of formats the IPS rules file is expecting for the custom_protocols and category_file be provided.

I am running nProbe in a docker, here is the arguments in use for the docker:

-i nf:25 --ips-mode /data/nprobe/ips-config/ips-rules.conf -n none -b 1 --ndpi-custom-protos /data/nprobe/proto.txt

Here is the version:

Welcome to nProbe v.10.0.230103 for x86_64-pc-linux-gnu
with native PF_RING acceleration.
Built with nDPI 4.4.0-3746-299fc4d1

Here is my rules config:

### Category files ###
{ "category_file": "/data/nprobe/nfw_malware_list.txt" }

### Custom protocols definition ###
{ "custom_protocols": "/data/nprobe/proto.txt" }

# Pool definition
{"pool":{"id":1,"name":"User Networks","ip": [ "192.168.2.0/24","10.0.95.0/24" ], "mac": []},"policy": {"id": 1} }

# Policy definition
{"policy":{"id":0,"name":"Default Rule", "default_marker": "pass", "markers": { "categories": { "Malware": "drop" } } } }
{"policy":{"id":1, "name":"Drop Users", "default_marker": "pass", "markers": { "categories": { "Malware": "drop" }, "protocols": { "Twitch": "drop", "CustomProtocolA": "drop"} } } }

Here's my protocols list:

udp:9993@CustomProtocolA

Here's my categories list:

zerotier.com    100

I also tried to call that categories list using the blacklist format for ntopng, and then putting that file in my rules config, to no success either:

{"name":"ZeroTier List","format":"hosts","enabled":true,"update_interval":86400,"url":"/data/nprobe/nfw_malware_list.txt","category":"malware"}

Here is some output from the log:

06/May/2024 10:00:19 [RuleManager.cpp:54] [line 2] Loading { "category_file": "/data/nprobe/nfw_malware_list.txt" }
06/May/2024 10:00:19 [RuleManager.cpp:54] [line 5] Loading { "custom_protocols": "/data/nprobe/proto.txt" }
06/May/2024 10:00:19 [RuleManager.cpp:54] [line 9] Loading {"pool":{"id":1,"name":"User Networks","ip": [ "192.168.2.0/24","10.0.95.0/24" ], "mac": []},"policy": {"id": 1} }
06/May/2024 10:00:19 [RuleManager.cpp:54] [line 12] Loading {"policy":{"id":0,"name":"Default Rule", "default_marker": "pass", "markers": { "categories": { "Malware": "drop" } } } }
06/May/2024 10:00:19 [RuleManager.cpp:54] [line 13] Loading {"policy":{"id":1, "name":"Drop Facebook", "default_marker": "pass", "markers": { "categories": { "Malware": "drop" }, "protocols": { "Twitch": "drop", "CustomProtocolA": "drop"} } } }
06/May/2024 10:00:19 [ips.c:116] Loaded IPS rules from /data/nprobe/ips-config/ips-rules.conf
06/May/2024 10:00:19 [nfq.c:119] Successfully connected to NF_QUEUE 25
06/May/2024 10:00:19 [nprobe.c:11395] Capturing packets from interface nf:25 [snaplen: 16384 bytes]
06/May/2024 10:00:19 [nprobe.c:10300] Loading nDPI custom protocols from /data/nprobe/proto.txt
06/May/2024 10:00:20 [nprobe.c:4114] ---------------------------------
06/May/2024 10:00:20 [nprobe.c:4117] Average traffic: [316.00 pps][All Traffic 2.01 Mb/sec][IP Traffic 1.95 Mb/sec][ratio 0.97]
06/May/2024 10:00:20 [nprobe.c:4125] Current traffic: [316.00 pps][2.01 Mb/sec]
06/May/2024 10:00:20 [nprobe.c:4133] L7 Proto                   Diff      Total
06/May/2024 10:00:20 [nprobe.c:4147]    CustomProtocolA/305       859 B      859 B

Blocking for predefined applications and categories works fine:

C:\Users>curl -m 5 -I www.twitch.tv
curl: (28) Resolving timed out after 5014 milliseconds

But anything in my custom files is not getting blocked.

[l0crian1]

After some further testing, blocks against custom protocols work if they're hosts. 'ip', 'tcp/udp', and 'nbpf' do not work, though they do show as being identified in the log (except 'nbpf').

proto.txt:

nbpf:"host 103.195.103.66 and proto 17"@TEST1
ip:103.195.103.66@TEST2
ip:128.223.51.103@TEST3
tcp:22@TEST3
host:"youtube.com"@TEST4

Matches in log:

06/May/2024 16:00:25 [nprobe.c:4133] L7 Proto                   Diff      Total
06/May/2024 16:00:25 [nprobe.c:4147]    TEST2/306                 120 B      120 B
06/May/2024 16:00:25 [nprobe.c:4147]    TEST3/307             127.61 KB  250.33 KB
06/May/2024 16:00:25 [nprobe.c:4147]    TEST4/308              14.97 KB   34.97 KB

If I put the logging to verbose, it appears that the matches against 'ip' and 'tcp' have a Marker of '0' instead of '2':

06/May/2024 17:20:21 [engine.c:3707] Emitting Flow: [->][unknown] 192.168.2.88:9993 -> 103.195.103.66:9993 [98 pkt/6056 bytes][ifIdx 0->0][0.0 sec][TEST2/306][init Unknown][AS: 0 -> 23470][IPS Marker: 0]
06/May/2024 17:20:18 [engine.c:3707] Emitting Flow: [->][unknown] 128.223.51.103:22 -> 192.168.2.21:8355 [21958 pkt/12187232 bytes][ifIdx 0->0][0.0 sec][TEST3/307][init Unknown][AS: 3582 -> 0][IPS Marker: 0]
06/May/2024 17:21:14 [engine.c:3707] Emitting Flow: [->][unknown] 192.168.2.21:10669 -> 9.9.9.9:53 [11 pkt/487 bytes][ifIdx 0->0][0.0 sec][CNL: 0.257 ms][SNL: 1.487 ms][DNS.TEST4/308][init 192.168.2.21][AS: 0 -> 19281][IPS Marker: 2]

[cardigliano]

This has been fixed, please update and let us know

[l0crian1]

This has been fixed, please update and let us know

I tested this. Working great for UDP/TCP, but if I use IP for a protocol, the engine doesn't match it to the proper custom protocol. It calls it Unknown, but for a custom protocol ID instead of 0:

08/May/2024 14:11:47 [nprobe.c:4422]    Unknown/0             713.92 KB    4.14 MB
08/May/2024 14:11:47 [nprobe.c:4422]    Unknown/414            96.62 MB   96.63 MB

Another issue is even though the log says it reloaded the IPS rules after detecting a change, the newly added rule doesn't actually take effect until I restart the service.