nfsen features / problems for an nfsen user

[candlerb]

Hello,

I have nfsen-ng installed. Here are some things I'm having problems with but was able to do in nfsen. It could be simply due to operator error and I can't work out how to do it.

Single time bin queries

In nfsen, it was easy to move the slider to point to any single 5-minute "bin" - e.g. where there's a nice peak - and then do a query below it.

I can't see how to do this in nfsen-ng. There is a very coarse slider at the top (3 year period), and given the graph below there's a small zoom slider below that, which I can zoom on in stages. But changing the lower zoom slider doesn't carry forward when I switch from "Graphs" to "Flows".

On the "Graphs" page there's a button "Copy from graph", but it's greyed out. When I switch to "Flows" this button vanishes completely.

Range selection

Related to the above: in the "Flows" page there's only the coarse slider at the top. I can switch to "24 hours" and move back and forth one day at a time using the arrows; but I can't see how to say, for example, "I'm only interested in 21:10 to 22:30"

In nfsen, I could just split the time indicator, and move the start and end positions to 21:10 and 22:30, and queries would cover that range.

Time period display

This is more just an observation than a problem.

In the Graph page, it took me a while to work out how to zoom in far enough: I had to first click the "24 hour" button at the top.

Here is the peak as seen by nfsen-ng, corresponding to the peak at 18:35 of 34.2 Mbps in the nfsen screenshot earlier:

It shows as 4.2MB/sec (which is the same). However I don't really like the "ramp up, ramp down / join-the-dots" line; I think that nfsen is more accurate by showing it as a bar, covering the time period ("the average value from 18:30 to 18:35 was X") rather than a point ("the value at 18:35 was X").

Having said that, nfsen is not always precise with start/end times either. It displays "t(start) 18:35, t(end) 18:35" when I think it means "t(start) 18:30, t(end) 18:35".

Translated source/destination

In nfsen I can view NAT-outside (translated) source and destination addresses; I can't see them in nfsen-ng.

** nfdump -M /var/nfsen/profiles-data/live/gw  -T  -r 2019/11/14/nfcapd.201911141835 -c 20
nfdump filter:
any
Date first seen          Event  XEvent Proto      Src IP Addr:Port          Dst IP Addr:Port     X-Src IP Addr:Port        X-Dst IP Addr:Port   In Byte Out Byte
2019-11-14 18:34:44.130 INVALID  Ignore TCP     128.223.157.25:443   ->   XX.XXX.XXX.XXX:47408   128.223.157.25:443   ->      10.12.253.2:47408    13629        0
2019-11-14 18:34:44.440 INVALID  Ignore TCP      10.12.254.107:40183 ->    52.25.210.146:443     81.174.145.162:40183 ->    52.25.210.146:443        181        0
2019-11-14 18:34:44.690 INVALID  Ignore TCP       13.107.18.11:443   ->   XX.XXX.XXX.XXX:60210     13.107.18.11:443   ->    10.12.254.123:60210       88        0
2019-11-14 18:34:44.690 INVALID  Ignore TCP      10.12.254.123:60210 ->     13.107.18.11:443     XX.XXX.XXX.XXX:60210 ->     13.107.18.11:443         40        0

Aside: nfsen doesn't let you aggregate on those fields, unless you patch the PHP slightly:

root@nfsen:/var/www/html/nfsen# diff -u details.php.orig details.php
--- details.php.orig	2018-06-24 16:07:11.246513005 +0000
+++ details.php	2018-06-25 09:01:51.294234237 +0000
@@ -27,7 +27,9 @@

 $IPStatOption = array ( 'Flow Records',
 						'Any IP Address', 'SRC IP Address', 'DST IP Address',
+						'Xlat SRC IP Address', 'Xlat DST IP Address',
 						'Any Port', 'SRC Port', 'DST Port',
+						'Xlat SRC Port', 'Xlat DST Port',
 						'Any interface', 'IN interface', 'OUT interface',
 						'Any AS',  'SRC AS',   'DST AS',
 						'Next Hop IP', 'Next Hop BGP IP', 'Router IP',
@@ -42,7 +44,9 @@

 $IPStatArg	  = array ( '-s record',
 						'-s ip',   '-s srcip',   '-s dstip',
+						'-s xsrcip','-s xdstip',
 						'-s port', '-s srcport', '-s dstport',
+						'-s xsrcport', '-s xdstport',
 						'-s if',   '-s inif',    '-s outif',
 						'-s as',   '-s srcas',   '-s dstas',
 						'-s nhip', '-s nhbip', 	 '-s router',

Minor issues

1. I was able to get error "Got 400 - Bad Request. Expected type int for datestart" when playing the the zoom slider at the bottom of the Graphs page, especially zooming in towards the latest data (i.e. current time).

2. If I go to the Flows page and select IP Aggregation - Destination - IPv4 subnets, then it looks like this:

The display suggests if I do nothing more, aggregation will be at /24 boundaries. But if I leave the /24 at default, I get an error in the nfdump command:

Got 503 - Service unavailable. NfDump: Initialization failed. /usr/local/bin/nfdump -M '/var/nfsen/profiles-data/live/gw' -R '2019/10/24/nfcapd.201910241950:2019/10/25/nfcapd.201910251940' -c '20' -o 'csv' -a '-Adstip4' 2>&1

Running this command manually shows the actual error:

'dstip4' needs subnet bits too aggregate

Also I can't enter "/24" in this field, it has to be "24" only. Once that's done, it works.

3. The format 1'649.16 for duration is unusual. Is this 1649.16 seconds? I guess so. (To me it looks like 1 minute 649.16 seconds, but that wouldn't make sense).

[Dona21]

Hello Brian,

Thanks you very much for opening this new issue. As per today, nfsen-ng is not a 1 to 1 replacement for nfsen. There are some features that are not implemented. Concerning your questions / comments:

Single time bin queries

You will notice that when you select a predefined time slot, the button “Copy from graph” is disabled. If you use the zoom to isolate a peak you can then click this button and it will carry forward when you switch from "Graphs" to "Flows" or “Statistics”.

Range selection

Related to the above, if you need a smaller range than 24 hours, you can define your own slot manually by using the zoom and this will become the new time slot used by the two arrows to move back and forward. Your queries will then cover that range. However, for smaller slots than 24 hours, you will lose the visibility of that range under “Flows” and “Statistics”.

Translated source/destination

Not sure I understood this point clearly. I assume we are talking about the output showed in the “Flows” and “Statistics”. Is that correct? The output there needs to be reworked. As far as I remember it’s not flexible enough and we did not implement it as we initially wanted to (lack of time). I do not have a running version of nfsen on hand, does this NAT translation happen automatically or do you have to specify it somewhere?

Minor issues

Thanks for pointing them out.

@mbolli : do you have some time available soon ? We could meet and try to solve some open/reported points ?

[candlerb]

You will notice that when you select a predefined time slot, the button “Copy from graph” is disabled. If you use the zoom to isolate a peak you can then click this button and it will carry forward when you switch from "Graphs" to "Flows" or “Statistics”.

Thank you - I was being dense. I didn't notice that the "Copy from graph" button became active after moving the zoom slider.

If I have the 24 hour view open, then I have to move the zoom slider a little to activate "Copy from graph". But it seems if I don't do that, then the same time period is used in the Flows and Statistics pages, so that's fine (now I realise that's the case).

Translated source/destination

Not sure I understood this point clearly.

These are additional fields generated in the netflow data from certain devices, such as firewalls (e.g. Cisco ASA) and some routers which do NAT. The examples I showed were from a Mikrotik router.

Digging further: I built nfdump with --enable-nsel configure flag, and I think this may be a pre-requisite to get these fields. They appear in -o nsel (which is the default when nfdump is built that way), but not in -o line, -o long or -o extended. Also unfortunately not with -o csv either, which is what nfsen-ng uses.

They do appear in -o json though: here's an example.

{
	"type" : "FLOW",
	"sampled" : 0,
	"export_sysid" : 1,
	"t_first" : "2019-11-13T18:44:33.720",
	"t_last" : "2019-11-13T18:44:46.770",
	"proto" : 6,
	"src4_addr" : "216.58.208.110",
	"dst4_addr" : "XX.XXX.XXX.XXX",
	"src_port" : 443,
	"dst_port" : 52979,
	"fwd_status" : 0,
	"tcp_flags" : ".A..S.",
	"src_tos" : 0,
	"in_packets" : 31,
	"in_bytes" : 5950,
	"input_snmp" : 17,
	"output_snmp" : 10,
	"src_xlt_port" : "443",
	"dst_xlt_port" : "52979",
	"src4_xlt_ip" : "216.58.208.110",
	"dst4_xlt_ip" : "10.12.254.120",
	"label" : "<none>"
}

This is probably better than writing a custom fmt:<format> to get them.

You'll also see them described in man nfdump under -A and -s:

            xsrcip     X-late source IP address, if compiled with NSEL support
            xdstip     X-late destination IP address, if compiled with NSEL support
            xsrcport   X-late source port, if compiled with NSEL support
            xdstport   X-late destination port, if compiled with NSEL support

---

Aside: the Cisco ASA is a more difficult beast again, because the flows it generates are bidirectional: they have in bytes and out bytes as separate columns in the same flow. Due to a quirk of nfdump, the "in bytes" are bytes from src to destination (which most people would think of as "outbound"), and "out bytes" are bytes from destination to src. I do have to deal with ASA sometimes, and it's a pain: because the definition of "src" or "dst" depends on whether the flow originated from the inside or outside, aggregating on src or dst becomes very difficult. (That is: in some flows my internal addresses are "src", and in other flows they are "dst").

---

One other minor point. When I do Statistics > Statistic for (Any IP Address) > Order by (Byte), the total bytes is not shown as a a column, although I would find it useful. The columns I get are:

Start Time - first seen	End Time - last seen	Duration	Protocol	ip/bytes	Flows	Flows (%)

(where ip/bytes shows IP address only). But the command nfsen-ng sends is:

/usr/local/bin/nfdump -M '/var/nfsen/profiles-data/live/gw' -R '2019/11/13/nfcapd.201911131845:2019/11/14/nfcapd.201911141835' -n '10' -o 'csv' -s 'ip/bytes'

and the CSV it returns includes additional columns like packets/packets% and bytes/bytes%:

ts,te,td,pr,val,fl,flP,pkt,pktP,byt,bytP,pps,bps,bpp

I guess this was done to reduce screen clutter, but it would be nice to be able to turn on the additional columns (if there is already a way, I can't see it). I see there are a series of checkboxes under the "Search" dropdown, but they are all turned on already.

[wimpie007]

Translated source/destination
is a showstopper for us unfortunately... NAT translations are crucial.
Unfortunately, because ik like the interface more then the old nfsen...