rita import issue when DNS is turned on

[devdroid0]

originally having an issue with messages in the output from rita regarding dns:
not found {map[domain:mcafee.com] map[$inc:map[subdomain_count:-1]] explodedDns}
Get lots of messages like the one above just different domains … in the thousands
FNAME FDATE ERRCNT LNCNT NONERR
testag_43.out 2022-02-15 497194 503312 6118

I've engaged Active Countermeasures support. As a result DNS was enabled in the rita config.yml config file. the run went from just a few hours to not completing after about 26 hours and crashing the instance it was running on.

We are running Rita 4.2.0
memory and cpu stats durint current run of rita:
top - 16:09:52 up 21:24, 2 users, load average: 3.22, 3.06, 3.06
Tasks: 262 total, 1 running, 153 sleeping, 0 stopped, 0 zombie
%Cpu0 : 13.6 us, 0.3 sy, 0.0 ni, 86.1 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu1 : 14.9 us, 0.3 sy, 0.0 ni, 84.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu2 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu3 : 9.9 us, 0.0 sy, 0.0 ni, 90.1 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu4 : 9.7 us, 0.0 sy, 0.0 ni, 90.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu5 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu6 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu7 : 21.9 us, 0.3 sy, 0.0 ni, 77.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu8 : 0.0 us, 0.3 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu9 : 21.9 us, 0.3 sy, 0.0 ni, 77.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu10 : 11.3 us, 0.0 sy, 0.0 ni, 88.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu11 : 0.3 us, 0.0 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 65968436 total, 11508076 free, 13857284 used, 40603076 buff/cache
KiB Swap: 15728636 total, 15728636 free, 0 used. 58506964 avail Mem

there were no errors posted when it crashed. just the last date/time anything was posted to file.

Host OS: Ubuntu 18.04

summary of the zeek files being processed:
num host size
236 hostname1 44.4G
256 hostname2 36.9G
178 hostname3 9.0G
212 hostname4 45.7G
208 hostname5 3.6G
193 hostname6 9.9G

prior to making DNS true, a rita run for one day was about 3 to 5 hours depending on how busy the previous day was

This is the command we are using:
/usr/local/bin/rita import --rolling --numchunks 7 --chunk ${chunk} ${idir}/${dt} ${dbname} > ${ofile}
chunk is based on the previous day's day value

I've attached the names of the files that are being used for this run.
var_pulse_logs.txt

mongo log entries that may be useful:

2022-03-03T00:11:23.184+0000 I WRITE [conn857] warning: log line attempted (689kB) over max size (10kB) (lots of these)

last successful entry:
2022-03-03T01:35:26.679+0000 I COMMAND [conn867] command rita_intel_dataset.hos
t command: find { find: "host", filter: { blacklisted: true }, skip: 0, noCursor
Timeout: true, $db: "rita_intel_dataset" } planSummary: COLLSCAN keysExamined:0
docsExamined:50683 cursorExhausted:1 numYields:415 nreturned:0 reslen:111 locks:
{ Global: { acquireCount: { r: 832 } }, Database: { acquireCount: { r: 416 } },
Collection: { acquireCount: { r: 416 } } } protocol:op_query 547ms

2022-03-03T03:01:45.411+0000 I COMMAND [PeriodicTaskRunner] task: UnusedLockCleaner took: 436ms
then nothing until the the server was restarted:
2022-03-03T13:54:58.956+0000 I CONTROL [main] ***** SERVER RESTARTED *****

There are no filesystems out of space.

[devdroid0]

The original dropped of the stars in the input path:
/usr/local/bin/rita import --rolling --numchunks 7 --chunk ${chunk} ${idir}/*${dt}* ${dbname} > ${ofile}

[devdroid0]

Not sure this is relevant, but the input files are symbolic links to their zeek/bro source directory because there are 6 hosts that are being aggregated for input to rita. if there is a better way to do it please let us know.

[devdroid0]

INFO: 2022-03-04 19:36:35: RITA completed
elapsed time: 13:31:34