fix(ossec): run "disconnected submissions" checks from systemd timers; report on saved output

[cfm]

Status

Ready for review

Description of Changes

Fixes #6982 by preventing OSSEC from having to sudo to run manage.py check-disconnected-{db,fs}-submissions directly, which adds noise to /var/log/auth.log and thus OSSEC's own daily "Successful Logins" report. Instead, like manage.py were-there-submissions-today, these commands now:

1. are defined in systemd services,
2. are triggered by systemd timers 23 hours after boot; and
3. write their standard output to files under /var/lib/securedrop/;
4. which OSSEC monitors and alerts on changes.

Testing

1. Install SecureDrop onto production VMs (e.g.).
2. Set up OSSEC alerts.
3. make build-debs on this branch.
4. Install them:
   • Application Server:
     • securedrop-app-code
     • securedrop-ossec-agent
   • Monitor Server:
     • securedrop-ossec-server
5. Submit at least two submissions.
6. SSH into the Application Server and start a sudo -s session. (If you accidentally exit it between here and step the end of step 10, you'll have to factor this into your count of authentication_success entries. Or you can start over from here.)
7. SSH into the Monitor Server and start a sudo -s session (If you accidentally exit it between here and step the end of step 10, you'll have to factor this into your count of authentication_success entries. Or you can start over from here.)
8. Do the following dance, in order:

	Server	Command	Significant output
8.1	Monitor	cat /var/ossec/logs/alerts/alerts.log | /var/ossec/bin/ossec-reportd -f group authentication_success	Make a note of the count of authentication_success entries.
8.2	Application	systemctl start securedrop-check-disconnected-db-submissions.service
8.3	Monitor	cat /var/ossec/logs/alerts/alerts.log | /var/ossec/bin/ossec-reportd -f rule 400800	Report completed and zero alerts post-filter.
8.4	Application	systemctl start securedrop-check-disconnected-fs-submissions.service
8.5	Monitor	cat /var/ossec/logs/alerts/alerts.log | /var/ossec/bin/ossec-reportd -f rule 400801	Report completed and zero alerts post-filter.
8.6	Monitor	cat /var/ossec/logs/alerts/alerts.log | /var/ossec/bin/ossec-reportd -f group authentication_success	You should have the same number of authentication_success entries as in step 8.1.

9. On the Application Server, delete one of the submissions from the database (via sqlite3, leaving it on the filesystem) and another from the filesystem (via rm, leaving it in the database).
10. Do the following dance, in order:

	Server	Command	Significant output
10.1	Monitor	cat /var/ossec/logs/alerts/alerts.log | /var/ossec/bin/ossec-reportd -f group authentication_success	Make a note of the count of authentication_success entries.
10.2	Application	systemctl start securedrop-check-disconnected-db-submissions.service
10.3	Monitor	cat /var/ossec/logs/alerts/alerts.log | /var/ossec/bin/ossec-reportd -f rule 400800	1 match for rule 400800.
10.4	Application	systemctl start securedrop-check-disconnected-fs-submissions.service
10.5	Monitor	cat /var/ossec/logs/alerts/alerts.log | /var/ossec/bin/ossec-reportd -f rule 400801	1 match for rule 400801.
10.6	Monitor	cat /var/ossec/logs/alerts/alerts.log | /var/ossec/bin/ossec-reportd -f group authentication_success	You should have the same number of authentication_success entries as in step 10.1.

11. At the end of the day, you should receive level-1 reports for rules 400800 and 400801.
12. Bonus: The next day, after you have not logged into the servers, you should not receive a Daily report: Successful logins report.

Deployment

After #6780, upgrades should get the new systemd units automatically when secuerdrop-app-codes is upgraded. We should check that the new ossec-agent ossec.conf and ossec-server local_rules.xml are also applied on upgrade.

Checklist

If you made changes to the system configuration:

• Configuration tests pass

If you made non-trivial code changes:

• I have written a test plan and validated it for this PR

Choose one of the following:

• I have opened a PR in the docs repo for these changes, or will do so later
• I would appreciate help with the documentation
• These changes do not require documentation

[legoktm]

Minor nitpick, otherwise code LGTM. Running through the test plan now...

[legoktm]

Step 8 of the test plan worked properly, but I'm running into issues on step 10.

On my app server:

root@app:/var/lib/securedrop# systemctl start securedrop-check-disconnected-db-submissions.service
root@app:/var/lib/securedrop# ls
backups  db.sqlite  disconnected_db_submissions.txt  disconnected_fs_submissions.txt  journalist.pub  keys  SecureDrop.asc  shredder  source_v3_url  store  submissions_today.txt  tmp
root@app:/var/lib/securedrop# cat disconnected_db_submissions.txt
There are submissions in the database with no corresponding files. Run "manage.py list-disconnected-db-submissions" for details.
root@app:/var/lib/securedrop# systemctl start securedrop-check-disconnected-fs-submissions.service
root@app:/var/lib/securedrop# cat disconnected_fs_submissions.txt
There are files in the submission area with no corresponding records in the database. Run "manage.py list-disconnected-fs-submissions" for details.

but on my mon server no rules match:

root@mon:/home/sdadmin# cat /var/ossec/logs/alerts/alerts.log | /var/ossec/bin/ossec-reportd -f rule 400800
2023/10/18 02:47:22 ossec-reportd: INFO: Started (pid: 268630).
2023/10/18 02:47:27 ossec-reportd: INFO: Report completed and zero alerts post-filter.
root@mon:/home/sdadmin# cat /var/ossec/logs/alerts/alerts.log | /var/ossec/bin/ossec-reportd -f rule 400801
2023/10/18 02:48:04 ossec-reportd: INFO: Started (pid: 268674).
2023/10/18 02:48:09 ossec-reportd: INFO: Report completed and zero alerts post-filter.

[legoktm]

OK but I just checked my email and a little over an hour after I posted that comment I got the following OSSEC alert in my inbox:

OSSEC HIDS Notification.
2023 Oct 18 04:00:49

Received From: (app) 10.20.2.2->cat /var/lib/securedrop/disconnected_db_submissions.txt
Rule: 400800 fired (level 1) -> "Indicates that submissions in the database are missing their corresponding files."
Portion of the log(s):

ossec: output: 'cat /var/lib/securedrop/disconnected_db_submissions.txt': There are submissions in the database with no corresponding files. Run "manage.py list-disconnected-db-submissions" for details.



 --END OF NOTIFICATION

[cfm]

Oops; that's my omission in the test plan, @legoktm. The systemctl invocation writes the file, and the ossec-reportd invocation reports on the check of the file. But the check itself is still scheduled by:

securedrop/securedrop/debian/ossec-agent/var/ossec/etc/ossec.conf

Line 134 in 154b0bc

<frequency>90000</frequency>

Apply something like...

--- a/securedrop/debian/ossec-agent/var/ossec/etc/ossec.conf
+++ b/securedrop/debian/ossec-agent/var/ossec/etc/ossec.conf
@@ -131,13 +131,13 @@
   <localfile>
     <log_format>command</log_format>
     <command>cat /var/lib/securedrop/disconnected_db_submissions.txt</command>
-    <frequency>90000</frequency>
+    <frequency>60</frequency>
   </localfile>
 
   <localfile>
     <log_format>command</log_format>
     <command>cat /var/lib/securedrop/disconnected_fs_submissions.txt</command>
-    <frequency>90000</frequency>
+    <frequency>60</frequency>
   </localfile>
 
   <localfile>

...and do...

root@app:/home/sdadmin# /var/ossec/bin/ossec-control reload

...and then you should be able to ossec-reportd as expected, with a Last report value always in the preceding minute.

[legoktm]

Perfect! And it definitely worked because I started getting 2 OSSEC alerts every minute :)

[legoktm]

This is great, test plan checked out too. Will merge once staging build passes!