Store audit events in PG

[trstephen]

There's a lot going on here! It's a big commit so here's an overview

Needs DistributedDesigns/worker#20 to send AuditEvents

Architecture

• AuditEvents come in over the audit_event queue. They're immediately stored in Redis.
  • Events come in at the same speed as our TPS (17.8k). Under the old architecture, writing these to disk resulted in a backlog of >800k message in RMQ and it was likely to crash. Adding PG into the mix slows the consumption rate even further and then 10 user run (10k tx) would have ~3k messages backlogged in RMQ.
  • A configurable (default 5) number of workers takes messages out of RMQ and puts them directly into Redis. This takes keeps the peak RMQ backlog <3k during the 1000 user run 👌
• More workers are responsible for clearing Redis and inserting into PG.
  • I use the same RPUSH / BLPOP pattern that's used in the worker.
  • Logs get in to PG eventually.
• Another worker watches the dumplog queue and queries PG for the user's logs. The results are saved to a unique file.
  • Ideally, we don't want to have to return this file name to the worker via RPC. This is possible if the worker and audit logger can know the filename ahead of time (which I'm complicating by trying to avoid collisions). I expect this to naming convention to change once we know how to pass the actual file / URI to the FE user.
  • Writing logs to unique files makes it easier to develop since I don't have to worry about overwriting my past attempts.

Quotes

Are sniffed the same was as before but instead of assembling the XML entry and writing that to a file it converts the Quote to AuditEntry and saves that to PG.

Docker compose

This is the first repo that uses it. It's an okay system! (And the only one that allows >1 environment variable to be passed to the postgres image...)

Initializing the DB

The docker-compose file creates a user and db for us via environment variables. It also runs auditInit.sql on db start.

TODO

• This runs on prod but this long standing issue with docker and CentOS means that the VM quickly runs out of disk space. I think this problem will happen whenever we do a lot of writes to disk so we'll have to deal with this problem if we want persistence. (RMQ and Redis both keep everything in memory which is why this problem just came up).

[trstephen]

Deleted a bunch of files on the prod VM and things seem okay 😖