Use Redis as datastore

[vekexasia]

Redis is capable of an outstanding amount of op/secs. Instead of using postgres as single datastore we should put redis in front of postgres ensuring a fair amount of operations per second.

Proposed solution

• Redis being used to perform all simple read/write operations
• Postgres being used as a readonly mechanism for APIs leveraging SQL
• Postgres being used as a slow - sync later - datastore

What needs to be done for this to work

• Implement a sync mechanism
  • When a new block is added to redis it should be added to postgres within a convenient timeframe
  • When a new block is removed it should be removed from pg too.
• Writes should be only performed in a single point of code and only blocks and transactions should be written directly from code. (see below)
• Account updates (balances, votes, etc) should be done using a postgres data trigger.
• When node starts redis must be cleaned and restored using data coming from PG.
• We should calculate the amount of memory needed to store the needed data in memory (Redis) and prompt the node operator if there is not enough ram available.
• In memory data should not expire.
• retry mechanism in case data saving fails.
• automatic process exit if slow-storage is too desyncd from "fast-storage" (Forcing node to rollback to last version in db). NOTE: in sync mode slow-storage could easily be out-of-sync by several blocks. A timestamp mechanism should also be used.
• produce tuned redis.conf

What are the benefits

• Slow I/O separation from main logic.
• postgres will become the ultimate source of truth.
• atomic block saving easier to perform.
• Better TPS value (see Enhance TPS value #121)
• Partial achievement of CQRS.
• Possibility to log all slow-storage write operations for better debugging.

Edit: Not needed right now.