Pause and resume data ingest

[poundifdef]

Today, when data is queued, we immediately consume it and upload to the database. However, we want the ability to pause and resume workers for a specific table, or for all tables in a destination. This is useful because sometimes we perform database migrations and need to pause one table.

Step 1: One queue per table

• Instead of a single queue for all data, we should have a separate queue for each table where we insert data
• If the queue doesn't exist, we should automatically create it
• You'll need to refactor the Queue interface.
• Workers will need to be refactored to consume from multiple queues

This doesn't have to be done with multiple queues if there are other ways (ie, message groups) that would work.

Step 2: API Endpoint to pause queue for single table and entire destination

• We want an API endpoint to let us pause+unpause the queue for a single table
• That endpoint should also let us pause+unpause the queue for a destination (which will have the effect of pausing all tables)
• This endpoint will be authenticated
• The pause status should be stored in the Database. You'll want to add methods to that interface.

Step 3: Workers should respect the pause status

• Now that the workers understand how to use multiple queues, and the database stores the pause status for each destination and table, workers should respect the pause status and only consume from queues that are active.