Create scripts to manage creation and configuration of Wikimedia Cloud VMs

[ragesoss]

The Dashboard relies on several VMs running on Wikimedia Cloud:

Programs & Events Dashboard database server

This runs MariaDB, and is configured to provide access to the database from other servers in Wikimedia Cloud. The data itself is stored on an attached Cinder volume.

Programs & Events Dashboard Sidekiq

This runs some sidekiq workers, including the one responsible for the 'long update' queue which processes the most resource-intensive course updates.

Programs & Events Dashboard Web

This runs a lot of things (too many):

• The Passenger process and Apache webserver that handle web requests.
• A number of Sidekiq workers
• The Redis database that powers the Sidekiq job system
• The non-database non-code assets, including profile pictures and the temporary CSV files that are used for data downloads and other things.

Database replica endpoint

The service that makes queries to the Replica database and returns Article and Revision data. (This is used by both Programs & Events Dashboard as well as Wiki Education Dashboard.)

Server replacement

We periodically need to replace old VMs with newer ones running supported up-to-date versions of Debian. Currently, that's very tedious because we have not automated the server setup process or moved all the data to Cinder volumes that can be re-used by new VMs.

We should:

• Move Redis persistence to a Cinder volume
• Create a new (small) VM just for Redis
• Move the non-code non-database assets to a Cinder volume
• Create a new (large) VM for all the Sidekiq workers that currently share a VM with the web service
• Create automation scripts that can efficiently set up fresh VMs for each of the ones we need (web, redis, default sidekiq, large-job sidekiq, database, replica endpoint)