Slims down upgrade testing logic

[conorsch]

Status

Ready for review

Description of Changes

Resolves a few issues:

• Closes Support in-place upgrades for Focal VMs #5512
• Closes Upgrade scenario is broken #5883
• Closes [xenial] Deprecate Vagrantfile in favor of Molecule scenarios #3208

Changes proposed in this pull request:

Makes a few changes to refresh the upgrade testing logic for
compatibility with Focal VMs. Specifically:

• Prod VMs are now configured via Molecule
• Removes all use of custom-built Vagrant boxes
• Clarifies upgrade steps (see also related docs PR)

Docs will be updated separately.

Testing

The docs PR at freedomofpress/securedrop-docs#227 should be reviewed in tandem with this PR. Check those docs and make sure the procedures there work for the use case of SD pre-release QA. At a high-level, those steps are:

0. Make sure you're using libvirt-based VMs (upgrade scenario does not support Qubes env)
1. molecule create -s libvirt-prod-focal
2. Boot up admin workstation and install against those prod VMs with ./securedrop-admin install
3. make build-debs on host (ok to run this in parallel with step 2 to save time)
4. make upgrade-start on host, to set up local apt repo
5. Back in admin workstation, run the playbook securedrop-apt-local.yml (make sure to source the admin venv first, see docs)
6. Confirm you can upgrade packages inside the VMs

Deployment

Dev-only.

[zenmonkeykstop]

• molecule prod VM scenario works great
• make upgrade-start also good, no issues setting up apt server
• Repeated fails on the securedrop-apt-local.yml playbook, failing to add the 10.0.1.7 repo
• once repo added, sudo unattended-upgrades -d works fine, upgrading packages to local versions.

This is a good simplification overall and spares the effort of maintaining the upgrade boxes - there is some flakiness on my system wrt. the playbook Add local repo step, which occasionally fails for one or both servers with an "apt cache update failed" message (this is existing code tho so not the fault of this PR).

Holding off on approving to poke at that playbook error but otherwise this looks OK to go.

[conorsch]

Add local repo step, which occasionally fails for one or both servers with an "apt cache update failed" message

Might be more straightforward to debug if you break that up into several tasks, like a copy to write the repo, then a separate task to update apt lists afterward. I've seen that error before while testing, but it was inconsistent, and I don't have a better suggestion for debugging than separating the steps involved.

[zenmonkeykstop]

Context for the many fun and interesting ways in which this can fail both deterministically and non- : ansible/ansible#30754

[zenmonkeykstop]

Can't reproduce the apt cache update failure on two fresh installs, calling it a flake and merging.

[zenmonkeykstop]

Approved, LGTM as apt cache issue is unrelated to changes.