From 8c7bba47c14c5cd8c20e4877ba4812ac86ac4eaf Mon Sep 17 00:00:00 2001 From: Conor Schaefer Date: Tue, 4 May 2021 16:23:57 -0700 Subject: [PATCH] Removes VirtualBox references throughout docs Supplements https://github.com/freedomofpress/securedrop/pull/5922. Edits the dev docs to remove explicit mentions of Virtualbox, preferring Libvirt (or Qubes) for any of the VM docs. Mostly this affects the macOS docs. For macOS, we're now more clearly directing folks to use the container dev env. --- docs/development/setup_development.rst | 50 +---- docs/development/upgrade_testing.rst | 7 +- docs/development/virtual_environments.rst | 49 +---- docs/development/virtualizing_tails.rst | 239 +--------------------- 4 files changed, 11 insertions(+), 334 deletions(-) diff --git a/docs/development/setup_development.rst b/docs/development/setup_development.rst index 8293b0ab7..0ed9c02fd 100644 --- a/docs/development/setup_development.rst +++ b/docs/development/setup_development.rst @@ -8,12 +8,10 @@ Overview SecureDrop is a multi-machine design. To make development and testing easy, we provide a set of virtual environments, each tailored for a -specific type of development task. We use Vagrant, VirtualBox, and +specific type of development task. We use Vagrant, Molecule, and Docker and our Ansible playbooks can provision these environments on either virtual machines or physical hardware. -.. note:: SecureDrop is written in Python 3 only. - Quick Start ----------- @@ -188,7 +186,7 @@ Setting Up a Multi-Machine Environment .. note:: You do not need this step if you only plan to work on the web application or the documentation. -To get started, you will need to install Vagrant, VirtualBox, Docker, and +To get started, you will need to install Vagrant, Libvirt, Docker, and Ansible on your development workstation. @@ -201,7 +199,7 @@ Ubuntu or Debian GNU/Linux sudo apt-get update sudo apt-get install -y build-essential libssl-dev libffi-dev python3-dev \ - dpkg-dev git linux-headers-$(uname -r) virtualbox + dpkg-dev git linux-headers-$(uname -r) We recommend using the most recent version of Vagrant available in your distro's package repositories. For Debian Stable, that's ``2.2.3`` at the time @@ -227,9 +225,6 @@ from the `Vagrant Downloads page`_ and then install it. instructions in Vagrantfile that would enable vagrant-cachier are currently commented out. -VirtualBox should be at least version 5.x. See `GitHub #1381`_ for documentation -of incompatibility with the older VirtualBox 4.x release series. - Finally, install Ansible so it can be used with Vagrant to automatically provision VMs. We recommend installing Ansible from PyPi with ``pip`` to ensure you have the latest stable version. @@ -259,41 +254,8 @@ development-related tooling. Using `virtualenvwrapper macOS ~~~~~ -Install the dependencies for the development environment: - -#. Vagrant_ -#. VirtualBox_ -#. Ansible_ -#. rsync >= 3.1.0 - -If you use homebrew-cask_ to manage macOS apps, you can install Vagrant and -VirtualBox that way. As for Ansible, we strongly recommend installing it -in a virtual environment using ``virtualenvwrapper`` and ``pip``, so as not to -install the older version we use system-wide. You should create your virtualenv -using the Python 3 install on your system. If you are using a -different version, the path to ``virtualenvwrapper.sh`` will differ. Running -``pip show virtualenvwrapper`` should help you find it. - -.. code:: sh - - sudo easy_install pip # if you don't already have pip - sudo -H pip install -U virtualenvwrapper --ignore-installed six - source /usr/local/bin/virtualenvwrapper.sh - mkvirtualenv -p python3 securedrop - -.. note:: You'll want to add the command to source ``virtualenvwrapper.sh`` - to your ``~/.bashrc`` (or whatever your default shell configuration - file is) so that the command-line utilities ``virtualenvwrapper`` - provides are automatically available in the future. - -The version of rsync installed by default on macOS is extremely out-of-date, as is Apple's custom. We recommend using Homebrew_ to install a modern version (3.1.0 or greater): ``brew install rsync``. - -.. _Vagrant: https://www.vagrantup.com/downloads -.. _VirtualBox: https://www.virtualbox.org/wiki/Downloads -.. _Ansible: https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html -.. _Homebrew: https://brew.sh/ -.. _homebrew-cask: https://sourabhbajaj.com/mac-setup/Vagrant/README.html - +Developers on macOS should use the Docker-based container environment. +We don't support running VMs on macOS. Fork & Clone the Repository ~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -325,6 +287,6 @@ Qubes To configure a multi-machine evironment in Qubes, follow the Quick Start instructions above to create a standalone VM named ``sd-dev``, then follow the Linux instructions above to install the -required packages, *omitting* Virtualbox. +required packages. Then, complete the steps described in :doc:`qubes_staging`. diff --git a/docs/development/upgrade_testing.rst b/docs/development/upgrade_testing.rst index 3795b70e7..2f5b59155 100644 --- a/docs/development/upgrade_testing.rst +++ b/docs/development/upgrade_testing.rst @@ -19,10 +19,9 @@ You can use this scenario to test the upgrade process, using using either locally-built .debs or packages from the FPF test repo at https://apt-test.freedom.press/. Both options are described below. -.. note:: The upgrade scenario uses QEMU/KVM via Vagrant's libvirt provider, in - place of the default Virtualbox provider. If you haven't already done so, - you'll need to set up the libvirt provider before proceeding. For - more information, see :ref:`libvirt_provider`. +.. note:: The upgrade scenario uses QEMU/KVM via Vagrant's libvirt provider. + If you haven't already done so, you'll need to set up the libvirt provider + before proceeding. For more information, see :ref:`libvirt_provider`. .. _upgrade_testing_local: diff --git a/docs/development/virtual_environments.rst b/docs/development/virtual_environments.rst index 7363d146b..aac86f822 100644 --- a/docs/development/virtual_environments.rst +++ b/docs/development/virtual_environments.rst @@ -15,9 +15,6 @@ one. .. note:: If you plan to alter the configuration of any of these machines, make sure to review the :ref:`config_tests` documentation. -.. note:: If you see test failures due to ``Too many levels of symbolic links`` - and you are using VirtualBox, try restarting VirtualBox. - .. _staging_vms: Staging @@ -57,48 +54,6 @@ To rebuild the local packages for the app code and update the staging VMs: The Debian packages will be rebuilt from the current state of your local git repository and then installed on the staging servers. -.. note:: If you are using macOS and you run into errors from Ansible - such as ``OSError: [Errno 24] Too many open files``, you may need to - increase the maximum number of open files. Some guides online suggest - a procedure to do this that involves booting to recovery mode - and turning off System Integrity Protection (``csrutil disable``). - However this is a critical security feature and should not be - disabled. Instead follow this procedure to increase the file limit. - - Set ``/Library/LaunchDaemons/limit.maxfiles.plist`` to the following: - - .. code:: sh - - - - - - Label - limit.maxfiles - ProgramArguments - - launchctl - limit - maxfiles - 65536 - 65536 - - RunAtLoad - - ServiceIPC - - - - - The plist file should be owned by ``root:wheel``: - - .. code:: sh - - sudo chown root:wheel /Library/LaunchDaemons/limit.maxfiles.plist - - This will increase the maximum open file limits system wide on macOS - (last tested on 10.11.6). - The web interfaces and SSH are available over Tor. A copy of the the Onion URLs for *Source* and *Journalist Interfaces*, as well as SSH access, are written to the Vagrant host's ``install_files/ansible-base`` directory. @@ -173,8 +128,7 @@ playbook from running with Vagrant-specific info. You can provision production VMs from an Admin Workstation (most realistic), or from your host. If your host OS is Linux-based and you plan to use an Admin Workstation, you will need to switch Vagrant's default virtualization provider -from ``virtualbox`` to ``libvirt``. The Admin Workstation VM configuration -under Linux uses QEMU/KVM, which cannot run simultaneously with Virtualbox. +to ``libvirt``. Instructions for both installation methods follow. @@ -370,4 +324,3 @@ SSH Access By default, direct SSH access is not enabled in the prod environment. You will need to log in over Tor after initial provisioning or set ``enable_ssh_over_tor`` to "false" during ``./securedrop-admin tailsconfig``. - diff --git a/docs/development/virtualizing_tails.rst b/docs/development/virtualizing_tails.rst index 8ace62773..c559a69cc 100644 --- a/docs/development/virtualizing_tails.rst +++ b/docs/development/virtualizing_tails.rst @@ -12,244 +12,7 @@ Tails in a virtual machine. .. _`Tails`: https://tails.boum.org -macOS ------ - -For the macOS instructions, you will use VirtualBox to create a Tails VM that -you can use to install SecureDrop on ``app-prod`` and ``mon-prod``. - -Create a VirtualBox VM -~~~~~~~~~~~~~~~~~~~~~~ - -1. Open VirtualBox -2. Click **New** to create a new VM with the following options: - - * **Name**: "Admin Workstation" - * **Type**: "Linux" - * **Version**: "Debian (64-bit)" - -.. note:: You may call the VM a different name, but you must replace - "Admin Workstation" later on in these instructions with the name you select. - -3. Click **Continue**. -4. At the prompt, configure at least 2048 MB of RAM. Click **Continue**. -5. Leave the default **Create a virtual hard disk now** selected and click - **Create**. All the default options (**Hard disk file type: VDI (VirtualBox - Disk Image)** and **Dynamically allocated**) are fine. Click **Create**. - -Booting Tails -~~~~~~~~~~~~~ - -Now that the VM is set up, you are ready to boot to Tails. Select the new VM -in the VirtualBox sidebar, and click **Settings**. - -1. Click **Storage**. -2. Click **Empty** under **Controller: IDE**. -3. Click the CD icon next to **Optical Drive:** and click **Choose Virtual - Optical Disk File**. -4. Navigate to the Tails ISO to boot from. -5. Click **General** then **Advanced**. -6. Under **Shared Clipboard** select **Bidirectional** instead of **Disabled**. - This option will enable you to transfer text from your host to the Tails VM, - which we will use later on in these steps. - - .. note:: Alternatively you can open these docs in Tor Browser in Tails. - This will obviate the need to copy and paste between the guest - and host OS. - -Install Tails -~~~~~~~~~~~~~ - -Next you will install Tails onto the Virtual Hard Disk Image. Start the VM, boot -to Tails, and enter an administration password and start Tails. - -.. note:: For all the instructions that follow, you will need to configure an - administration password each time you boot Tails. - -1. Copy the following patch and save it as ``installer.patch`` in a folder in - your Tails VM: - -.. code:: Diff - - --- /usr/lib/python2.7/dist-packages/tails_installer/creator.py 2018-01-22 14:59:40.000000000 +0100 - +++ /usr/lib/python2.7/dist-packages/tails_installer/creator.py.mod 2018-03-05 05:15:00.000000000 -0800 - @@ -595,16 +595,6 @@ class LinuxTailsInstallerCreator(TailsInstallerCreator): - self.log.debug('Skipping non-removable device: %s' - % data['device']) - - - # Only pay attention to USB and SDIO devices, unless --force'd - - iface = drive.props.connection_bus - - if iface != 'usb' and iface != 'sdio' \ - - and self.opts.force != data['device']: - - self.log.warning( - - "Skipping device '%(device)s' connected to '%(interface)s' interface" - - % {'device': data['udi'], 'interface': iface} - - ) - - continue - - - # Skip optical drives - if data['is_optical'] and self.opts.force != data['device']: - self.log.debug('Skipping optical device: %s' % data['device']) - --- /usr/lib/python2.7/dist-packages/tails_installer/gui.py 2018-01-22 14:59:40.000000000 +0100 - +++ /usr/lib/python2.7/dist-packages/tails_installer/gui.py.mod 2018-03-05 05:15:00.000000000 -0800 - @@ -568,16 +568,6 @@ class TailsInstallerWindow(Gtk.ApplicationWindow): - self.devices_with_persistence.append(info['parent']) - continue - pretty_name = self.get_device_pretty_name(info) - - # Skip devices with non-removable bit enabled - - if not info['removable']: - - message =_('The USB stick "%(pretty_name)s"' - - ' is configured as non-removable by its' - - ' manufacturer and Tails will fail to start on it.' - - ' Please try installing on a different model.') % { - - 'pretty_name': pretty_name - - } - - self.status(message) - - continue - # Skip too small devices, but inform the user - if not info['is_device_big_enough_for_installation']: - message =_('The device "%(pretty_name)s"' - -2. Now run the following two commands in a Terminal in your Tails VM: - -.. code:: sh - - sudo patch -p0 -d/ < installer.patch - sudo /usr/bin/python -tt /usr/bin/tails-installer -u -n --clone -P -m -x - -3. The **Tails Installer** will appear. Click **Install Tails**. -4. Once complete, navigate to **Applications**, **Utilities** and open **Disks**. -5. Click on the disk named "Tails" and click the Play icon to mount the disk. -6. Next open ``/media/amnesia/Tails/syslinux/live*.cfg`` and delete all instances - of ``live-media=removable``. -7. Shut down the VM. - -Boot to Tails Hard Drive Install -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Now we will remove the CD and boot to the Tails we just installed on our -virtual hard drive. From macOS you should: - -1. Click the VM in the sidebar of VirtualBox and click **Settings**. -2. Click **Storage** and select the Tails .iso under **Controller: IDE**. -3. Click the CD icon, then **Remove Disk from Virtual Drive**. -4. Click **Ok**. -5. Start the VM. - -Configure Persistence -~~~~~~~~~~~~~~~~~~~~~ - -Now in your booted Tails VM you should: - -1. Configure an admin password when prompted. -2. Copy the following patch to the Tails VM and save it as ``persistence.patch``: - -.. code:: Diff - - --- /usr/share/perl5/Tails/Persistence/Setup.pm 2017-06-30 09:56:25.000000000 +0000 - +++ /usr/share/perl5/Tails/Persistence/Setup.pm.mod 2017-07-20 07:17:48.472000000 +0000 - @@ -404,19 +404,6 @@ - - my @checks = ( - { - - method => 'drive_is_connected_via_a_supported_interface', - - message => $self->encoding->decode(gettext( - - "Tails is running from non-USB / non-SDIO device %s.")), - - needs_drive_arg => 1, - - }, - - { - - method => 'drive_is_optical', - - message => $self->encoding->decode(gettext( - - "Device %s is optical.")), - - must_be_false => 1, - - needs_drive_arg => 1, - - }, - - { - method => 'started_from_device_installed_with_tails_installer', - message => $self->encoding->decode(gettext( - "Device %s was not created using Tails Installer.")), - -3. To apply the patch, from the Terminal run: - -.. code:: sh - - sudo patch -p0 -d/ < persistence.patch - -4. Navigate to **Applications** then **Tails** and click **Configure - persistent volume**. Configure a persistent volume enabling all persistence - options. - -Shared Folders -~~~~~~~~~~~~~~ - -1. In macOS, click on the Tails VM in VirtualBox and then go to - **Settings**. -2. Click on **Shared Folders** and click the button on the right hand side to - add the folder. Navigate to the location of the SecureDrop repository on - your local machine. Check **Auto-mount**. Do not check - **Read-only**. - -3. Now reboot your Tails VM, decrypt the Persistent volume, and run the following - commands in a **Terminal** in Tails: - -.. code:: sh - - mkdir ~/Persistent/securedrop - echo 'if [ ! -d ~/Persistent/securedrop/install_files ]; then sudo mount -t vboxsf -o uid=$UID,gid=$(id -g) securedrop ~/Persistent/securedrop; fi' >> /live/persistence/TailsData_unlocked/dotfiles/.bashrc - -The first time you open a Terminal in that session you will be prompted for your -sudo password and the shared folder will be mounted. Each time you open a -Terminal thereafter in the Tails session, your sudo password will not be needed. - -Allow the Guest to Create Symlinks -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Finally, you'll need to allow the guest to create symlinks, which are -`disabled by default in VirtualBox`_. - -.. _`disabled by default in VirtualBox`: https://www.virtualbox.org/ticket/10085#comment:12 - -Shut down the Tails VM, and in your host run: - -.. code:: sh - - VBoxManage setextradata "Admin Workstation" VBoxInternal2/SharedFoldersEnableSymlinksCreate/securedrop 1 - -.. note:: If you named your Tails VM something other than "Admin Workstation", - you can run ``VBoxManage list vms`` to get the name of the Virtual Machine. - -Finally, restart VirtualBox. - -Configure Networking -~~~~~~~~~~~~~~~~~~~~ - -In order to communicate with the server VMs, you'll need to attach this -virtualized *Admin Workstation* to the ``securedrop`` network. - -.. warning:: If you named the SecureDrop repository something other than - ``securedrop``, you should connect your VM to the network of the same name. - -With the *Admin Workstation* VM turned off, you should: - -1. Click on the VM in VirtualBox. -2. Click **Settings**. -3. Click **Network** and then **Adapter 2**. -4. Enable this network adapter and attach it to the **Internal Network** called - ``securedrop``. -5. Click OK and start the VM. - -Now you should be able to boot to Tails, decrypt the Persistent volume, -navigate to ``~/Persistent/securedrop`` and proceed with the :ref:`production -install `. - -Disable Shared Clipboard (Optional) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -1. Click on the VM in VirtualBox. -2. Click **Settings**. -3. Click **General** and then **Advanced**. -4. Now that you are finished with copy pasting the patches above you can change - the **Shared Clipboard** from **Bidirectional** back to **Disabled**. +Only libvirt-based virtualized, on a Linux host, is supported. Linux -----