operations

Refinery operations

Supervisor (available in the refinery-platform virtualenv) is used to control Django development server and Celery processes only. Run the following commands from the refinery-platform virtual environment in Vagrant VM (or use the full path to the binary in the virtualenv directly. eg: ~/.virtualenvs/refinery-platform/bin/supervisorctl).

To start Supervisor daemon:

supervisord

To check status of processes under its control:

supervisorctl status

To manage processes:

supervisorctl <start|stop|restart> <process_name>

Please note: Supervisor is not configured to start at boot time. It starts automatically only when VM is provisioned:

• during the very first vagrant up
• any subsequent vagrant <reload|up> --provision or vagrant provision.

Most other service dependencies are installed using system packages and all are managed using SysV init scripts (Supervisor and virtual environment are not required):

• PostgreSQL: $ sudo service postgresql <start|stop|restart|status> (on AWS this runs in a completely separated RDS instance)
• Apache: $ sudo service apache2 <start|stop|restart|status>
• RabbitMQ: $ sudo service rabbitmq-server <start|stop|restart|status>
• Solr: $ sudo service solr <start|stop|restart|status>
• Memcached: $ sudo service memcached <start|stop|restart|status>

To update Solr index:

manage.py update_index [--batch-size=XXX]

If working with memory less than 4GB, set --batch-size to a smaller value than the default 1,000.

Refinery configuration

Configuration settings are stored in refinery/config/config.json. This file is created by Puppet from refinery/config/config.json.erb template during provisioning if it doesn't exist already. config.json.erb is populated with values set in the Puppet manifest: deployment/puppet/manifests/site.pp

Site configuration modes

By default the VM starts in dev mode. Also note that re-provisioning will reset the the VM to default settings.

Description

• prod: Production settings.
• dev: Development settings. Starts Django in debug mode.
• djdt: Development settings for Django Debug Toolbar.
• gdev: UI development settings. Uses unminimized and raw CSS. Suitable for using grunt watch.

Changing configurations

Run the following commands on the host:

workon refinery-deployment
fab vm conf:<mode>

Then remove port 8000 from the domain name of the Site object. Note: Django dev server will not serve static files when prod mode is enabled (DEBUG=False).

Code deployment

To deploy the latest code from Github to Vagrant VM, run the following from the refinery-deployment environment on the host:

fab vm update

Logs

Logs for Django dev server and Celery can be accessed using Supervisor:

supervisorctl tail [-f] <runserver|celery-w1|celery-w2|celery-beat|celerycam>

Actual log file locations:

Vagrant VM

• /vagrant/refinery/log - Django dev server, Celery, Solr, Supervisor
• /var/log/apache2 - Apache/mod_wsgi
  • Contains logs:
    • access
    • default_error
    • error
    • other_vhosts_access
    • refinery_access
    • refinery_error

AWS

• /var/log/cloud-init and /var/log/cloud-init-output - logs from instance initialization
• /srv/refinery-platform/refinery/log - Django dev server, Celery, Solr, Supervisor
• /var/log/apache2 - Apache/mod_wsgi
  • Contains logs:
    • access
    • default_error
    • error
    • other_vhosts_access
    • refinery_access
    • refinery_error

Galaxy cleanup

1. Delete unused histories, libraries and workflows using Galaxy GUI
2. Purge Histories and Datasets in Galaxy
3. Delete files from /mnt/galaxy/tmp

CloudMan instructions:

sudo su - galaxy
cd /mnt/galaxy/galaxy-app
source .venv/bin/activate

to purge histories and libraries (does not remove datasets from disk):

./scripts/cleanup_datasets/cleanup_datasets.py ./config/galaxy.ini -d 0 -r --purge_histories
./scripts/cleanup_datasets/cleanup_datasets.py ./config/galaxy.ini -d 0 -r --purge_libraries

to remove datasets from disk:

./scripts/cleanup_datasets/cleanup_datasets.py ./config/galaxy.ini -d 0 -r --purge_datasets