Development Environment

1. Pre-commit hook
2. PyCharm 5
3. Django debug toolbar
4. Refinery configuration modes

---

Pre-commit hook

Prior to committing any changes to Refinery, our pre-commit hooks should be installed. Generally, pre-commit hooks are simple scripts that run automatically when calling git commit. We use Pre-Commit to manage our pre-commit hooks. In our case we run Flake8 and JSHint to check for simple formatting issue regarding our Python JavaScript files.

Install

Pre-commit hooks need to be installed only once after a fresh git clone has been retrieved:

$ workon refinery-deployment
$ pre-commit install

Update

When you change .pre-commit-config.yaml then you have to manually update pre-commit. Unfortunately there's no update command that just works. You have to manually uninstall, clean, and install the hook again.

$ workon refinery-deployment
$ pre-commit uninstall
$ pre-commit clean
$ pre-commit install

Note: Do not run pre-commit autoupdate! This will edit .pre-commit-config.yaml and overwrite all version with the very latest commit hash of git, which I think (Fritz) is dangerous because the last commit might not be a stable release.

Configuration

The hooks themselves are configured in .pre-commit-config.yaml. For more information visit http://pre-commit.com/.

PyCharm Magic

PyCharm enthusiasts need to create a custom bash profile to ensure that they activate Refinery's deployment environment in order to access pre-commit. Follow these steps:

1. Create a new file .pycharmrc-refinery in your favorite location, e.g. /my/favorite/location/

2. Open that file and add the following line

   source ~/.bashrc
   workon refinery-deployment
   

   We are assuming that you've set up virtualenvwrapper and all the depending stuff in your .bashrc. If that's not the case then please source the correct bash profile; e.g. source /where/ever/.myFancyBashProfile.

3. Open PyCharm and got to your project settings. The at Tools -> Terminal -> Shell path and set it to:

   /bin/bash --rcfile /my/favorite/location/.pycharmrc-refinery
   

4. Next time you want to commit some nasty stuff that PEP8 doesn't like your commit will fail and an error message will enlighten you. 🖖

PyCharm 5

Enable Django support for the project (set refinery-platform/refinery directory as Django project root).

Remote debugging with a remote virtual environment interpreter

Configure a remote interpreter (Vagrant and/or SSH). Make sure to select the Python interpreter from the Refinery virtualenv on the remote host (e.g.: /home/vagrant/.virtualenvs/refinery-platform/bin/python - not the system default).

Django dev server

Create run/debug configuration for Django server:

• set Host to 0.0.0.0, Port to 8000
• add to Environment variables: DJANGO_SETTINGS_MODULE=config.settings.<CONFIG> and PYTHONUNBUFFERED=1
  • <CONFIG> is a deployment mode supported by our Fabric script (fab vm conf:<CONFIG>) and by default one of dev, prod, gdev, or djdt.
  • The value of <CONFIG> used for the DJANGO_SETTINGS_MODULE has to match the value of <CONFIG> used when launching the VM.
• choose a corresponding remote interpreter

Celery

NOTE: Adding CELERY_ALWAYS_EAGER = True to your settings file will run Celery tasks on the main thread, making debugging much easier.

Create run/debug configuration for Python:

• set Script to the path to of manage.py on the host
• set Script parameters to run the Celery worker, for example: celeryd -c 4 -Q celery --events
• set Environment variables and remote interpreter as above

Additional settings for SSH connection:

• add to Environment variables (to load packages installed into virtual environment): PYTHONPATH=/path/to/remote/virtualenv/refinery-platform/lib/python2.7/site-packages
• add to Path mappings (project directory for manage.py): /path/to/local/refinery-platform/refinery /path/to/remote/refinery-platform/refinery

Optionally, forward ports if remote host is behind a firewall, e.g.: ssh -L18000:localhost:8000 remote-host-name. Then connect to http://localhost:18000/ in web browser.

Troubleshooting:

1. Remote debugger may exit prematurely with code -1 the first time after configuration is created or changed - simply restart it and it should run successfully the second time.
2. Ignore New process is launching (breakpoints won't work in the new process) message from the debugger - breakpoints do work if "Attach to subprocess automatically while debugging" is enabled.
3. The default configuration after (re-)provisioning is prod

Django debug toolbar

To enable the Django debug toolbar run fab vm conf:djdt on the host. Finally, if you want to debug the API append &debug to your endpoint of choice, e.g. http://192.168.50.50:8000/api/v1/data_sets/?format=json&debug. This wraps the JSON output as HTML, which in turn will load the toolbar.

Note: The &debug query parameter only works when the toolbar is enabled.