Management commands (and other improvements)

[153957]

Use Django management commands for frequently executed tasks instead of a separate script and an endpoint.

Benefits: Cleaner cron jobs, easier to test, use Django logging to guide log output, easier to build in options, no publicly available endpoint.

Todo:

• Add easy install dev requirements make command (make devinstall)
• Improve Model.Meta, add verbose_name(_plural), use lists etc..
• Disable XMLRPC Proxy calls during tests (if VPN_PROXY == None).
• Build dictionaries for API using queryset methods instead of list comprehensions
• Move manage.py up one directory for easier usage and less complicated config
• Rework analysissessions
  • Use already found coincidences in ESD (CoincidenceQueries)
  • Lazily get traces from API to reduce required database storage (Analysis sessions space usage #38)
  • Double check trace alignment in frontend
  • Use number of particles + pulseheight for mips
  • Combine date and time database fields into datetime field (coincidences/events)
• Rework/improve histogram jobs
  • Add tests with real data files
  • Test histogram models
  • Test checks
  • Test jobs
• Rework status display views
  • Make network summary a class based view
  • Remove unnecessary source views, can be created with extra kwargs in urls
  • Reduce SQL queries by using select_related/prefetch_related
  • Replace {% url name link.0 link.1 … %} reverse lookups by get_absolute_url
• Rework maps
  • Use new data based status for map status indicators
  • DRYer code, less queries
• Test that the new commands work as expected
  • createsessions
  • updatehistograms
• Remove unused pulseheight fits (monitoring)
• Add caching (pip and conda) on Travis
• Improve data/coincidence download reliability
• Improve logging in case of notable errors/warnings
• Add cronic in case command still ends in error, see Improved logging of cron tasks #188 - This can be more easily added once we upgrade to CentOS 7 (Migrate to CentOS 7 #230) when chronic is included in the moreutils package (available via yum/EPEL)

[153957]

For more consistent tests this would benefit from HiSPARC/sapphire#186

[tomkooij]

Wow! +1

[153957]

I think this is about ready to be merged. I did not actually test 5ea7973 with an actual local VM, but it seems correct. Additional tests might be nice, for example for the jobs for coincidences/reconstructions/station offsets.. but that can be done later.

@tomkooij Could you review this? (and merge & deploy if you wish).

[153957]

The tests with actual data are mostly important for a transition to Python 3, to ensure that that all keeps working as expected..

[davidfokkema]

There are problems with 5ea7973. The publicdb_base variable is also used in roles/publicdb/defaults/main.yml for the filesystem location of publicdb. So I got this building the VM:

TASK [publicdb : install packages in conda environment using pip] **************
fatal: [publicdb]: FAILED! => {
    "changed": false,
    "cmd": "http://localhost:80/publicdb/publicdb_venv/bin/pip list --format=freeze",
    "rc": 2
}

MSG:

[Errno 2] No such file or directory

because of these lines in provisioning/roles/publicdb/defaults/main.yml

publicdb_base: /srv/
publicdb_path: "{{ publicdb_base }}publicdb/"

publicdb_code: "{{ publicdb_path }}www/"
publicdb_venv: "{{ publicdb_path }}publicdb_venv"

Both uses of publicdb_base mean entirely different things, unfortunately.

[153957]

Good catch, thanks!
I will rename it to publicdb_host or publicdb_base_url

[kaspervd]

Thanks @153957!