From 6610277b93829434f3afb9f16dc6e0ca7c489042 Mon Sep 17 00:00:00 2001 From: Ilan Date: Wed, 19 Aug 2015 12:38:15 +0200 Subject: [PATCH 1/4] docs: moved Configuration to its own section * added more info on schedule catch up. --- docs/cluster.rst | 6 +- docs/configure.rst | 188 +++++++++++++++++++++++++++++++++++++++++++++ docs/index.rst | 1 + docs/install.rst | 181 ------------------------------------------- docs/schedules.rst | 9 +++ 5 files changed, 201 insertions(+), 184 deletions(-) create mode 100644 docs/configure.rst diff --git a/docs/cluster.rst b/docs/cluster.rst index 7a9d7f7f..49118899 100644 --- a/docs/cluster.rst +++ b/docs/cluster.rst @@ -42,14 +42,14 @@ Stopping the cluster with ctrl-c or either the ``SIGTERM`` and ``SIGKILL`` signa 16:44:14 [Q] INFO Process-1:9 stopped monitoring results 16:44:15 [Q] INFO Q Cluster-31781 has stopped. -The number of workers, optional timeouts, recycles and cpu_affinity can be controlled via the :ref:`configuration` settings. +The number of workers, optional timeouts, recycles and cpu_affinity can be controlled via the :doc:`configure` settings. Multiple Clusters ----------------- You can have multiple clusters on multiple machines, working on the same queue as long as: - They connect to the same Redis server or Redis cluster. -- They use the same cluster name. See :ref:`configuration` +- They use the same cluster name. See :doc:`configure` - They share the same ``SECRET_KEY`` for Django. Using a Procfile @@ -80,7 +80,7 @@ An example :file:`circus.ini` :: Note that we only start one process. It is not a good idea to run multiple instances of the cluster in the same environment since this does nothing to increase performance and in all likelihood will diminish it. -Control your cluster using the ``workers``, ``recycle`` and ``timeout`` settings in your :ref:`configuration` +Control your cluster using the ``workers``, ``recycle`` and ``timeout`` settings in your :doc:`configure` Architecture ------------ diff --git a/docs/configure.rst b/docs/configure.rst new file mode 100644 index 00000000..fe76ffe3 --- /dev/null +++ b/docs/configure.rst @@ -0,0 +1,188 @@ +Configuration +------------- +.. py:currentmodule:: django_q + +Configuration is handled via the ``Q_CLUSTER`` dictionary in your :file:`settings.py` + +.. code:: python + + # settings.py example + Q_CLUSTER = { + 'name': 'myproject', + 'workers': 8, + 'recycle': 500, + 'timeout': 60, + 'compress': True, + 'save_limit': 250, + 'queue_limit': 500, + 'cpu_affinity': 1, + 'label': 'Django Q', + 'redis': { + 'host': '127.0.0.1', + 'port': 6379, + 'db': 0, } + } + +All configuration settings are optional: + +name +~~~~ + +Used to differentiate between projects using the same Redis server. Defaults to ``'default'``. +This can be useful if you have several projects using the same Redis server. + +.. note:: + Tasks are encrypted. When a worker encounters a task it can not decrypt, it will be discarded. + +workers +~~~~~~~ + +The number of workers to use in the cluster. Defaults to CPU count of the current host, but can be set to a custom number. [#f1]_ + +recycle +~~~~~~~ + +The number of tasks a worker will process before recycling . Useful to release memory resources on a regular basis. Defaults to ``500``. + +.. _timeout: + +timeout +~~~~~~~ + +The number of seconds a worker is allowed to spend on a task before it's terminated. Defaults to ``None``, meaning it will never time out. +Set this to something that makes sense for your project. Can be overridden for individual tasks. + +compress +~~~~~~~~ + +Compresses task packages to Redis. Useful for large payloads, but can add overhead when used with many small packages. +Defaults to ``False`` + +.. _save_limit: + +save_limit +~~~~~~~~~~ + +Limits the amount of successful tasks saved to Django. + - Set to ``0`` for unlimited. + - Set to ``-1`` for no success storage at all. + - Defaults to ``250`` + - Failures are always saved. + +.. _sync: + +sync +~~~~ + +When set to ``True`` this configuration option forces all :func:`async` calls to be run with ``sync=True``. +Effectively making everything synchronous. Useful for testing. Defaults to ``False``. + +.. _queue_limit: + +queue_limit +~~~~~~~~~~~ + +This does not limit the amount of tasks that can be queued overall on Redis, but rather how many tasks are kept in memory by a single cluster. +Setting this to a reasonable number, can help balance the workload and the memory overhead of each individual cluster. +It can also be used to manage the loss of data in case of a cluster failure. +Defaults to ``None``, meaning no limit. + +label +~~~~~ + +The label used for the Django Admin page. Defaults to ``'Django Q'`` + +.. _catch_up: + +catch_up +~~~~~~~~ +The default behavior for schedules that didn't run while a cluster was down, is to play catch up and execute all the missed time slots until things are back on schedule. +You can override this behavior by setting ``catch_up`` to ``False``. This will make those schedules run only once when the cluster starts and normal scheduling resumes. +Defaults to ``True``. + +redis +~~~~~ + +Connection settings for Redis. Defaults:: + + redis: { + 'host': 'localhost', + 'port': 6379, + 'db': 0, + 'password': None, + 'socket_timeout': None, + 'charset': 'utf-8', + 'errors': 'strict', + 'unix_socket_path': None + } + +For more information on these settings please refer to the `Redis-py `__ documentation + +.. _django_redis: + +django_redis +~~~~~~~~~~~~ + +If you are already using `django-redis `__ for your caching, you can take advantage of its excellent connection backend by supplying the name +of the cache connection you want to use:: + + # example django-redis connection + Q_CLUSTER = { + 'name': 'DJRedis', + 'workers': 4, + 'timeout': 90, + 'django_redis: 'default' + } + + + +.. tip:: + Django Q uses your ``SECRET_KEY`` to encrypt task packages and prevent task crossover. So make sure you have it set up in your Django settings. + +cpu_affinity +~~~~~~~~~~~~ + +Sets the number of processor each worker can use. This does not affect auxiliary processes like the sentinel or monitor and is only useful for tweaking the performance of very high traffic clusters. +The affinity number has to be higher than zero and less than the total number of processors to have any effect. Defaults to using all processors:: + + # processor affinity example. + + 4 processors, 4 workers, cpu_affinity: 1 + + worker 1 cpu [0] + worker 2 cpu [1] + worker 3 cpu [2] + worker 4 cpu [3] + + 4 processors, 4 workers, cpu_affinity: 2 + + worker 1 cpu [0, 1] + worker 2 cpu [2, 3] + worker 3 cpu [0, 1] + worker 4 cpu [2, 3] + + 8 processors, 8 workers, cpu_affinity: 3 + + worker 1 cpu [0, 1, 2] + worker 2 cpu [3, 4, 5] + worker 3 cpu [6, 7, 0] + worker 4 cpu [1, 2, 3] + worker 5 cpu [4, 5, 6] + worker 6 cpu [7, 0, 1] + worker 7 cpu [2, 3, 4] + worker 8 cpu [5, 6, 7] + + +In some cases, setting the cpu affinity for your workers can lead to performance improvements, especially if the load is high and consists of many repeating small tasks. +Start with an affinity of 1 and work your way up. You will have to experiment with what works best for you. +As a rule of thumb; cpu_affinity 1 favors repetitive short running tasks, while no affinity benefits longer running tasks. + +.. note:: + + The ``cpu_affinity`` setting requires the optional :ref:`psutil ` module. + +.. py:module:: django_q + +.. rubric:: Footnotes + +.. [#f1] Uses :func:`multiprocessing.cpu_count()` which can fail on some platforms. If so , please set the worker count in the configuration manually or install :ref:`psutil` to provide an alternative cpu count method. diff --git a/docs/index.rst b/docs/index.rst index edd6f27d..d557fe9d 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -32,6 +32,7 @@ Contents: :maxdepth: 2 Installation + Configuration Tasks Schedules Cluster diff --git a/docs/install.rst b/docs/install.rst index e6b61f6e..80f16956 100644 --- a/docs/install.rst +++ b/docs/install.rst @@ -21,181 +21,6 @@ Installation - Make sure you have a `Redis `__ server running somewhere and know how to connect to it. -.. _configuration: - -Configuration -------------- - -Configuration is handled via the ``Q_CLUSTER`` dictionary in your :file:`settings.py` - -.. code:: python - - # settings.py example - Q_CLUSTER = { - 'name': 'myproject', - 'workers': 8, - 'recycle': 500, - 'timeout': 60, - 'compress': True, - 'save_limit': 250, - 'queue_limit': 500, - 'cpu_affinity': 1, - 'label': 'Django Q', - 'redis': { - 'host': '127.0.0.1', - 'port': 6379, - 'db': 0, } - } - - -name -~~~~ - -Used to differentiate between projects using the same Redis server. Defaults to ``'default'``. -This can be useful if you have several projects using the same Redis server. - -.. note:: - Tasks are encrypted. When a worker encounters a task it can not decrypt, it will be discarded. - -workers -~~~~~~~ - -The number of workers to use in the cluster. Defaults to CPU count of the current host, but can be set to a custom number. [#f1]_ - -recycle -~~~~~~~ - -The number of tasks a worker will process before recycling . Useful to release memory resources on a regular basis. Defaults to ``500``. - -.. _timeout: - -timeout -~~~~~~~ - -The number of seconds a worker is allowed to spend on a task before it's terminated. Defaults to ``None``, meaning it will never time out. -Set this to something that makes sense for your project. Can be overridden for individual tasks. - -compress -~~~~~~~~ - -Compresses task packages to Redis. Useful for large payloads, but can add overhead when used with many small packages. -Defaults to ``False`` - -.. _save_limit: - -save_limit -~~~~~~~~~~ - -Limits the amount of successful tasks saved to Django. - - Set to ``0`` for unlimited. - - Set to ``-1`` for no success storage at all. - - Defaults to ``250`` - - Failures are always saved. - -.. _sync: - -sync -~~~~ - -When set to ``True`` this configuration option forces all :func:`async` calls to be run with ``sync=True``. -Effectively making everything synchronous. Useful for testing. Defaults to ``False``. - -.. _queue_limit: - -queue_limit -~~~~~~~~~~~ - -This does not limit the amount of tasks that can be queued overall on Redis, but rather how many tasks are kept in memory by a single cluster. -Setting this to a reasonable number, can help balance the workload and the memory overhead of each individual cluster. -It can also be used to manage the loss of data in case of a cluster failure. -Defaults to ``None``, meaning no limit. - -label -~~~~~Dev - -The label used for the Django Admin page. Defaults to ``'Django Q'`` - -redis -~~~~~ - -Connection settings for Redis. Defaults:: - - redis: { - 'host': 'localhost', - 'port': 6379, - 'db': 0, - 'password': None, - 'socket_timeout': None, - 'charset': 'utf-8', - 'errors': 'strict', - 'unix_socket_path': None - } - -For more information on these settings please refer to the `Redis-py `__ documentation - -.. _django_redis: - -django_redis -~~~~~~~~~~~~ - -If you are already using `django-redis `__ for your caching, you can take advantage of its excellent connection backend by supplying the name -of the cache connection you want to use:: - - # example django-redis connection - Q_CLUSTER = { - 'name': 'DJRedis', - 'workers': 4, - 'timeout': 90, - 'django_redis: 'default' - } - - - -.. tip:: - Django Q uses your ``SECRET_KEY`` to encrypt task packages and prevent task crossover. So make sure you have it set up in your Django settings. - -cpu_affinity -~~~~~~~~~~~~ - -Sets the number of processor each worker can use. This does not affect auxiliary processes like the sentinel or monitor and is only useful for tweaking the performance of very high traffic clusters. -The affinity number has to be higher than zero and less than the total number of processors to have any effect. Defaults to using all processors:: - - # processor affinity example. - - 4 processors, 4 workers, cpu_affinity: 1 - - worker 1 cpu [0] - worker 2 cpu [1] - worker 3 cpu [2] - worker 4 cpu [3] - - 4 processors, 4 workers, cpu_affinity: 2 - - worker 1 cpu [0, 1] - worker 2 cpu [2, 3] - worker 3 cpu [0, 1] - worker 4 cpu [2, 3] - - 8 processors, 8 workers, cpu_affinity: 3 - - worker 1 cpu [0, 1, 2] - worker 2 cpu [3, 4, 5] - worker 3 cpu [6, 7, 0] - worker 4 cpu [1, 2, 3] - worker 5 cpu [4, 5, 6] - worker 6 cpu [7, 0, 1] - worker 7 cpu [2, 3, 4] - worker 8 cpu [5, 6, 7] - - -In some cases, setting the cpu affinity for your workers can lead to performance improvements, especially if the load is high and consists of many repeating small tasks. -Start with an affinity of 1 and work your way up. You will have to experiment with what works best for you. -As a rule of thumb; cpu_affinity 1 favors repetitive short running tasks, while no affinity benefits longer running tasks. - -.. note:: - - The ``cpu_affinity`` setting requires the optional :ref:`psutil ` module. - Requirements ------------ @@ -240,9 +65,3 @@ Optional $ pip install hiredis - -.. py:module:: django_q - -.. rubric:: Footnotes - -.. [#f1] Uses :func:`multiprocessing.cpu_count()` which can fail on some platforms. If so , please set the worker count in the configuration manually or install :ref:`psutil` to provide an alternative cpu count method. diff --git a/docs/schedules.rst b/docs/schedules.rst index ceb9005c..dc095a54 100644 --- a/docs/schedules.rst +++ b/docs/schedules.rst @@ -41,6 +41,15 @@ You can manage them through the :ref:`admin_page` or directly from your code wit minutes = 5, next_run = arrow.utcnow().replace(hour=18, minute=0)) + +Missed schedules +---------------- +If your cluster has not run for a while, the default behavior for the scheduler is to play catch up with the schedules and keep executing them until they are up to date. +For example, if your cluster was down for a day and you start it again. Any hourly schedules you had will run 24 times, once every schedule loop, until they are back in sync. +This behavior is intended to facilitate schedules that poll or gather statistics, but might not be suitable to your particular situation. +You can change this by settings the :ref:`catch_up` configuration setting to ``False``. +The scheduler will then skip execution of scheduled events in the past. Instead those tasks will run only once and normal scheduling resumes. + Management Commands ------------------- From da5e265333f1e65cfc448b8b9051e61f4b5caa2f Mon Sep 17 00:00:00 2001 From: Ilan Date: Wed, 19 Aug 2015 12:39:56 +0200 Subject: [PATCH 2/4] Updated comment on catch up --- django_q/conf.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/django_q/conf.py b/django_q/conf.py index 9a011abc..9fb5f6d7 100644 --- a/django_q/conf.py +++ b/django_q/conf.py @@ -78,7 +78,7 @@ class Conf(object): SYNC = conf.get('sync', False) # If set to False the scheduler won't execute tasks in the past. - # Instead it will reschedule the next run in the future. Defaults to True. + # Instead it will run once and reschedule the next run in the future. Defaults to True. CATCH_UP = conf.get('catch_up', True) # Use the secret key for package signing From 5ccd86a25449fbc003e7f67fe87253f8336b807b Mon Sep 17 00:00:00 2001 From: Ilan Date: Wed, 19 Aug 2015 12:40:29 +0200 Subject: [PATCH 3/4] v0.5.3 --- django_q/__init__.py | 2 +- docs/conf.py | 2 +- setup.py | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/django_q/__init__.py b/django_q/__init__.py index 47c95de2..72b57bb9 100644 --- a/django_q/__init__.py +++ b/django_q/__init__.py @@ -9,6 +9,6 @@ from .cluster import Cluster from .monitor import Stat -VERSION = (0, 5, 2) +VERSION = (0, 5, 3) default_app_config = 'django_q.apps.DjangoQConfig' diff --git a/docs/conf.py b/docs/conf.py index fd9d4f05..4775ee9a 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -72,7 +72,7 @@ # The short X.Y version. version = '0.5' # The full version, including alpha/beta/rc tags. -release = '0.5.2' +release = '0.5.3' # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. diff --git a/setup.py b/setup.py index 2f3cdfdc..6f370eaa 100644 --- a/setup.py +++ b/setup.py @@ -26,7 +26,7 @@ def run(self): setup( name='django-q', - version='0.5.2', + version='0.5.3', author='Ilan Steemers', author_email='koed00@gmail.com', keywords='django task queue worker redis multiprocessing', From 9fbf85947226a7b1989122ea4cbbf66b0666507a Mon Sep 17 00:00:00 2001 From: Ilan Date: Wed, 19 Aug 2015 12:56:04 +0200 Subject: [PATCH 4/4] Updates to Django 1.7.10 and 1.8.4 --- .travis.yml | 4 ++-- README.rst | 2 +- docs/index.rst | 2 +- docs/install.rst | 2 +- setup.py | 2 +- 5 files changed, 6 insertions(+), 6 deletions(-) diff --git a/.travis.yml b/.travis.yml index 31bed915..eff67cd7 100644 --- a/.travis.yml +++ b/.travis.yml @@ -8,8 +8,8 @@ python: - "3.4" env: - - DJANGO=1.8.3 - - DJANGO=1.7.9 + - DJANGO=1.8.4 + - DJANGO=1.7.10 install: - pip install -q django==$DJANGO diff --git a/README.rst b/README.rst index 9bac1e28..2043b285 100644 --- a/README.rst +++ b/README.rst @@ -32,7 +32,7 @@ Requirements - `Arrow `__ - `Blessed `__ -Tested with: Python 2.7 & 3.4. Django 1.7.9 & 1.8.3 +Tested with: Python 2.7 & 3.4. Django 1.7.10 & 1.8.4 Installation diff --git a/docs/index.rst b/docs/index.rst index d557fe9d..1b151dd9 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -24,7 +24,7 @@ Features - Python 2 and 3 -Django Q is tested with: Python 2.7 & 3.4. Django 1.7.9 & 1.8.3 +Django Q is tested with: Python 2.7 & 3.4. Django 1.7.10 & 1.8.4 Contents: diff --git a/docs/install.rst b/docs/install.rst index 80f16956..1297921a 100644 --- a/docs/install.rst +++ b/docs/install.rst @@ -29,7 +29,7 @@ Django Q is tested for Python 2.7 and 3.4 - `Django `__ Django Q aims to use as much of Django's standard offerings as possible - The code is tested against Django version `1.7.9` and `1.8.3`. + The code is tested against Django version `1.7.10` and `1.8.4`. - `Django-picklefield `__ diff --git a/setup.py b/setup.py index 6f370eaa..aad85717 100644 --- a/setup.py +++ b/setup.py @@ -40,7 +40,7 @@ def run(self): test_requires=['pytest', 'pytest-django', ], cmdclass={'test': PyTest}, classifiers=[ - 'Development Status :: 3 - Alpha', + 'Development Status :: 4 - Beta', 'Environment :: Web Environment', 'Framework :: Django', 'Intended Audience :: Developers',