A list of tasks for celery to manage tasks from users
This project uses pipenv, you should install pipenv to configure and run this project.
$ pipenv install
You can find a requirements.txt file too, so if you don't want to install pipenv you can install all dependencies using normal pip:
$ pip install -r requirements.txt
$ celery -A tipi_tasks worker -B -l info
If you're using pipenv environment you should run with the pipenv run command
$ pipenv run celery -A tipi_tasks worker -B -l info
There's a docker-compose for development that deploys a mongodb a redis and the python app with all deps.
To run this you should edit the .env file in: dockerdev/.env and set your
path in the APP_PATH variable.
To run all services:
$ cd dockerdev
$ docker-compose up -d
You'll have three containers running:
- tipi_tasks
- tipi_db
- tipi_redis
You should be able to enter in any of this containers with the docker exec, like:
# to get a shell
$ docker exec -ti tipi_tasks sh
# to get a python shell
$ docker exec -ti tipi_tasks pipenv run ipython
# to run the test script
$ docker exec -ti tipi_tasks pipenv run ./test-task.py
$ docker exec -ti tipi_redis sh
$ docker exec -ti tipi_db bash
$ docker cp testdb.js tipi_db:/tmp/
$ docker exec -ti tipi_db mongo /tmp/testdb.js
To launch a task you should import the task from the tipi_tasks module and
then call with the delay, for example:
from tipi_tasks import test
task = test.test_task.apply_async((3, 2))The task will be sent to the celery queue and will be executed by the broker.
This module uses a python file as a configuration, if you use this project
copying the source code you can edit the config file tipi_tasks/config.py.
To avoid conflicts in deployment with git the config file tries to load a
local_config.py file if exists that replaces all defined variables there.
So to use a custom configuration you should create a file called
local_config.py and place it in the directory where you'll launch the
celery worker and tasks.
The email template used to validate searches is placed in
tipi_tasks/templates/validation.html, but this can be replaced with the
configuration. There's a config variable called TEMPLATE_DIR that you
can override to use a custom template directory. For example:
# local_config.py
TEMPLATE_DIR = '/var/www/templates/'
SPARKPOST_API = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
FROM_EMAIL = 'sparkpost@wadobo.com'
DEBUG = True
CLEAN_EMAILS_TIMEOUT = 10In this example, the file validation.html should exists inside the directory
/var/www/templates.
You can use some variables in the template that will be replaced with the corresponding values:
- Email: email
- Email hash: email_id
- Search hash: hash
- Days left to confirm: timeout
To use this inside the html file you should follow the handlebars-style template language, something like:
<html>
<body>
Confirmation for the Email: {{email}}. You've {{timeout}} days to confirm this.
<br/>
Please, click in the following link to validate:
<a href="http://mydomain.com/validate/email/{{email_id}}/{{hash}}/>Validate</a>
</body>
</html>
Add new tasks to this project is really easy, you can add a new module with more
shared_task or periodic_task and then do the import * in the __init__.py
file.
In this version it's not possible to define custom tasks that return values or to track the task status. To be able to do that the celery app must define a backend and there's a problem right now with the latest python version (3.7) and the current celery version (4.2.0). This problem should be fixed in the next celery release.
This projects uses celery for all tasks, so it's really easy to extend the list of tasks available to use in other modules.