Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor so that jobs are launched in separate docker containers #431

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

dtenenba
Copy link
Contributor

This is just getting started but I made it a draft PR so that it does not get lost in the sea of branches.

Basically the goal is to make it so that the motuz core containers can be restarted at any time (to deploy code updates, etc) without interrupting running jobs.

The way to do this is to make each copy/check job a separate docker container. You can launch a container within a container if you have Docker installed in the container and /var/lib/docker.sock on the host is mounted to /var/lib/docker.sock in the container.

The child container that gets launched needs to mount the same filesystems as its parent, but this is easy to figure out programmatically using the Docker API.

I've modified the celery container to have docker installed. But there is still a lot more to do:

  • Have a think about whether the celery container still needs to exist? If it is restarted, it will forget about running jobs. Jobs already phone in to the database so maybe the web app container (not the celery container) can launch jobs, and we just use the database to see what jobs are in flight? This needs plenty of testing....

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant