New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CHORE] Current state vs Future state #374

Open

christopherpickering opened this issue Apr 28, 2023 · 1 comment

Labels

💪 chore

Member

christopherpickering commented Apr 28, 2023 •

edited

Current State

Atlas hub is a 3 part tool:

scheduler (flask app api) runs on a single gunicorn process using apscheduler
runner (flask app api) runs on a many cpu as are avialable
web (flask app) runs a few gunicorn processes

This system works remarkably well. The three apps use the same database model and connect to the same db for logging.

Drawbacks

three apps to manage
was easy to build and maintain for a small scheduler but more difficult and code heavy than other options

Future State

Flask API + Remix web app
Celery job scheduler

Advantages

two apps to manage
Remix frontend makes it easier to make a more responsive UI
No need to limit a scheduler app to a single process
Can be implemented on multiple servers
Celery has a more promising future than apscheduler

christopherpickering added the 💪 chore label

christopherpickering pinned this issue

Member Author

christopherpickering commented Jun 27, 2023

Converting to celery from apscheduler is nearly a 1:1 convert with different imports.

https://docs.celeryq.dev/en/stable/userguide/periodic-tasks.html
https://flask.palletsprojects.com/en/2.3.x/patterns/celery/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment