Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

State save in different process #221

Open
dnephin opened this issue Apr 6, 2013 · 1 comment
Open

State save in different process #221

dnephin opened this issue Apr 6, 2013 · 1 comment

Comments

@dnephin
Copy link
Contributor

dnephin commented Apr 6, 2013

Currently state is serialized in the same process as the twisted event loop. This should be done in another process. The state serialization process will need the following api:

  • configure - reconfigure the state serialization implementation
  • save - save some state
  • restore - restore state for jobs/services
  • shutdown - shutdown the process and implementation

This could be done with the multiprocessing module by opening a process before entering the twisted eventloop, or by creating a twisted.ProcessProtocol.

@dnephin
Copy link
Contributor Author

dnephin commented Apr 17, 2013

Part of this should be to split Jobrun out, and make it a top level entity that can be saved with its own key (currently only job and service are top level entities). This would greatly reduce the amount of data that is serialized. It will probably make state restore a little slower, but that should be acceptable.

Additionally, it would allow us to save more job history.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Development

No branches or pull requests

1 participant