Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

easy-to-reuse version of multiprocessing_plugin #8

Open
beOn opened this issue Apr 3, 2019 · 2 comments
Open

easy-to-reuse version of multiprocessing_plugin #8

beOn opened this issue Apr 3, 2019 · 2 comments

Comments

@beOn
Copy link
Contributor

beOn commented Apr 3, 2019

I've done a bit of an overhaul on the multiprocessing_plugin. The work's not quite done yet, and I haven't added docs, but I think the changes are pretty useful and would be happy to clean it up and provide some docs and support scripts if you'd like it in master. Eh, I also need to add support for handling events and spikes, but that should be pretty easy. Oh yeah, and for writing events back to the pipe. Anyways:

Everything's in my fork of this repo, under the 'blp' branch, in the plugin 'blp_trace'. The two key files are:

base classes: multiproc_base.pyx
example usage: blp_trace.pyx

In short:

  • Most of the ugly stuff, and all of the stuff related to threads, processes, and pipes, happens in abstract base classes that users shouldn't have to touch.
  • To make a new multiprocessing plugin, you can just make a plugin subclass and a processor subclass
  • I've added a couple of thread-safe circular buffer types
  • I've added a "pipe cleaner" thread that cleans out the input pipe rapidly and copies incoming data to buffers that get passed to preprocessing stages (or directly to the plotting stage)
  • I've added a base class (and an example usage) for handling preprocessing on background (non-plotting) threads.
  • The coordination of pipe reads, preprocessing, and plotting threads is all handled in the base classes
  • none of the base classes depend on the use of Matplotlib timers: the user is free to select whatever plotting or processing framework they like
  • with a little bit of work, this would be easy to generalize to handle not just plotting, but any kind of work people might want to do in a separate process... which could be pretty useful

I'm forgetting something, I'm sure. And I'm still working out the kinks, but the current version runs and plots.

@beOn
Copy link
Contributor Author

beOn commented Apr 3, 2019

Oh, right, and I pull the path for the python executable from the environment, and added an anaconda environment file.

@Claybarn
Copy link
Collaborator

Claybarn commented Apr 3, 2019

Looks good! Let me know when all the kinks are worked out. Definitely will be useful. We can make your version default then move the older version into legacy folder in case it is still need.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants