Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow different parallelization schemes/frameworks #56

Open
kain88-de opened this issue Sep 7, 2018 · 1 comment
Open

Allow different parallelization schemes/frameworks #56

kain88-de opened this issue Sep 7, 2018 · 1 comment

Comments

@kain88-de
Copy link
Member

Currently, pmda assumes that all problems can be solved with a map-reduce scheme that maps over the frames. This scales well for a lot of problems. If an analysis type requires a lot of work per frame it might make sense to also parallelize the work on a single frame. This is not supported in the current.
I would, therefore, suggest adding a new base class FrameMapReduceBase that is a direct copy of ParallelAnalysisBase and replace the later. The structure would then be.

class ParallelAnalysisBase:
    """ define public API for *any* pmda class """
    def run(self, get=None, scheduler=None, n_jobs=-1, *args, **kwargs)
        if get is not None:
           scheduler = get
        self._run(scheduler=scheduler, n_jobs=n_jobs, *args, **kwargs)
        return self

    def _run(self, scheduler=None, n_jobs=-1, *args, **kwargs):
        raise NotImplementedError

class FrameMapReduceBase(ParallelAnalysisBase):
    """ Implement Map-Reduce and 'map' over frames """
    def _run(self, scheduler=None, n_jobs=-1, *args, **kwargs):
       # current implementation

This would better document the parallelization scheme we use and be very explicit about what functions we expect for a user-facing API. Right now this would only be the run method and it should return a reference to the class. If an analysis uses multiple-cores but a different scheme to generate the DAG-graph like #47 it would only need to inherit from ParallelAnalysisBase and is free to implement it's own DAG.

@orbeckst
Copy link
Member

orbeckst commented Sep 7, 2018

I like this idea.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants