Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use future API to support parallel AlignTraj #68

Open
kain88-de opened this issue Sep 21, 2018 · 0 comments
Open

Use future API to support parallel AlignTraj #68

kain88-de opened this issue Sep 21, 2018 · 0 comments

Comments

@kain88-de
Copy link
Member

This is a proof of concept script how to use the futures api of dask to support the writing of results into a trajectory while still working on frames in parallel. It won't scale out a lot. It also assumes that the single_frame function returns a namedtuple with coordinates. The buffer can potentially blow up so use with care.

def dask_execution(single_frame, trajectory, client)
     # have writer in main process
     with Writer as w:
         futures = client.map(single_frame, trajectory)
         idx = 0
         for f in as_completed(futures):
             res = f.result()
             buf[res.frame] = res.xyz
             # check if we got a  result to write
             while True:
                 # write everything we can *in order*
                 if idx in buf:
                     w.write(buf[idx])
                     del buf[idx]
                     idx += 1
                # we can't write anything so break
                 else:
                     break
        # write leftover frames
         for i in sorted(buf):
             w.write(buf[i])
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants