Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nested parallelism #34

Open
mr-j0nes opened this issue Jun 27, 2017 · 1 comment
Open

Nested parallelism #34

mr-j0nes opened this issue Jun 27, 2017 · 1 comment

Comments

@mr-j0nes
Copy link
Collaborator

mr-j0nes commented Jun 27, 2017

I wonder if this is possible:

A <= (B <= C >> D >=E)

E.g. this could work in this way:
A: reads filenames from a file system and send each to B
B: reads file and sends every record to C
C modifies record and sends to D
D modifies record and sends to E
E uploads records to DB and when all records belonging to a certain file are processed, marks file as done.

So the parallelism is on file level and on record level..

I know in the documentation says "not yet implemented nested split/joins" but just double checking..

@jonathan-beard
Copy link
Member

This one, I'm gonna have to double check on. It should work, but I wanna check my notes in the code to be sure. I've been spending a lot of my spare time cleaning up the DSL template parser. I'll get back to you soon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants