Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parallelize ApplyByCols and ApplyToRows #40

Open
marcglobality opened this issue Oct 10, 2020 · 3 comments
Open

Parallelize ApplyByCols and ApplyToRows #40

marcglobality opened this issue Oct 10, 2020 · 3 comments

Comments

@marcglobality
Copy link

marcglobality commented Oct 10, 2020

First of all, thanks for this amazing library, really useful. Now going to the issue / request:

It seems to me that ApplyByCols and ApplyToRows could have a parameter to make them run in parallel. Has this ever been considered as a feature? I think it could speed up pipelines quite a lot, esp. useful for big dataframes. WDYT?

---- Edit by @shaypal5 : added in v0.0.67 -----
Also, I see that pipeline.fit and pipeline.transform have the timed bool. Would it be possible to add the same for apply? I know I can do pipeline.transform(...., time=True), but don't see a reason why apply cannot have it

@shaypal5
Copy link
Collaborator

  1. Hmm. Never considered this. I will definitely accept PRs around parallelization! :)
  2. Definitely. Perhaps it was just an oversight. This should be an easy PR to make. Perhaps you can write it?

Also, would you mind separating this into two distinct issues?
The first one is complex, while the other I can label with the first good issue label. :)

@yarkhinephyo
Copy link
Contributor

Hi! I would like to make a new issue for the No.2 and work on it if its okay :)

@shaypal5
Copy link
Collaborator

Yeah, definitely! Go fo it and let me know how it goes! :)

@shaypal5 shaypal5 changed the title Parallel and timed apply Parallelized ApplyByCols and ApplyToRows Nov 15, 2021
@shaypal5 shaypal5 changed the title Parallelized ApplyByCols and ApplyToRows Parallelize ApplyByCols and ApplyToRows Nov 15, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants