Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[GRAPH][ENHANCEMENT] Optimize SubtakGraph generation #3341

Open
zhongchun opened this issue Apr 20, 2023 · 0 comments · May be fixed by #3342
Open

[GRAPH][ENHANCEMENT] Optimize SubtakGraph generation #3341

zhongchun opened this issue Apr 20, 2023 · 0 comments · May be fixed by #3342

Comments

@zhongchun
Copy link
Contributor

zhongchun commented Apr 20, 2023

Problem

In gen_subtask_graph, Mars always create new out chunks even if the out chunk already exists. It costs a lot of time if there are plenty of chunks. Actually, there is no need to create out chunks except the FetchChunk.
I did a comparison, in which one creates new out chunks and the other does not. The result shows that the cost is reduced from 122.92s to 56.63s with about 53.93% reduction.

So we should optimize the SubtaskGraph generation and make it more efficient.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant