Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement ReducePartition #13

Open
1 task done
wsry opened this issue Dec 6, 2021 · 1 comment
Open
1 task done

Implement ReducePartition #13

wsry opened this issue Dec 6, 2021 · 1 comment
Assignees
@TanYuxin-tyx

Comments

@wsry
Copy link
Collaborator

wsry commented Dec 6, 2021 •
edited by TanYuxin-tyx

Motivation

As described in the document, ReducePartition is a good supplement of the current MapPartition. It has several good features, for example, it can benefit streaming and hybrid shuffle (mixed streaming and batch) and it is good for performance for some scenarios.

Changes

It is big change including many aspects, including shuffle API enhancement (Flink side), ShuffleMaster change, shuffle write/read change, network protocol change, storage change, fault tolerance change and so on. We need to Implement it step by step.

Tasks

Test

  • UT test.
  • E2E test.
  • Stability test.
  • Test manually on a cluster.
@TanYuxin-tyx TanYuxin-tyx mentioned this issue Mar 7, 2022
Closed
@v-qiunan
Copy link

v-qiunan commented Oct 17, 2022 •
edited

@TanYuxin-tyx Is this feature already implemented ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@TanYuxin-tyx TanYuxin-tyx

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@wsry @TanYuxin-tyx @v-qiunan