[FEATURE] support use skip list to store shuffleBuffer in memory

[xianjingfeng]

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before asking

• I have searched in the issues and found no similar issues.

Describe the feature

Currently, we use linkedList to store shuffleBuffer in memory. If we assign a lot of memory(1TB) to the shuffle server, the performance will not be good while getting data from memory. Because every request needs to look for lastBlockId from the head position.

Other benefits of using skip list:

1. Fix [Bug] ShuffleReadClientImpl occurs StackOverflowError. #926
2. We don't need to sort data while flushing data to disk. [Improvement][AQE] Sort MapId before the data are flushed #137

Motivation

No response

Describe the solution

No response

Additional context

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

[jerqi]

We need to guarantee the order of data, otherwise we will lose the data.

[xianjingfeng]

We need to guarantee the order of data, otherwise we will lose the data.

Get it. This feature will not support slow start. I'm going to make it an optional feature.