[Improvement] pick partitions instead of shuffles for flushing #1717
Description
Code of Conduct
- I agree to follow this project's Code of Conduct
Search before asking
- I have searched in the issues and found no similar issues.
What would you like to be improved?
Currently, if the used memory reach the high watermark, the shuffle server will pick the shuffle with max size to flush.
In this case, if the size of a shuffle is very large, it will cause long GC pauses and high disk pressure.
How should we improve?
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!