Spark 3.5: Add max allowed failed commits to RewriteDataFiles when partial progress is enabled #9611
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
Author
|
@amogh-jahagirdar This is based on our discussion in #9400, but I'd like to go one step further. Throwing exception and fail can work with alert system easier than logging an error. WDYT? |
aokolnychyi
reviewed
Feb 9, 2024
aokolnychyi
reviewed
Feb 9, 2024
core/src/main/java/org/apache/iceberg/actions/BaseCommitService.java
Outdated
Show resolved
Hide resolved
manuzhang
commented
Feb 20, 2024
| } else if (failedCommits > maxFailedCommits) { | ||
| String errorMessage = | ||
| String.format( | ||
| "%s is true but %d rewrite commits failed. This is more than the maximum allowed failures of %d. " |
Member
Author
There was a problem hiding this comment.
I should mention this message is generated by GitHub copilot.
manuzhang
force-pushed
the
partial-progress-max-failed-commits
branch
2 times, most recently
from
4827464 to
8faaa67
Compare
manuzhang
force-pushed
the
partial-progress-max-failed-commits
branch
from
8faaa67 to
cbd5198
Compare
manuzhang
force-pushed
the
partial-progress-max-failed-commits
branch
from
cbd5198 to
0e09e09
Compare
Contributor
|
Sorry for the delay. I didn't forget. |
aokolnychyi
reviewed
Apr 12, 2024
...k/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java
Outdated
Show resolved
Hide resolved
aokolnychyi
reviewed
Apr 12, 2024
spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteDataFilesAction.java
Outdated
Show resolved
Hide resolved
manuzhang
force-pushed
the
partial-progress-max-failed-commits
branch
from
0e09e09 to
8069af1
Compare
nastra
reviewed
Apr 12, 2024
| String.format( | ||
| "%s is true but %d rewrite commits failed. This is more than the maximum allowed failures of %d. " | ||
| + "Check the logs to determine why the individual commits failed. If this is persistent it may help to " | ||
| + "increase %s which will break the rewrite operation into smaller commits.", |
Contributor
There was a problem hiding this comment.
nit: instead of break maybe use split?
nastra
reviewed
Apr 12, 2024
| table.refresh(); | ||
|
|
||
| List<Object[]> postRewriteData = currentData(); | ||
| assertEquals("We shouldn't have changed the data", originalData, postRewriteData); |
Contributor
There was a problem hiding this comment.
Suggested change
| assertEquals("We shouldn't have changed the data", originalData, postRewriteData); | |
| assertThat(postRewriteData).isEqualTo(originalData); |
nastra
approved these changes
Apr 12, 2024
Contributor
nastra
left a comment
nastra
left a comment
There was a problem hiding this comment.
LGTM with some nits that would be good to address
manuzhang
force-pushed
the
partial-progress-max-failed-commits
branch
from
8069af1 to
d13f167
Compare
Contributor
|
Looks like there are some related test failures: |
Contributor
|
I think we should continue to use |
…rtial progress is enabled
manuzhang
force-pushed
the
partial-progress-max-failed-commits
branch
from
d13f167 to
0c42809
Compare
aokolnychyi
approved these changes
Apr 15, 2024
Contributor
|
Thanks, @manuzhang! Thanks for reviewing, @nastra @RussellSpitzer! |
sasankpagolu
pushed a commit
to sasankpagolu/iceberg
that referenced
this pull request
Oct 27, 2024
zachdisc
pushed a commit
to zachdisc/iceberg
that referenced
this pull request
Dec 23, 2024
3 tasks
nastra
added a commit
to nastra/iceberg
that referenced
this pull request
Mar 24, 2025
…mits this backports apache#9449 and apache#9611 to Spark 3.4
nastra
added a commit
to nastra/iceberg
that referenced
this pull request
Mar 24, 2025
…mits this backports apache#9449 and apache#9611 to Spark 3.4
nastra
added a commit
to nastra/iceberg
that referenced
this pull request
Mar 24, 2025
…mits this backports apache#9449 and apache#9611 to Spark 3.4
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When partial progress is enabled,
rewrite_data_filesSpark app can succeed without any "progress" (commits). This PR adds an optionpartial-progress.max-failed-commitsand a RuntimeException will be thrown if the number of failed commits exceed it. The default value ispartital-progress.max-commitsto be consistent with current behavior.