fix(prover-client): reject stale job promises and count timeouts toward retry limit#21842
Merged
PhilWindle merged 4 commits intomerge-train/spartanfrom Apr 21, 2026
Merged
Conversation
…rd retry limit Fixes A-711: cleanUpProvingJobState was deleting promises without settling them first, causing any awaiter to hang forever. Now rejects each unsettled promise before removal. Fixes A-715: timed-out jobs were re-enqueued without incrementing the retry counter, allowing them to loop forever bypassing maxRetries. Now increments the retry count on each timeout re-enqueue. Adds regression tests for both fixes. Made-with: Cursor
…job promise Made-with: Cursor
PhilWindle
requested changes
Mar 31, 2026
| @@ -632,8 +637,10 @@ export class ProvingBroker implements ProvingJobProducer, ProvingJobConsumer, Pr | |||
| const now = this.msTimeSource(); | |||
| const msSinceLastUpdate = now - metadata.lastUpdatedAt; | |||
| if (msSinceLastUpdate >= this.jobTimeoutMs) { | |||
| const retries = this.retries.get(id) ?? 0; | |||
Collaborator
There was a problem hiding this comment.
I don't think we want to make this change. We should limit this PR to the other change only.
Member
Author
There was a problem hiding this comment.
undid A-715 fixes, will do in different PR
Collaborator
There was a problem hiding this comment.
This setting of this.retries should have been removed shouldn't it?
alexghr
reviewed
Mar 31, 2026
spypsy
force-pushed
the
spy/fix-proving-broker-audit-711-715
branch
3 times, most recently
from
6ecca5c to
1da727e
Compare
…A-711) Made-with: Cursor
spypsy
force-pushed
the
spy/fix-proving-broker-audit-711-715
branch
from
1da727e to
5034e17
Compare
PhilWindle
approved these changes
Apr 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
cleanUpProvingJobStatewas callingdeferred.promise.catch(() => {})beforedeferred.reject()to suppress unhandled rejections, but this doesn't work —.catch()creates a new branched promise; any code already awaiting the original promise still receives an unhandled rejection. Fixed by resolving with{ status: 'rejected', reason: '...' }instead, consistent with how the rest of the class settles promises, and making unhandled rejections impossible.Fixes A-711