[IMPROVED] Stage meta operations during Raft catchup#7540
Merged
neilalexander merged 4 commits intomainfrom Nov 19, 2025
Merged
[IMPROVED] Stage meta operations during Raft catchup#7540neilalexander merged 4 commits intomainfrom
neilalexander merged 4 commits intomainfrom
Conversation
server/raft.go
Outdated
| if n.catchup != nil && n.catchup.sub != nil { | ||
| n.unsubscribe(n.catchup.sub) | ||
| } else { | ||
| // Signal we've started catching up. |
Member
There was a problem hiding this comment.
"Signal to the upper layer that the following entries are catchup entries, up until the nil guard."
server/jetstream_cluster.go
Outdated
| mset.mu.RUnlock() | ||
|
|
||
| for i, e := range ce.Entries { | ||
| // Ignore if lower catchup is started. |
Member
There was a problem hiding this comment.
May make sense for the comments here to explain why we'd ignore the Raft catchup, i.e. because there's an upper-layer catchup for gaps, and because we no-op sequences we've seen before?
Member
Author
There was a problem hiding this comment.
Updated but kept it at the bit more generic:
// Ignore if lower-level catchup is started.
// We don't need to optimize during this, all entries are handled as normal.For streams upper layer catchup and no-op sequences existing is not changed by this new EntryCatchup entry. So more highlighting that the normal flow is kept and there's no need to optimize here.
MauriceVanVeen
force-pushed
the
maurice/catchup-recovery
branch
6 times, most recently
from
57570cc to
e415658
Compare
Signed-off-by: Maurice van Veen <github@mauricevanveen.com>
Signed-off-by: Neil Twigg <neil@nats.io>
Signed-off-by: Maurice van Veen <github@mauricevanveen.com>
MauriceVanVeen
force-pushed
the
maurice/catchup-recovery
branch
2 times, most recently
from
0690d20 to
1c29c31
Compare
Signed-off-by: Maurice van Veen <github@mauricevanveen.com>
MauriceVanVeen
force-pushed
the
maurice/catchup-recovery
branch
from
1c29c31 to
4e26946
Compare
neilalexander
added a commit
that referenced
this pull request
Nov 26, 2025
wallyqs
pushed a commit
to wallyqs/nats-server
that referenced
this pull request
Jan 6, 2026
Backport of PR nats-io#7540 to release-v2.11.12. When a Raft node enters catchup mode from another server, it now enters a "recovery mode" that stages changes into recovery updates, allowing added and deleted consumers to become no-ops during this process. Key changes: - Add EntryCatchup entry type in raft to signal catchup start/end - Modify cancelCatchup to push nil entry when catchup completes - Modify createCatchup to send EntryCatchup entry at catchup start - Update monitorCluster to handle EntryCatchup and track recovery state - Change applyMetaEntries signature to return (isRecovering, didSnap, error) - Handle nil ce entries in monitorStream and monitorConsumer - Ignore EntryCatchup entries in applyStreamEntries and applyConsumerEntries Signed-off-by: Claude <noreply@anthropic.com>
neilalexander
added a commit
that referenced
this pull request
Feb 13, 2026
…eta snapshot (#7824) `TestJetStreamClusterDeleteConsumerWhileServerDown` (and others) would fail if the restarted server couldn't install a snapshot during shutdown. This happened if the server was a follower of the meta layer and committed the consumer create from a heartbeat, which isn't stored in the log. So, when the server restarted it didn't know it could commit/apply this consumer create again (since it already did so prior to restart). Then, when the meta snapshot was received to catch this server up, it would not properly remove the consumer as it wasn't tracked in its assignments. Marked as 2.12+ as #7540 is only cherry-picked there. Signed-off-by: Maurice van Veen <github@mauricevanveen.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When the Raft node underlying the meta layer enters catchup from another server, it gets placed in the same "recovery mode" such that it can stage changes into
ru *recoveryUpdatessuch that added and deleted consumers become a noop.Signed-off-by: Maurice van Veen github@mauricevanveen.com