[IMPROVED] Remove no interest messages from head of stream#7766
[IMPROVED] Remove no interest messages from head of stream#7766neilalexander merged 1 commit intomainfrom
Conversation
neilalexander
left a comment
neilalexander
left a comment
There was a problem hiding this comment.
I'm concerned about the potential size that rmseqs can reach in memory and the potential slowness of calling RemoveMsg O(n) times.
In the first pass, it seems like we could just find the lowest asflr of known consumers and compact up to that? And then on a second pass, we can probably make better estimations from psim in the store without having to page every sequence into memory first.
MauriceVanVeen
force-pushed
the
maurice/interest-thresh-cleanup
branch
4 times, most recently
from
517f093 to
6643dc0
Compare
Have updated the PR to do just that. However, the Therefore, using the |
Update implementation to match the revised PR nats-io#7766 approach: - Use store.Compact(asflr) instead of iterating through messages - Calculate minimum chkflr across all consumers for the compact floor - Update checkStateForInterestStream to properly calculate retryAsflr for filtered consumers using LoadNextMsg to find next matching message This approach is more efficient: - No iteration through individual messages - No lock held during cleanup (Compact is atomic) - Leverages existing chkflr tracking per consumer - Handles filtered consumers correctly
MauriceVanVeen
force-pushed
the
maurice/interest-thresh-cleanup
branch
from
6643dc0 to
03f065d
Compare
Signed-off-by: Maurice van Veen <github@mauricevanveen.com>
MauriceVanVeen
force-pushed
the
maurice/interest-thresh-cleanup
branch
from
03f065d to
2a2df9d
Compare
2 participants
When using interest-based streams, messages that no longer have interest should be removed. For example when deleting a consumer that was the last one marking interest of subject A, then all messages matching subject A should be purged from the stream. If this goes above a 100k threshold, this linear scan will be expensive to perform so we don't block any other progress from happening. However, this would leave those messages around until a stream limit is reached.
This PR doesn't fully fix the issue, since the full scan is still expensive, but it does remove any messages at the "head" of the stream that have lost interest. For an active interest stream that's continuously consumed, these messages will be removed naturally without needing to block by going through them and removing immediately, as they'll naturally end up at the "head" of the stream and are eligible for removal.
Resolves #6747, #7754
Signed-off-by: Maurice van Veen github@mauricevanveen.com