[IMPROVED] MQTT: Retained msgs handling with cross account and transform#7636
Merged
derekcollison merged 1 commit intomainfrom Dec 12, 2025
Merged
[IMPROVED] MQTT: Retained msgs handling with cross account and transform#7636derekcollison merged 1 commit intomainfrom
derekcollison merged 1 commit intomainfrom
Conversation
This PR allows retained messages to work even if a situation where the retained message stream sources from another account and has subject transform. Regardless of that, the addition and removal of retained messages are now happening from the retained message stream's JS consumer callback. Previously, when receiving a retained message with an empty payload the server would remove the retained message on the given subject and notify the rest of the servers in the network with a NATS message. With this PR, the server simply stores the message with the empty body and all servers in the cluster can process this "delete" of a retained message by handling the fact that the payload is empty in `processRetainedMsg` which is the callback for the JS consumer. This PR still keeps the network notification for backward compatibility and makes sure that the older server cannot decode (and therefore ignore) a stored retained message with an empty payload. This PR also fixes the fact that if a retained message can no longer be found in the stream, it is appropriately removed from the map. Also, when recovering a retained message, the topic/subject is recreated based on the subject of the message read from the stream (minus the prefix). This allows subject transform to be properly supported. The server was previously adding shadow subscriptions' subject in the list of retained message subjects that need to be loaded, which I believe is wrong since those are on different account. No message would be found on the origin subject on the import account. The PR also includes various other fixes. One thing not done in this PR is to replace the use of Sublist with a GenericSublist that could be more useful/efficient, but would have to port the ReverseMatch to the generic sublist. This is trivial but would have added more changes to this already big PR. This can be done in a different one. Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
kozlovic
commented
Dec 11, 2025
| // Run from various go routines (JS consumer, etc..). | ||
| // No lock held on entry. | ||
| func (as *mqttAccountSessionManager) processRetainedMsg(_ *subscription, c *client, _ *Account, subject, reply string, rmsg []byte) { | ||
| func (as *mqttAccountSessionManager) processRetainedMsg(sub *subscription, c *client, acc *Account, subject, reply string, rmsg []byte) { |
Member
Author
There was a problem hiding this comment.
sub and acc are not needed. The change was done during dev but I forgot to revert I guess. I will update as part of other requested changes that may come as part of the review.
neilalexander
added a commit
that referenced
this pull request
Dec 15, 2025
neilalexander
added a commit
that referenced
this pull request
Dec 18, 2025
Includes the following: - #7553 - #7555 - #7579 - #7578 - #7581 - #7585 - #7586 - #7588 - #7593 - #7594 - #7595 - #7596 - #7597 - #7598 - #7601 - #7604 - #7605 - #7610 - #7616 - #7614 - #7622 - #7619 - #7624 - #7625 - #7627 - #7636 - #7637 - #7643 - #7648 - #7634 - #7655 - #7656 Signed-off-by: Neil Twigg <neil@nats.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR allows retained messages to work even if a situation where the retained message stream sources from another account and has subject transform.
Regardless of that, the addition and removal of retained messages are now happening from the retained message stream's JS consumer callback. Previously, when receiving a retained message with an empty payload the server would remove the retained message on the given subject and notify the rest of the servers in the network with a NATS message. With this PR, the server simply stores the message with the empty body and all servers in the cluster can process this "delete" of a retained message by handling the fact that the payload is empty in
processRetainedMsgwhich is the callback for the JS consumer.This PR still keeps the network notification for backward compatibility and makes sure that the older server cannot decode (and therefore ignore) a stored retained message with an empty payload.
This PR also fixes the fact that if a retained message can no longer be found in the stream, it is appropriately removed from the map.
Also, when recovering a retained message, the topic/subject is recreated based on the subject of the message read from the stream (minus the prefix). This allows subject transform to be properly supported.
The server was previously adding shadow subscriptions' subject in the list of retained message subjects that need to be loaded, which I believe is wrong since those are on different account. No message would be found on the origin subject on the import account.
The PR also includes various other fixes. One thing not done in this PR is to replace the use of Sublist with a GenericSublist that could be more useful/efficient, but would have to port the ReverseMatch to the generic sublist. This is trivial but would have added more changes to this already big PR. This can be done in a different one.
Signed-off-by: Ivan Kozlovic ivan@synadia.com