Continuous Membership Reconciliation for streams

[SimonUnge]

Proposed Changes

This PR implements automatic membership reconciliation for RabbitMQ streams, a mirror of the existing quorum queue membership reconciliation feature, as discussed in this thread created by @tschuettig

In its current form, it behaves verbatim like the feature for Quorum Queues.

In the discussion, @kjnilsson brought up that "member changes for streams isn't as safe to do as for e.g. quorum queues which means that we're probably not quite ready for any such automation." but I wanted us to have a starting point to work from. It is an opt in feature and uses the same steps you'd manually do using the CLI stream commands. It has the same conditions for growing (and shrinking) as quorum queues - perhaps this in an area where we could add more checks, e.g that all replicas are healthy etc?

If this is the route to go, we could create shared code for QQ and Stream auto reconciliation, but I wanted to keep it separate in this PR to be able to clearly discuss the feature.

Core Implementation

rabbit_stream_periodic_membership_reconciliation.erl - Main gen_server that performs periodic membership reconciliation for streams
rabbit_stream_event_subscriber.erl - Event subscriber that triggers reconciliation on cluster events (node up/down, policy changes)

Configuration

Added stream-specific configuration schema entries in rabbit.schema:

stream.continuous_membership_reconciliation.enabled                                                                                                                                                                                                                                                                                                                                                                                                                                                
stream.continuous_membership_reconciliation.auto_remove                                                                                                                                                                                                                                                                                                                                                                                                                                            
stream.continuous_membership_reconciliation.interval                                                                                                                                                                                                                                                                                                                                                                                                                                               
stream.continuous_membership_reconciliation.trigger_interval                                                                                                                                                                                                                                                                                                                                                                                                                                       
stream.continuous_membership_reconciliation.target_group_size    

Types of Changes

What types of changes does your code introduce to this project?
Put an x in the boxes that apply

• Bug fix (non-breaking change which fixes issue #NNNN)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause an observable behavior change in existing systems)
• Documentation improvements (corrections, new content, etc)
• Cosmetic change (whitespace, formatting, etc)
• Build system and/or CI

Checklist

Put an x in the boxes that apply.
You can also fill these out after creating the PR.
This is simply a reminder of what we are going to look for before merging your code.

• Mandatory: I (or my employer/client) have have signed the CA (see https://github.com/rabbitmq/cla)
• I have read the CONTRIBUTING.md document
• I have added tests that prove my fix is effective or that my feature works
• All tests pass locally with my changes
• If relevant, I have added necessary documentation to https://github.com/rabbitmq/rabbitmq-website
• If relevant, I have added this change to the first version(s) in release-notes that I expect to introduce it

Further Comments

If this is a relatively large or complex change, kick off the discussion by explaining why you chose the solution
you did and what alternatives you considered, etc.

[LoisSotoLopez]

When considering the shrinking of members the proposed code takes into account the number of running nodes. I wonder if it would be interesting to also take into account the target group size.

Would it be interesting for customers to be able to reduce from e.g. 7 to 5 the target group size to reduce the number of members of a stream on a 7 nodes cluster?

[SimonUnge]

When considering the shrinking of members the proposed code takes into account the number of running nodes. I wonder if it would be interesting to also take into account the target group size.

Thats a good question. I personally think it is too scary for automation to take that decision, which node(s) should it remove, how would that decision be made? My guts says this is something we should leave to a human to take action on, unless we can come up with a safe and predictable choice.

Would it be interesting for customers to be able to reduce from e.g. 7 to 5 the target group size to reduce the number of members of a stream on a 7 nodes cluster?