Cross-cluster merge risk in Kubernetes due to stale IP during CLUSTER MEET

[NikiTesla]

We would like to understand whether the following is a known limitation of Valkey Cluster.

In Kubernetes, we have automation on top of Valkey Cluster for cluster state management and re-adding/reconnecting nodes. A concern is that CLUSTER MEET is IP-based, while DNS results and pod IP ownership may change over time.

This creates a race where an IP resolved for one node may become stale before CLUSTER MEET is actually processed. In the worst case, that IP could already belong to a node from another cluster. If such a CLUSTER MEET is accepted, it seems possible for gossip to
spread foreign topology and effectively merge two independent clusters.

A pre-check before CLUSTER MEET does not seem sufficient, because the address can still change between validation and request handling.

We also noticed that cluster-announce-hostname helps with topology advertisement, but as far as we understand, CLUSTER MEET itself is still based on ip port, so the initial join path remains tied to IP identity.

Is this a known limitation of the current cluster design? Are there recommended ways to safely handle this kind of reconciliation in dynamic environments like Kubernetes?

[stevelipinski]

Not that this is the "right" solution, but the way we handle this in Kubernetes is to assign a K8s Service to each Pod (using a selector of statefulset.kubernetes.io/pod-name: ). We create the Service with the same name as the Pod, although that is not necessarily required - just makes things easier. We also set spec.publishNotReadyAddresses: true.
With this approach, we get a "fixed" (Service) IP for each Pod. We can then resolve those IPs once (via DNS or K8s service env injection) and use as static IPs.

One thing to be aware of is if you have any kind of elasticity or scaling - you'll need to take that into account, as the same mechanism (helm in our case) needs to scale the Service(s) when scaling the Pod controller as well.

Again, not necessarily ideal, but that's what we've done since pre-valkey and continue to do currently.

[NikiTesla]

Thank you, this does sound like a useful mitigation.
One concern for us is scale. We want to support users scaling storage clusters quite aggressively, potentially to thousands of nodes, so having a dedicated Service per pod may introduce additional operational overhead. Still, this seems manageable as long as the Services are scaled in lockstep with the storage pods.