Removal of priority 1 machines when their count matches desired replica

[takoverflow]

How to categorize this issue?

/area auto-scaling
/kind bug enhancement
/priority 3

What happened:
In the current machine-set replica management, if the count of machines for a set is the same as the desired replicas, then no creation/removal of machines occurs. Except for the case when there are failed machines that need replacement. This results in machines with priority 1 annotation i.e. machines that are marked for deletion by ClusterAutoscaler to never be processed for removal unless the replica count changes. This leads to nodes with no workload on them never getting removed by MCM. (Refer Live Issue 8962)

func (c *controller) manageReplicas(ctx context.Context, allMachines []*v1alpha1.Machine, machineSet *v1alpha1.MachineSet) error {
	...
	allMachinesDiff := len(allMachines) - int(machineSet.Spec.Replicas)
	machinesWithoutUpdateSuccessfulLabelDiff := len(machinesWithoutUpdateSuccessfulLabel) - int(machineSet.Spec.Replicas)

	if allMachinesDiff < 0 {
	    // Create required Machines
	} else if machinesWithoutUpdateSuccessfulLabelDiff > 0 {
	   // Remove excess machines without update successful
	}
	
	var staleMachines []*v1alpha1.Machine
	for _, m := range machinesWithoutUpdateSuccessfulLabel {
		if machineutils.IsMachineFailed(m) {
			staleMachines = append(staleMachines, m)
		}
	}

	if len(staleMachines) >= 1 {
		// Terminate stale machines
	}
}

In an ideal case, every machine that gets priority 1 annotation by CA is because CA decided to scale down and would've reduced the Machine Deployment replica count as well, however external actors such as gardenlet can apply stale replica count undoing CA's work. This results is scenarios where scale-down isn't triggered and priority 1 annotated machines persist in the cluster.

What you expected to happen:
Machines with priority 1 annotation are removed.

How to reproduce it (as minimally and precisely as possible):
Mark a bunch of existing machines manually with priority 1 annotation, observe that MCM doesn't replace them if the replica count doesn't change.

Anything else we need to know:
One possible mitigation to explore is to ensure that in case count of priority 1 machines matches desired replica i.e. the scenario where CA would've scaled down the deployment to 0 replicas without gardenlet interference can be handled by having additional check being done by the machine deployment controller i.e:

1. Check current MCD replicas: `n`
2. I = Get machine names with priority 1 annotation

Handle edge-case where len(I) = n -> Reduce MCD replicas to `0`.