feat: wire namespace affinity scheduler plugin by weng271190436 · Pull Request #461 · kubefleet-dev/kubefleet

weng271190436 · 2026-02-23T22:53:07Z

Description of your changes

Namespace affinity scheduler plugin that schedules RP to clusters with namespace based on MemberCluster namespaces status.

The user scenario:

Platform team only place namespace to a few clusters.
Application team use RP to place their services to all or some of the clusters where their namespace is available.

Fixes #501

I have:

Run make reviewable to ensure this PR is ready for review.

How has this code been tested

Special notes for your reviewer

codecov · 2026-02-24T00:11:44Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Signed-off-by: Wei Weng <Wei.Weng@microsoft.com>

weng271190436 · 2026-02-26T20:11:00Z

test/e2e/mixed_placement_test.go

-				return nil
-			}
-			Eventually(rpStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update RP %s status as expected", rpName)
+			rpStatusUpdatedActual := rpStatusUpdatedActual(appConfigMapIdentifiers(), []string{memberCluster2EastCanaryName, memberCluster3WestProdName}, nil, "0")


Previously we expect memberCluster1EastProdName to fail. Now with namespace affinity plugin, memberCluster1EastProdName won't be selected so the test expectation changes

weng271190436 · 2026-02-26T20:13:47Z

test/e2e/enveloped_object_placement_test.go

 		}
 	})

+	It("should wait for namespace collection to sync on all member clusters", func() {


Namespace collection can take a while to finish so placing this in all RP tests where a CRP was created to place a namespace to member clusters where RP resources go

What happens if the namepace is not reported in the MC yet? The scheduler should be triggered again when the namespace is reported. Does the eventually block in the following IT not enough?

When the namespace is finally reported in MC, the member cluster watcher can detect it and triggers placement reconciliation.

The eventually block can work if we wait long enough. The current 10 second wait time is not enough. And waiting for namespace collection makes it more deterministic

After pulling most waitForNamespaceCollectionOnClusters into its own It block based on
michaelawyu's comment, many of the waits are after createRP because many createRP happens inside BeforeAll. So waitForNamespaceCollectionOnClusters doesn't have to happen before createRP.

There are exceptions. For example in "Test cluster scale out and shrink using pickN policy with namespaced staged update run," we expect PickN where N = 1 to pick kind-cluster-3. This can also be deterministic if the namespaces are ready before createRP.

Signed-off-by: Wei Weng <Wei.Weng@microsoft.com>

ryanzhang-oss · 2026-03-03T01:17:54Z

pkg/scheduler/profile/profile_test.go

+}
+
+// TestNewProfileWithOptions tests the creation of a scheduling profile with custom options.
+func TestNewProfileWithOptions(t *testing.T) {


what does this test actually test?

I am testing calling NewProfile with option

ryanzhang-oss · 2026-03-03T01:20:19Z

pkg/scheduler/profile/profile.go

 	// default plugin list
 	clusterAffinityPlugin := clusteraffinity.New()
 	if opts.ClusterAffinityPlugin != nil {
 		clusterAffinityPlugin = *opts.ClusterAffinityPlugin


I wonder why is ClusterAffinityPlugin customizable while others are static?

We want to avoid merge conflict when backporting so allowing customization here while using the customization capability in Azure/fleet

Azure/fleet uses a customized cluster affinity plugin to support cluster affinity based on VM SKU availability

https://github.com/Azure/fleet/pull/1221/changes

ryanzhang-oss · 2026-03-03T01:31:23Z

test/e2e/enveloped_object_placement_test.go

 		}
 	})

+	It("should wait for namespace collection to sync on all member clusters", func() {


What happens if the namepace is not reported in the MC yet? The scheduler should be triggered again when the namespace is reported. Does the eventually block in the following IT not enough?

test/e2e/resource_placement_namespace_affinity_test.go

michaelawyu

Added some comments, PTAL.

test/e2e/utils_test.go

michaelawyu · 2026-03-03T07:10:13Z

test/e2e/resource_placement_drift_diff_test.go

 		Eventually(crpStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update CRP status as expected")
+
+		By("waiting for namespace to be collected on all member clusters")
+		waitForNamespaceCollectionOnClusters(nsName, allMemberClusterNames)


Hi weng271190436! Just a nit: I understand that this might be a must considering that it takes ~15 seconds for a namespace to be reported and our eventually block runs only for 10 seconds; but since it is a necessary step, would it be better if we make it a separate It block when applicable?

I pulled them into It blocks in most cases. There are 3 cases where waiting in It block doesn't work and I commented below

test/e2e/utils_test.go

michaelawyu · 2026-03-03T07:21:38Z

test/e2e/setup.sh

            --set workApplierRequeueRateLimiterMaxFastBackoffDelaySeconds=5 \
            --set propertyProvider=$PROPERTY_PROVIDER \
            --set region=${REGIONS[$i]} \
+            --set enableNamespaceCollectionInPropertyProvider=true \


Hi weng271190436! Since this is enabled per member agent and as a part of the property provider, we might need a check in the setup as well to make sure that it is running properly.

The member clusters are not joined to the hub during setup.sh. If they are joined I can verify that the namespaces are populated on the hub membercluster CRs. But since they are not joined, I can't think of a good way to make sure that it is running properly here. When they are joined, I check NamespaceCollectionSucceededCondType in utils_test.go to make sure that it is working.

Hi Wei! I mean that there was a separate part of the check at the environment setup step, checkIfAzurePropertyProviderIsWorking(), for verifying if property provider is working properly before the tests start up.

Come to think of that, for the test steps that require the property provider, we also need to mark them specifically so that they can be skipped if the environment has not provider available (isAzurePropertyProviderEnabled).

michaelawyu · 2026-03-03T07:24:33Z

pkg/scheduler/profile/profile.go


 	// default plugin list
 	clusterAffinityPlugin := clusteraffinity.New()
 	if opts.ClusterAffinityPlugin != nil {


Yeah, a note on this part: originally we added the profile support so that developers (or users) can switch scheduler profiles as needed (e.g., for some placements, use profile A, and the others profile B). Each profile can have its own plugin chain (e.g., B has cluster affinity support, A does not); it seems that the options part was added to achieve the same purpose? Of course we haven't exposed the profile switch API yet.

I checked the code and the options is not in use at all. Was it added back then to faciliate the development of the RP API?

Doesn't seem to be RP related. But it was added back recently https://teams.microsoft.com/l/message/19:13f7d68c3be840eb9281e41b6d1c15ea@thread.skype/1763091181077?tenantId=72f988bf-86f1-41af-91ab-2d7cd011db47&groupId=e121dbfd-0ec1-40ea-8af5-26075f6a731b&parentMessageId=1763068248219&teamName=Azure%20Container%20Compute&channelName=SIG%20Multi-Cluster%20-%20Dataplane&createdTime=1763091181077

I think I get it. We want to avoid merge conflict when backporting so allowing customization here while using the customization capability in Azure/fleet

https://github.com/Azure/fleet/pull/1221/changes

Signed-off-by: Wei Weng <Wei.Weng@microsoft.com>

…namespace-affinity-plugin

Signed-off-by: Wei Weng <Wei.Weng@microsoft.com>

weng271190436 · 2026-03-06T15:15:18Z

test/e2e/mixed_placement_test.go

 				Expect(memberCluster.KubeClient.Create(ctx, &ns)).To(Succeed())
 			}

+			By("waiting for namespace to be collected on all member clusters")


Didn't convert this to It block (based on a comment) because the RP created right below depends on namespaces created above. So need to wait inside BeforeAll

weng271190436 · 2026-03-06T15:17:44Z

test/e2e/staged_updaterun_test.go

+			// Ensure the namespace collection is synced on all member clusters before creating the RP.
+			// We expect kind-cluster-3 to be picked by PickN with N=1 policy due to the alphabetical order of cluster names.
+			// If kind-cluster-3 does not have namespace synced during schedule time, the RP might pick kind-cluster-1 or kind-cluster-2.
+			By("should wait for namespace collection to sync on all member clusters")


Here also needs to wait inside BeforeAll. Commented on why

weng271190436 · 2026-03-06T15:31:33Z

test/e2e/resource_placement_with_custom_config_test.go

 		crpStatusUpdatedActual := crpStatusUpdatedActual(workNamespaceIdentifiers(), allMemberClusterNames, nil, "0")
 		Eventually(crpStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update CRP status as expected")

+		By("waiting for namespace collection to sync on all member clusters")


In custom test, we expect "should not update RP status immediately". If I wait for namespace collection after the RP creation, the RP is likely to be updated when we expect it to be not updated yet.

Signed-off-by: Wei Weng <Wei.Weng@microsoft.com>

Makefile

michaelawyu

LGTM ;)

weng271190436 force-pushed the weiweng/namespace-affinity-plugin branch 2 times, most recently from eebd744 to 8fa620f Compare February 24, 2026 20:16

Wei Weng added 3 commits February 26, 2026 15:50

enable namespace affinity in e2e

0040654

Signed-off-by: Wei Weng <Wei.Weng@microsoft.com>

PickAll RP test

d8fa328

Signed-off-by: Wei Weng <Wei.Weng@microsoft.com>

add PickN test

08353f0

Signed-off-by: Wei Weng <Wei.Weng@microsoft.com>

weng271190436 force-pushed the weiweng/namespace-affinity-plugin branch from 32318e6 to 08353f0 Compare February 26, 2026 17:06

Wei Weng added 3 commits February 26, 2026 17:19

fix property provider expectation

6d9372d

Signed-off-by: Wei Weng <Wei.Weng@microsoft.com>

wait for namespace collection

7ef38ac

Signed-off-by: Wei Weng <Wei.Weng@microsoft.com>

fix rollout and mixed placement tests

0abd4ce

Signed-off-by: Wei Weng <Wei.Weng@microsoft.com>

weng271190436 commented Feb 26, 2026

View reviewed changes

weng271190436 marked this pull request as ready for review February 26, 2026 20:19

weng271190436 changed the title ~~feat: namespace affinity scheduler plugin~~ feat: wire namespace affinity scheduler plugin Feb 26, 2026

Wei Weng and others added 2 commits February 27, 2026 14:47

fix test coverage

0943719

Signed-off-by: Wei Weng <Wei.Weng@microsoft.com>

Merge branch 'main' into weiweng/namespace-affinity-plugin

c18b1b6

ryanzhang-oss reviewed Mar 3, 2026

View reviewed changes

michaelawyu reviewed Mar 3, 2026

View reviewed changes

weng271190436 and others added 9 commits March 5, 2026 10:34

Merge branch 'main' into weiweng/namespace-affinity-plugin

88b6c44

Merge branch 'main' into weiweng/namespace-affinity-plugin

1ea2212

address comments

cc58bca

Signed-off-by: Wei Weng <Wei.Weng@microsoft.com>

fix spelling

93b7d69

Signed-off-by: Wei Weng <Wei.Weng@microsoft.com>

comment on test behavior

71ce76b

Signed-off-by: Wei Weng <Wei.Weng@microsoft.com>

fix custom e2e test wait

1ff71ea

Signed-off-by: Wei Weng <Wei.Weng@microsoft.com>

fix e2e tests

ca324af

Signed-off-by: Wei Weng <Wei.Weng@microsoft.com>

fix custom test

50ba565

Signed-off-by: Wei Weng <Wei.Weng@microsoft.com>

Merge commit 'b625d1936a035174925c1264b6aeab4ca261277b' into weiweng/…

b1c572d

…namespace-affinity-plugin

weng271190436 force-pushed the weiweng/namespace-affinity-plugin branch from c848cb8 to b1c572d Compare March 5, 2026 23:41

Wei Weng added 2 commits March 6, 2026 14:37

fix mixed placement test

202b0b4

Signed-off-by: Wei Weng <Wei.Weng@microsoft.com>

wait for namespace collection earlier in one staged updaterun test

ceb570a

Signed-off-by: Wei Weng <Wei.Weng@microsoft.com>

weng271190436 commented Mar 6, 2026

View reviewed changes

weng271190436 and others added 3 commits March 6, 2026 13:23

Merge branch 'main' into weiweng/namespace-affinity-plugin

ba6c2ba

Merge branch 'main' into weiweng/namespace-affinity-plugin

2bf7bf6

use goimports version v0.42.0

da86391

Signed-off-by: Wei Weng <Wei.Weng@microsoft.com>

weng271190436 commented Mar 16, 2026

View reviewed changes

Makefile Show resolved Hide resolved

weng271190436 added 4 commits March 16, 2026 19:34

Merge branch 'main' into weiweng/namespace-affinity-plugin

bb52b36

Merge branch 'main' into weiweng/namespace-affinity-plugin

91d138a

Merge branch 'main' into weiweng/namespace-affinity-plugin

ef0a6d0

Merge branch 'main' into weiweng/namespace-affinity-plugin

0cc51f5

michaelawyu approved these changes Mar 25, 2026

View reviewed changes

weng271190436 merged commit 0c13d1a into kubefleet-dev:main Mar 25, 2026
22 of 23 checks passed

weng271190436 deleted the weiweng/namespace-affinity-plugin branch March 25, 2026 23:13

britaniar mentioned this pull request Mar 26, 2026

chore: Backport (03/26/2026) Azure/fleet#1280

Merged

1 task

Conversation

weng271190436 commented Feb 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of your changes

How has this code been tested

Special notes for your reviewer

Uh oh!

codecov bot commented Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

weng271190436 Mar 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

michaelawyu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

weng271190436 Mar 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

michaelawyu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

weng271190436 commented Feb 23, 2026 •

edited

Loading

codecov bot commented Feb 24, 2026 •

edited

Loading

weng271190436 Mar 5, 2026 •

edited

Loading

weng271190436 Mar 6, 2026 •

edited

Loading