Skip to content

test: Fix flaky test TestRingHash_RequestHashKeyConnecting in ringhash_e2e_test.go.#9079

Open
Pranjali-2501 wants to merge 4 commits intogrpc:masterfrom
Pranjali-2501:flaky-test-9030
Open

test: Fix flaky test TestRingHash_RequestHashKeyConnecting in ringhash_e2e_test.go.#9079
Pranjali-2501 wants to merge 4 commits intogrpc:masterfrom
Pranjali-2501:flaky-test-9030

Conversation

@Pranjali-2501
Copy link
Copy Markdown
Contributor

@Pranjali-2501 Pranjali-2501 commented Apr 21, 2026

Fixes: #9030

Problem

The flakiness was caused by the following race condition in the test execution flow:

  • The test dispatched the second RPC in a background goroutine almost immediately after the first RPC call. Due to race, this second RPC would often execute before the balancer had finished updating its state to reflect that the first RPC was already connecting. Seeing no active connections in progress, the second RPC would trigger a redundant connection attempt, failing the test.
  • Another reason of flake is test relied on a background goroutine with a fixed sleep to release a connection hold. The hold was often released before the second RPC reached the picker, missing the CONNECTING state window we wanted to test.

The Fix

Refactored the test to use a deterministic negative assertion flow that eliminates background races:

The second RPC is now sent with a short timeout and WaitForReady(true). Because the first connection is held, the second RPC blocks and fails with DeadlineExceeded as expected. We verify the failure and assert that no new connection attempts were triggered during this time. Then, we resume the hold in the main thread to let the test finish.

Verified it on forge for a million times: link

RELEASE NOTES: N/A

@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 21, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 77.32%. Comparing base (06fc26a) to head (61f0b16).
⚠️ Report is 3 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #9079      +/-   ##
==========================================
- Coverage   80.52%   77.32%   -3.21%     
==========================================
  Files         413      413              
  Lines       33543    43463    +9920     
==========================================
+ Hits        27012    33607    +6595     
- Misses       4316     7283    +2967     
- Partials     2215     2573     +358     

see 365 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky test [1/100K]: RingHash_RequestHashKeyConnecting

2 participants