[router] Fix flaky router e2e tests

[XinyueZhang369]

Motivation

Fix flaky tests: test_circuit_breaker_disable_flag and test_disable_retries_surfaces_failure

Modifications

Change tests to tolerate the 503 when worker is not fully started

Run tests multiple times without failure

Accuracy Tests

Benchmarking and Profiling

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.

[gemini-code-assist]

Summary of Changes

Hello @XinyueZhang369, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances the stability of router end-to-end tests by making them more resilient to transient conditions during worker initialization. The changes prevent tests from failing due to temporary service unavailability (HTTP 503) and ensure that test failures accurately reflect underlying issues, improving the reliability of the test suite.

Highlights

• Flaky Test Fixes: Addressed flakiness in test_circuit_breaker_disable_flag and test_disable_retries_surfaces_failure by introducing retry logic.
• Worker Startup Tolerance: Modified tests to tolerate HTTP 503 (Service Unavailable) responses during initial worker startup, ensuring tests only fail for actual issues rather than transient unavailability.
• Robustness: Implemented a loop to retry requests up to 8 times, asserting that any non-500 status code encountered during this period must be a 503, and ultimately confirming a 500 status code is received.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request addresses flaky e2e tests by introducing a retry mechanism to wait for workers to become fully available. The changes in test_circuit_breaker.py and test_retries.py replace a single request with a loop that tolerates initial 503 Service Unavailable responses before the expected 500 Internal Server Error is returned.

My review focuses on improving the new test logic for better maintainability and debuggability. I've suggested adding descriptive messages to assertions to make test failures more informative and pointed out code duplication that could be refactored into a shared helper function. These changes will make the test suite more robust and easier to maintain.

[gemini-code-assist]

The final assertion assert saw_500 lacks a descriptive message, making test failures harder to debug. Please add a message, similar to the assertion on line 46.

Additionally, this retry logic is duplicated across multiple tests. Consider extracting it into a shared helper function to improve maintainability. The magic number 8 could also be defined as a constant.

Suggested change

	assert saw_500
	assert saw_500, "Worker did not return status 500 after retries"

[gemini-code-assist]

This assertion would be more informative with a failure message to aid in debugging. As noted in the review for test_circuit_breaker.py, this retry logic is also a candidate for refactoring into a shared helper function to reduce code duplication.

Suggested change

	assert saw_500
	assert saw_500, "Worker did not return status 500 after retries"