Skip to content

Commit 9983920

Browse files
DaraanCopilotkamil-kaczmarek
authored andcommitted
[ci][RLlib] [test utils] Improve error message on some test-failures because of IndexError in test_utils.py (#54343)
## Why are these changes needed? In the CI I saw a fleaky test failing, but its error is not reported correctly: ```python [2025-07-04T00:01:57Z] Traceback (most recent call last): [2025-07-04T00:01:57Z] File "/root/.cache/bazel/_bazel_root/1df605deb6d24fc8068f6e25793ec703/execroot/com_github_ray_project_ray/bazel-out/k8-opt/bin/rllib/examples/connectors/multi_agent_observation_preprocessor.runfiles/com_github_ray_project_ray/rllib/examples/connectors/multi_agent_observation_preprocessor.py", line 137, in <module> [2025-07-04T00:01:57Z] run_rllib_example_script_experiment(base_config, args) [2025-07-04T00:01:57Z] File "/rayci/python/ray/rllib/utils/test_utils.py", line 1354, in run_rllib_example_script_experiment [2025-07-04T00:01:57Z] f"{[e.args[0].args[2] for e in results.errors]}" [2025-07-04T00:01:57Z] File "/rayci/python/ray/rllib/utils/test_utils.py", line 1354, in <listcomp> [2025-07-04T00:01:57Z] f"{[e.args[0].args[2] for e in results.errors]}" # <--- [2025-07-04T00:01:57Z] IndexError: tuple index out of range # <--- ``` In the logs the actual error does not appear because of the `IndexError` for ` f"{[e.args[0].args[2] for e in results.errors]}"`. I am not sure why `e.args[0].args[2]` is used, a nested Exception perhaps? This PR adds a guard to fallback to a simpler `repr(e)` in case the `IndexError` would accur again. This allows to investigate why this `IndexError` occures and raises the `RuntimeError` like expected. ## Related issue number NA ## Checks - [x] I've signed off every commit(by using the -s flag, i.e., `git commit -s`) in this PR. - [x] I've run `scripts/format.sh` to lint the changes in this PR. - [x] I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/ - Testing Strategy - [x] This PR is not tested, but improves other tests :) --------- Signed-off-by: Daraan <github.blurry@9ox.net> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Kamil Kaczmarek <kaczmarek.poczta@gmail.com> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
1 parent 1106b93 commit 9983920

File tree

1 file changed

+9
-3
lines changed

1 file changed

+9
-3
lines changed

rllib/utils/test_utils.py

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1362,11 +1362,17 @@ def run_rllib_example_script_experiment(
13621362

13631363
# Error out, if Tuner.fit() failed to run. Otherwise, erroneous examples might pass
13641364
# the CI tests w/o us knowing that they are broken (b/c some examples do not have
1365-
# a --as-test flag and/or any passing criteris).
1365+
# a --as-test flag and/or any passing criteria).
13661366
if results.errors:
1367+
# Might cause an IndexError if the tuple is not long enough; in that case, use repr(e).
1368+
errors = [
1369+
e.args[0].args[2]
1370+
if e.args and hasattr(e.args[0], "args") and len(e.args[0].args) > 2
1371+
else repr(e)
1372+
for e in results.errors
1373+
]
13671374
raise RuntimeError(
1368-
"Running the example script resulted in one or more errors! "
1369-
f"{[e.args[0].args[2] for e in results.errors]}"
1375+
f"Running the example script resulted in one or more errors! {errors}"
13701376
)
13711377

13721378
# If run as a test, check whether we reached the specified success criteria.

0 commit comments

Comments
 (0)