-
Notifications
You must be signed in to change notification settings - Fork 122
Description
Bug Description
When a runner throws an exception before completing its request (e.g., HTTP 404, connection error, gRPC failure), the original error message is replaced by a misleading KeyError: 'client_request_end'.
Why this happens
RequestContextManager wraps each runner execution in an async with block. The runner populates timing keys in self.ctx as it progresses:
on_client_request_start()→ setsctx["client_request_start"]on_request_start()→ setsctx["request_start"]- Runner does work (HTTP call, etc.)
on_request_end()→ setsctx["request_end_list"]on_client_request_end()→ setsctx["client_request_end"]
If the runner throws at step 3, steps 4-5 never run — ctx is missing those keys. Python guarantees __aexit__ still fires, and it tries to read all 4 values to propagate timing to the parent context:
async def __aexit__(self, exc_type, exc_val, exc_tb):
client_request_start = self.client_request_start
client_request_end = self.client_request_end # KeyError here
request_start = self.request_start
request_end = self.request_end
...The KeyError from __aexit__ replaces the original exception, so the user sees 'client_request_end' instead of the actual error.
How to reproduce
Any runner that fails mid-request will trigger this. For example:
- Create a custom runner that throws an exception after
on_client_request_start()but beforeon_client_request_end() - Register it and run a workload that uses it
class FailingRunner(Runner):
async def __call__(self, client, params):
request_context_holder.on_client_request_start()
request_context_holder.on_request_start()
raise Exception("actual error message")
# on_request_end() and on_client_request_end() never calledActual output:
Error: Cannot run task [my-task]: 'client_request_end'
Expected output:
Error: Cannot run task [my-task]: actual error message
This also occurs naturally when a transport.perform_request() call fails (e.g., connection timeout, HTTP error) before the timing context is fully populated.
Suggested fix
Guard __aexit__ to skip timing propagation when the context is incomplete, rather than making all properties lenient:
async def __aexit__(self, exc_type, exc_val, exc_tb):
if "client_request_end" not in self.ctx or "request_start" not in self.ctx:
self.ctx_holder.restore_context(self.token)
self.token = None
return False
# ... existing propagation logicThis preserves KeyError behavior on the properties (catching real bugs in normal flow) while allowing original exceptions to propagate correctly on the error path.
Affected file
osbenchmark/context.py—RequestContextManager.__aexit__(line 63)