Add call_id to LLM events for correlating requests#4281
Conversation
e69de9f to
d4b2828
Compare
|
Hey @vinibrsl, While streaming, there is no way to figure out which streaming chunks belongs to same chunk event, this PR add the response_id which is directly given by the provider. Though this only targets LLMStreamChunkEvent |
Thanks, @Vidit-Ostwal! That's great context. Unless I'm missing something here, Let me know if you think we should consolidate somehow or if keeping both makes sense. |
This makes more sense, The PR I add only targets the LLMStreamChunkEvent, which only if successful will be populated, but there are also other LLMStream events which needs to be coupled together, this PR adds that. Said that, this does makes the previous PR a bit redundant, as response_id has smaller scope, than call_id. which is trying to do the same functionality. |
We could follow up by marking response_id as deprecated, and eventually drop it in the next version. Thanks! |
It was merged today, 8 hrs back. |
d4b2828 to
2f0cdf1
Compare
When monitoring LLM events, consumers need to know which events belong to the same API call. Before this change, there was no way to correlate LLMCallStartedEvent, LLMStreamChunkEvent, and LLMCallCompletedEvent belonging to the same request.
2f0cdf1 to
3059c48
Compare
When monitoring LLM events, consumers need to know which events belong to the same API call. Before this change, there was no way to correlate LLMCallStartedEvent, LLMStreamChunkEvent, and LLMCallCompletedEvent belonging to the same request.
Note
Medium Risk
Introduces a new required field on all
LLMEventBaseevents and threads it through multiple providers and streaming paths, which could break any remaining event emissions that don’t setcall_idor rely on prior event schemas.Overview
Adds
call_idcorrelation to LLM events.LLMEventBasenow includes a requiredcall_id, and all LLM lifecycle events (started/chunks/completed/failed) are updated to emit this identifier.Implements call scoping and propagation. Introduces
llm_call_context()+get_current_call_id()(contextvars-backed) inBaseLLM, wrapsLLM.call/LLM.acalland native providercall/acallimplementations to establish a call scope, and attaches the currentcall_idto streaming chunks and error events.Updates and adds tests. Adjusts existing streaming/event tests to include
call_id, and adds VCR-backed tests + cassettes asserting same-call events share an ID and separate calls generate distinct IDs.Written by Cursor Bugbot for commit 3059c48. This will update automatically on new commits. Configure here.