-
Notifications
You must be signed in to change notification settings - Fork 2.2k
Open
Description
Problem
In owl/utils/enhanced_role_playing.py, the async method arun_society() (line ~564) has a copy-paste bug where assistant_response completion tokens are added to overall_prompt_token_count instead of overall_completion_token_count. Additionally, user_response completion tokens are never counted at all.
Code Comparison
run_society (sync, lines ~499-505) — CORRECT:
overall_completion_token_count += assistant_response.info["usage"].get(
"completion_tokens", 0
) + user_response.info["usage"].get("completion_tokens", 0)
overall_prompt_token_count += assistant_response.info["usage"].get(
"prompt_tokens", 0
) + user_response.info["usage"].get("prompt_tokens", 0)arun_society (async, lines ~564-570) — BROKEN:
overall_prompt_token_count += assistant_response.info["usage"].get(
"completion_tokens", 0 # <-- WRONG: completion tokens go to prompt counter
)
overall_prompt_token_count += assistant_response.info["usage"].get(
"prompt_tokens", 0
) + user_response.info["usage"].get("prompt_tokens", 0)
# user_response completion tokens are NEVER countedImpact
token_info["completion_token_count"]is always0when using the async pathtoken_info["prompt_token_count"]is inflated by the assistant's completion tokens- Any cost tracking, billing dashboard, or benchmark reporting that relies on
arun_society's token counts is silently wrong - The async path is the one used by background processing and web app integrations
Proposed Fix
# arun_society — fixed
overall_completion_token_count += assistant_response.info["usage"].get(
"completion_tokens", 0
) + user_response.info["usage"].get("completion_tokens", 0)
overall_prompt_token_count += assistant_response.info["usage"].get(
"prompt_tokens", 0
) + user_response.info["usage"].get("prompt_tokens", 0)This matches the correct logic already present in run_society.
Happy to submit a PR for this fix.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels