[OPIK-5689] [BE] fix: stop retrying rate-limited LLM calls and reduce log verbosity by ldaugusto · Pull Request #6132 · comet-ml/opik

ldaugusto · 2026-04-08T13:06:20Z

Details

Fix the Online Scoring retry storm generating ~28M rate_limit_exceeded errors/week and ~70M log lines/day.
The root cause is OpenAiErrorMessage.getCode() not mapping rate_limit_exceeded to HTTP 429, causing rate limit errors to be misclassified as 500 (retryable) instead of 429 (non-retryable) — triggering 3x Redis-level retries on top of langchain4j's inner retries.
Additionally, removes full Java stack traces from all error/retry log paths, replacing them with single-line error messages.

Map rate_limit_exceeded and insufficient_quota → 429, model_not_found → 404 in OpenAiErrorMessage
Remove full stack traces from BaseRedisSubscriber, ChatCompletionService, and OnlineScoringTraceThreadLlmAsJudgeScorer error logs
Note: the fix on OPIK-5605 already tackled part of this problem by reducing the error volume ~50%; this PR addresses the remaining root cause and logging issues

Change checklist

User facing
Documentation update

Issues

OPIK-5689

Testing

mvn test -Dtest="**/OpenAiErrorMessageTest" — 8 tests pass (new, parameterized tests for all error code → HTTP status mappings including rate_limit_exceeded → 429)
mvn test -Dtest="**/BaseRedisSubscriberTest" — 14 tests pass (verified new log format: error messages only, no stack traces)
mvn test -Dtest="**/ChatCompletionServiceTest" — 11 tests pass
Scenarios validated: known error codes map to correct HTTP statuses, unknown codes fall back to 500, null message returns null
Environment: local macOS, Docker (Redis testcontainer for BaseRedisSubscriberTest)
Staging deploy pending to validate log volume reduction in production

Documentation

… log verbosity Map OpenAI rate_limit_exceeded error to HTTP 429 so it's treated as a non-retryable client error, preventing unnecessary Redis stream retries. Remove full stack traces from error/retry logs across the online scoring pipeline to drastically cut log volume.

github-actions · 2026-04-08T13:06:36Z

📋 PR Linter Failed

❌ Missing Section. The description is missing the ## Details section.

❌ Missing Section. The description is missing the ## Change checklist section.

❌ Missing Section. The description is missing the ## Issues section.

❌ Missing Section. The description is missing the ## Testing section.

❌ Missing Section. The description is missing the ## Documentation section.

github-actions · 2026-04-08T13:10:13Z

Backend Tests - Integration Group 5

127 tests 127 ✅ 3m 13s ⏱️
30 suites 0 💤
30 files 0 ❌

Results for commit 702bbe6.

♻️ This comment has been updated with latest results.

apps/opik-backend/src/main/java/com/comet/opik/api/resources/v1/events/BaseRedisSubscriber.java

...in/java/com/comet/opik/api/resources/v1/events/OnlineScoringTraceThreadLlmAsJudgeScorer.java

apps/opik-backend/src/main/java/com/comet/opik/domain/llm/ChatCompletionService.java

...k-backend/src/test/java/com/comet/opik/infrastructure/llm/openai/OpenAiErrorMessageTest.java

andrescrz

This generally LGTM, just a bit of polishing before moving forward. Mostly about not losing debugging information and about double checking the quotas HTTP status code. The rest is minor.

apps/opik-backend/src/main/java/com/comet/opik/api/resources/v1/events/BaseRedisSubscriber.java

...in/java/com/comet/opik/api/resources/v1/events/OnlineScoringTraceThreadLlmAsJudgeScorer.java

.../opik-backend/src/main/java/com/comet/opik/infrastructure/llm/openai/OpenAiErrorMessage.java

andrescrz · 2026-04-08T13:28:30Z

...k-backend/src/test/java/com/comet/opik/infrastructure/llm/openai/OpenAiErrorMessageTest.java

+        var error = new OpenAiErrorMessage(
+                new OpenAiErrorMessage.OpenAiError("some error message", errorCode, "some_type"));


Very nit: Add builders (with to builder) and use them, to improve our code base.

Comment applies to all tests in this PR.

...k-backend/src/test/java/com/comet/opik/infrastructure/llm/openai/OpenAiErrorMessageTest.java

- Keep full Throwable in log calls (restore stack traces) but keep warn level for handled errors in BaseRedisSubscriber and ChatCompletionService - Consolidate duplicate log in OnlineScoringTraceThreadLlmAsJudgeScorer into single log.error with Throwable - Map insufficient_quota to 402 instead of 429 - Fix test naming convention, use whole-object assertions, merge standalone tests into parameterized

andrescrz

LGTM, just an optional comment that I didn't notice on the 1st review.

andrescrz · 2026-04-08T14:05:59Z

apps/opik-backend/src/main/java/com/comet/opik/domain/llm/ChatCompletionService.java

                    .ifPresent(llmProviderError -> failHandlingLLMProviderError(runtimeException, llmProviderError));

-            log.error(UNEXPECTED_ERROR_CALLING_LLM_PROVIDER, runtimeException);
+            log.warn(UNEXPECTED_ERROR_CALLING_LLM_PROVIDER, runtimeException);


Sorry as I didn't notice on the first review.

I'd maybe keep these two changes on this file at error level , as they're real errors and re-thrown.

Even better, probably these logs can and should be removed as the exception below should be printed somewhere if not caught or if logged by some Dropwizard error handle.

I'd say the best scenario would be removing these two if you confirm logs are duplicated.

ldaugusto requested a review from a team as a code owner April 8, 2026 13:06

github-actions bot assigned ldaugusto Apr 8, 2026

github-actions bot added java Pull requests that update Java code Backend tests Including test files, or tests related like configuration. labels Apr 8, 2026

Merge branch 'main' into daniela/opik-5689-fix-onlinescoring-retry-storm

f4595b1

ldaugusto force-pushed the daniela/opik-5689-fix-onlinescoring-retry-storm branch from 00faf36 to f4595b1 Compare April 8, 2026 13:13

baz-reviewer bot reviewed Apr 8, 2026

View reviewed changes

andrescrz reviewed Apr 8, 2026

View reviewed changes

baz-reviewer bot approved these changes Apr 8, 2026

View reviewed changes

andrescrz approved these changes Apr 8, 2026

View reviewed changes

thiagohora approved these changes Apr 8, 2026

View reviewed changes

ldaugusto merged commit a0f8ef6 into main Apr 8, 2026
76 checks passed

ldaugusto deleted the daniela/opik-5689-fix-onlinescoring-retry-storm branch April 8, 2026 14:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[OPIK-5689] [BE] fix: stop retrying rate-limited LLM calls and reduce log verbosity#6132

[OPIK-5689] [BE] fix: stop retrying rate-limited LLM calls and reduce log verbosity#6132
ldaugusto merged 3 commits intomainfrom
daniela/opik-5689-fix-onlinescoring-retry-storm

ldaugusto commented Apr 8, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Apr 8, 2026

Uh oh!

github-actions bot commented Apr 8, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

andrescrz left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

andrescrz Apr 8, 2026

Uh oh!

Uh oh!

Uh oh!

andrescrz left a comment

Uh oh!

andrescrz Apr 8, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		var error = new OpenAiErrorMessage(
		new OpenAiErrorMessage.OpenAiError("some error message", errorCode, "some_type"));

Conversation

ldaugusto commented Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Details

Change checklist

Issues

Testing

Documentation

Uh oh!

github-actions bot commented Apr 8, 2026

📋 PR Linter Failed

Uh oh!

github-actions bot commented Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Backend Tests - Integration Group 5

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

andrescrz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

andrescrz Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

andrescrz left a comment

Choose a reason for hiding this comment

Uh oh!

andrescrz Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ldaugusto commented Apr 8, 2026 •

edited

Loading

github-actions bot commented Apr 8, 2026 •

edited

Loading