Skip to content

[NA] [E2E] fix: increase online scoring test timeouts for GitHub-hosted runners #6024

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
AndreiCautisanu merged 1 commit into from
Apr 1, 2026
Merged

[NA] [E2E] fix: increase online scoring test timeouts for GitHub-hosted runners #6024

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions tests_end_to_end/typescript-tests/tests/online-scoring/online-scoring.spec.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ import { modelConfigLoader } from '../../helpers/model-config-loader';

// Timeout constants
const RULE_ACTIVATION_TIMEOUT = 10000; // 10 seconds for rule to fully activate in backend
const PAGE_REFRESH_TIMEOUT = 2000; // 2 seconds wait after page refresh
const PAGE_REFRESH_TIMEOUT = 3000; // 3 seconds wait after page refresh
Copy link
Copy Markdown
Contributor

@baz-reviewer baz-reviewer bot Apr 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we replace PAGE_REFRESH_TIMEOUT = 3000 and any page.waitForTimeout uses with retry-capable assertions or helper waits per .agents/skills/playwright-e2e/test-conventions.md, and remove the sleep in the moderation scoring retry loop?

Finding type: AI Coding Guidelines | Severity: 🟢 Low

Want Baz to fix this for you? Activate Fixer

Other fix methods

Fix in Cursor

Prompt for AI Agents:

Before applying, verify this suggestion against the current code. In
tests_end_to_end/typescript-tests/tests/online-scoring/online-scoring.spec.ts around
lines 10 to 10, the PAGE_REFRESH_TIMEOUT constant is set to 3000ms which introduces a
hard-coded delay >2s; per test conventions remove or reduce this constant and do not use
it for page.waitForTimeout. Replace any uses of PAGE_REFRESH_TIMEOUT and any
page.waitForTimeout calls (search around lines 120-140 and specifically 129-133) with
retry-capable helpers such as expect(...).toBeVisible({ timeout: 2000 }) or
helperClient.waitUntil/polling helpers. Also in the moderation scoring retry loop around
lines 106 to 113 (and any loop using sleeps), remove the sleep-based waits and implement
polling using expect/assert-with-timeout or a helper that retries until the moderation
column appears or a total timeout is reached, ensuring no single hard-coded pause
exceeds 2s.

const POST_RELOAD_TIMEOUT = 1000; // 1 second wait after page reload

const enabledModels = modelConfigLoader.getEnabledModelsForOnlineScoring();
Expand Down Expand Up @@ -62,6 +62,7 @@ test.describe('Online Scoring Tests', () => {
helperClient,
createProjectApi,
}) => {
test.setTimeout(120_000);
const providerSetupHelper = new AIProviderSetupHelper(page);
const rulesPage = new RulesPage(page);
const tracesPage = new TracesPage(page);
Expand Down Expand Up @@ -106,7 +107,7 @@ test.describe('Online Scoring Tests', () => {

// Retry loop: wait for Moderation column header in the table, then check score value.
// Scoring is async and Anthropic models can take longer, so we allow generous retries.
const maxAttempts = 15;
const maxAttempts = 25;
let moderationColumnIndex = -1;
let moderationValue = '';

Expand Down
Loading