[Fix] CI: e2e_ui_testing tests stale bundle on Ubuntu (cp -r merge semantics)

[yuneng-berri]

Summary

The problem

Every e2e_ui_testing run between the job's introduction on 2026-04-08 (commit d09d98a70a, adding the Build UI from source step) and the bundle-rebuild commit on 2026-04-18 (de790fd273) was testing a stale UI bundle — not the one produced from the branch's source.

The bundle that ended up in production had the double-prefix bug (NEXT_PUBLIC_BASE_URL="ui/" in .env.production combined with networking.tsx reading that env var, inlined by Next.js into a minified proxyBaseUrl = "/ui/" in the compiled JS). The trigger was in source from PR #25109 on 2026-04-06 — roughly 12 days before the broken bundle landed in _experimental/out/. During that window, e2e_ui_testing ran many times, built the broken bundle from source each time, and passed anyway.

Failure path (before fix)

The Build UI from source step ran:

cd ui/litellm-dashboard
npm run build
cp -r out/ ../../litellm/proxy/_experimental/out/

GNU cp (coreutils 8.32, which is what CircleCI's Ubuntu image ships) interprets cp -r src/ dest/ as "copy the source directory as a child of the destination" when the destination already exists. The command silently created litellm/proxy/_experimental/out/out/ — a nested dir the proxy does not serve — and left the served root litellm/proxy/_experimental/out/* untouched.

The proxy continued serving whatever bundle was checked into the repo. Playwright's globalSetup logged in against the stale (pre-PR) bundle, which worked fine. The freshly built bundle — the one that actually carried the bug — never reached the HTTP layer.

Fix

Replace the ambiguous cp -r with an unambiguous rm -rf + mv:

rm -rf ../../litellm/proxy/_experimental/out
mv out ../../litellm/proxy/_experimental/out

This cleanly swaps the destination for the freshly built bundle with no merging, no nested subdirectory, and no dependence on the quirks of whichever cp implementation the CI image ships.

Testing

Reproduced and verified end-to-end on Ubuntu 22.04 / GNU coreutils 8.32 (matching CircleCI):

State	ui/ literal occurrences in served bundle	Local e2e outcome
Bug source + current cp -r command	0 (stale bundle still served)	passes (stale bundle works)
Bug source + rm -rf / mv fix	9 (fresh broken bundle reaches proxy)	globalSetup times out at login — suite fully blocked, which is the intended signal
Bug removed + rm -rf / mv fix	0 (fresh clean bundle reaches proxy)	18 passed (2 pre-existing flakes unrelated to routing, 2 test.skip'd)

Local repro was done with the same stack CI uses: postgres 16, mock LLM server on 8090, python -m litellm.proxy.proxy_cli on 4000, cimg/python:3.12-browsers-equivalent build.

Why no new test infrastructure

The existing Playwright globalSetup already exercises what the user sees — it logs in as each role and waits for the post-login redirect. If the UI bundle can't reach the API, login POST 404s, the redirect never happens, waitForURL times out, and every downstream spec is blocked. That's the canonical catch for this class of bug; it just needed CI to actually hand it the freshly built bundle.

An earlier draft of this PR added a guardedPage fixture with request/response listeners. It was removed: status-based 4xx detection produced false positives on legitimate Next.js internals like /__next._tree.txt?_rsc=…, and the URL-pattern alternative required maintaining a ~50-entry API-verb list in parallel with networking.tsx. Once the CI rebuild-step bug is fixed, the existing globalSetup is sufficient.

Type

🐛 Bug Fix
🚄 Infrastructure

Follow-ups not in this PR

• Branch-filter gap. The e2e_ui_testing workflow filter runs on main and /litellm_.*/. Bundle-rebuild commits pushed to branches that don't match that pattern (e.g. yj_ui_build_apr18, which is how the broken bundle actually reached litellm_internal_staging on 2026-04-18) skip the UI suite entirely. Worth broadening the filter or path-triggering on _experimental/out/** changes.
• Branch protection. Mark the CircleCI e2e_ui_testing status as a required check on litellm_internal_staging / main in the repo settings UI.

[CLAassistant]

All committers have signed the CLA.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[greptile-apps]

Greptile Summary

This PR adds a "Build UI from source" step to the e2e_ui_testing CircleCI job, replacing the prior approach of testing against the checked-in _experimental/out/ bundle with a freshly compiled one. The fix correctly uses rm -rf + mv instead of cp -r to atomically replace the output directory, closing the gap where a broken compiled bundle could ship without the e2e suite ever running against it.

Note: the PR description extensively documents a guarded-page.ts Playwright fixture, meta-tests, unit tests, globalSetup.ts hardening, and a "Run guardedPage fixture meta-tests" CI step — none of which appear in the repository or in this diff. The commit message ("fix(ci): make e2e_ui_testing actually test the freshly built UI bundle") accurately reflects what was actually committed, but the PR description describes a substantially broader scope that was not implemented. The stated goal of adding an assertion path that would directly detect double-prefix API requests (/ui/ui/) remains unaddressed.

Confidence Score: 4/5

Safe to merge; the CI change is correct and a real improvement, but the guard fixture described as the primary deliverable was not committed.

The single changed file is a CI improvement that is mechanically correct. Remaining inline findings are P2 style/robustness suggestions. Score is 4 rather than 5 because the PR description commits to implementing the guarded-page.ts fixture and related spec migrations as part of this PR, but none of those files exist, leaving the 'no assertion path that would fail on a broken bundle' gap unaddressed.

No files require special attention. The only changed file (.circleci/config.yml) has two minor P2 findings.

Important Files Changed

Filename	Overview
.circleci/config.yml	Adds "Build UI from source" step to e2e_ui_testing that runs npm run build, replaces the checked-in bundle via rm -rf+mv, and pre-restructures HTML files. The core fix (test against fresh build) is correct and well-motivated. Two minor issues: the HTML restructuring is redundant with the proxy's own startup logic, and pipe-to-while subshell prevents mv failures from propagating.

Sequence Diagram

sequenceDiagram
    participant CI as CircleCI e2e_ui_testing
    participant FS as Filesystem
    participant Proxy as LiteLLM Proxy
    participant PW as Playwright

    CI->>FS: npm run build → out/
    Note over CI,FS: BEFORE: cp -r out/ → silently nested
    CI->>FS: rm -rf _experimental/out
    CI->>FS: mv out → _experimental/out (atomic replace)
    CI->>FS: find *.html → restructure to dir/index.html
    CI->>Proxy: Start proxy (--config)
    Proxy->>FS: _is_ui_pre_restructured() → True (skip)
    Proxy->>FS: StaticFiles mount /ui → _experimental/out
    CI->>PW: npx playwright test
    PW->>Proxy: HTTP requests against fresh build
    Note over PW,Proxy: Tests now exercise the actual compiled bundle

Loading

_{Reviews (3): Last reviewed commit: "fix(ci): make e2e_ui_testing actually te..." | Re-trigger Greptile}

[greptile-apps]

Empty allow-list may produce immediate false failures in the full suite

ALLOWED_ERROR_RESPONSES starts empty, but several migrated specs (e.g. unauthenticatedRedirect.spec.ts) may issue XHR/fetch calls whose responses legitimately land in the 4xx range (e.g. a 401 on an unauthenticated probe). Because the guard fires on any status >= 400 for xhr/fetch resource types, those tests will fail as soon as the full e2e_ui_testing job runs — not because the routing is broken, but because no allow-list entry exists for the expected-error path.

The PR description calls out "shake down ALLOWED_ERROR_RESPONSES" as a follow-up, which is the right approach. Just be prepared that the first CI run may flag pre-existing legitimate error responses and need entries added before the suite turns green.

[greptile-apps]

e2e_ui_testing serialization increases critical-path wall-time

Adding e2e_ui_testing to requires for build_and_test means the full Playwright suite (browser launch, globalSetup login flow, multi-spec run) must complete before either build_and_test or db_migration_disable_update_check can start. Previously these ran in parallel. Any transient UI flakiness or infra hiccup in the Playwright job will now block the entire merge pipeline. This is the intended design, but consider pairing it with a concrete flakiness budget or retry count on e2e_ui_testing so a single flaky test doesn't orphan an otherwise-clean backend change.