Skip to content

feat(web): add Playwright E2E test infrastructure#1405

Open
YoungY620 wants to merge 1 commit intomainfrom
web-test
Open

feat(web): add Playwright E2E test infrastructure#1405
YoungY620 wants to merge 1 commit intomainfrom
web-test

Conversation

@YoungY620
Copy link
Copy Markdown
Collaborator

@YoungY620 YoungY620 commented Mar 11, 2026

Related Issue

N/A — new test infrastructure, no prior issue.

Description

Establish end-to-end testing infrastructure for the Kimi Web frontend to catch regression bugs early when frontend code changes. Tests run against a real backend using the _scripted_echo provider for deterministic, LLM-free execution — no mocks, no flaky network calls.

What's included

  • web/playwright.config.ts — Chromium project, baseURL 127.0.0.1:5494, on-first-retry traces
  • Shared fixtures (web/tests/e2e/helpers/fixtures.ts):
    • createSession() — creates a session via the UI dialog
    • apiTracker — intercepts /api/** calls for request counting
    • consoleErrors — captures console.error and pageerror events
    • sendMessage() / waitForResponse() — helper functions
  • 6 spec files, 16 test cases:
    • session-lifecycle — create → send message → archive → delete
    • websocket-streaming — progressive response, multi-turn message order
    • session-switching — cross-session content isolation, event leak guard
    • refresh-resilience — page refresh recovery, API call storm detection
    • dynamic-bugs — console.error guard, slow network resilience, rapid send
    • navigation — URL ?session= sync on create/navigate/reload
  • echo-scripts.txt — 50 rounds of text: DSL responses for _scripted_echo
  • CI job (web-e2e in ci-kimi-cli.yml) — starts backend with _scripted_echo + config.toml, runs Playwright, uploads report/traces as artifacts
  • npm scripts: test:e2e, test:e2e:ui, test:e2e:install

How to run locally

# 1. Build web assets into the Python package
make build-web

# 2. Start backend with _scripted_echo
mkdir -p /tmp/kimi-e2e
SCRIPTS_PATH=$(realpath web/tests/e2e/fixtures/echo-scripts.txt)
cat > /tmp/kimi-e2e/config.toml <<TOML
default_model = "scripted"
[models.scripted]
provider = "scripted_provider"
model = "scripted_echo"
max_context_size = 100000
[providers.scripted_provider]
type = "_scripted_echo"
base_url = ""
api_key = ""
[providers.scripted_provider.env]
KIMI_SCRIPTED_ECHO_SCRIPTS = "$SCRIPTS_PATH"
KIMI_SCRIPTED_ECHO_TRACE = "1"
TOML
KIMI_SHARE_DIR=/tmp/kimi-e2e uv run kimi web --no-open --dangerously-omit-auth

# 3. Run tests
cd web && npm run test:e2e

No backend changes

All tests use the existing _scripted_echo provider and text: DSL. No modifications to packages/kosong/ or the Python backend.

Checklist

  • I have read the CONTRIBUTING document.
  • I have linked the related issue, if any.
  • I have added tests that prove my fix is effective or that my feature works.
  • I have run make gen-changelog to update the changelog.
  • I have run make gen-docs to update the user documentation.

Open with Devin

Set up Playwright E2E testing for the web frontend using _scripted_echo
provider for deterministic, LLM-free test execution.

- Add playwright.config.ts (chromium, baseURL 127.0.0.1:5494)
- Add shared fixtures: createSession, apiTracker, consoleErrors
- Add 6 spec files covering 16 test cases:
  - session-lifecycle: create, archive, delete
  - websocket-streaming: progressive response, multi-turn order
  - session-switching: cross-session isolation
  - refresh-resilience: recovery + API call storm detection
  - dynamic-bugs: console.error guard, slow network, rapid send
  - navigation: URL ?session= sync
- Add echo-scripts.txt (50 rounds of scripted responses)
- Add web-e2e CI job to ci-kimi-cli.yml
- Add npm scripts: test:e2e, test:e2e:ui, test:e2e:install
Copy link
Copy Markdown
Contributor

@devin-ai-integration devin-ai-integration bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 2 potential issues.

View 4 additional findings in Devin Review.

Open in Devin Review

forbidOnly: !!process.env.CI,
retries: process.env.CI ? 2 : 0,
workers: 1,
reporter: process.env.CI ? "github" : "html",
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 CI Playwright reporter "github" does not produce the playwright-report/ directory that the upload step expects

In web/playwright.config.ts:9, the reporter is set to "github" when process.env.CI is truthy. The "github" reporter only emits inline annotations to stdout—it does not generate a playwright-report/ directory. However, the CI workflow at .github/workflows/ci-kimi-cli.yml:335-338 uploads web/playwright-report/ with if: always(), clearly intending to preserve the HTML report for debugging failures. Because the directory is never created in CI, the artifact upload is effectively a no-op (the default if-no-files-found is warn), so no report will ever be available when tests fail in CI—defeating the purpose of the if: always() guard.

Suggested change
reporter: process.env.CI ? "github" : "html",
reporter: process.env.CI ? [["github"], ["html"]] : "html",
Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

KIMI_SCRIPTED_ECHO_TRACE = "1"
TOML
uv run kimi web --no-open --dangerously-omit-auth &
for i in $(seq 1 30); do curl -sf http://127.0.0.1:5494/healthz && break; sleep 1; done
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Health-check loop exits successfully even when the backend never starts

At .github/workflows/ci-kimi-cli.yml:326, the health-check loop for i in $(seq 1 30); do curl -sf … && break; sleep 1; done silently succeeds even if all 30 curl attempts fail, because the loop's last command is sleep 1 (exit code 0). The workflow then proceeds to run Playwright tests against a server that isn't running, producing confusing "connection refused" errors instead of a clear "backend failed to start" message. Adding a final check (e.g. curl -sf http://127.0.0.1:5494/healthz after the loop) would make the step fail fast with a clear diagnosis.

Suggested change
for i in $(seq 1 30); do curl -sf http://127.0.0.1:5494/healthz && break; sleep 1; done
for i in $(seq 1 30); do curl -sf http://127.0.0.1:5494/healthz && break; sleep 1; done
curl -sf http://127.0.0.1:5494/healthz || { echo "Backend failed to start after 30s"; exit 1; }
Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant