Add Docker-based E2E CLI test infrastructure with Hex1b WithDockerContainer#14974
Add Docker-based E2E CLI test infrastructure with Hex1b WithDockerContainer#14974mitchdenny merged 24 commits intorelease/13.2from
Conversation
|
🚀 Dogfood this PR with:
curl -fsSL https://raw.githubusercontent.com/dotnet/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 14974Or
iex "& { $(irm https://raw.githubusercontent.com/dotnet/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 14974" |
There was a problem hiding this comment.
Pull request overview
Adds Docker-based infrastructure for Aspire CLI end-to-end tests using Hex1b’s new WithDockerContainer support, aiming to make E2E runs reproducible and isolated by executing inside ephemeral containers.
Changes:
- Add a multi-stage Dockerfile and
.dockerignorefor building/running CLI E2Es in Docker. - Add Docker-specific E2E helper APIs to create/configure container-backed Hex1b terminals and install the CLI by environment.
- Introduce initial Docker-backed smoke + staging-channel E2E tests; bump Hex1b package versions to enable the new API.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/Aspire.Cli.EndToEnd.Tests/docker/Dockerfile.e2e | Builds an SDK-based runtime image and optionally bundles the CLI from source for Docker E2Es. |
| tests/Aspire.Cli.EndToEnd.Tests/Helpers/CliE2ETestHelpers.cs | Adds Docker install-mode detection, container terminal creation, prompt setup, and CLI installation helpers. |
| tests/Aspire.Cli.EndToEnd.Tests/DockerStagingChannelTests.cs | New Docker-backed test for aspire config staging-channel behavior. |
| tests/Aspire.Cli.EndToEnd.Tests/DockerSmokeTests.cs | New Docker-backed smoke test verifying aspire --version. |
| Directory.Packages.props | Updates Hex1b-related package versions. |
| .vscode/mcp.json | Updates Hex1b MCP server tool version for VS Code config. |
| .dockerignore | Reduces Docker build context size for the new Docker-based test image. |
🎬 CLI E2E Test RecordingsThe following terminal recordings are available for commit
📹 Recordings uploaded automatically from CI run #22887739309 |
mitchdenny
force-pushed
the
aspire-hex1b-update
branch
from
99de09e to
2c80d00
Compare
mitchdenny
force-pushed
the
aspire-hex1b-update
branch
2 times, most recently
from
1bff3b1 to
32d0355
Compare
mitchdenny
force-pushed
the
aspire-hex1b-update
branch
2 times, most recently
from
666c6d0 to
a4b9e41
Compare
mitchdenny
force-pushed
the
aspire-hex1b-update
branch
2 times, most recently
from
3b33891 to
9391c92
Compare
| @@ -0,0 +1,101 @@ | |||
| # Multi-stage Dockerfile for Aspire CLI E2E testing (.NET variant). | |||
There was a problem hiding this comment.
Should this be in a shared location for use by CLI E2E and deployment E2E?
And description and contents made generic for both.
|
I tried it on Windows and it still fails. I don't know whether that's expected at this stage or not. AI conversation:
|
mitchdenny
force-pushed
the
aspire-hex1b-update
branch
from
9391c92 to
b114f95
Compare
|
@JamesNK testing it on a devbox now and it looks like the timeout of 5 minutes isn't long enough for Windows (half the resources are being taken by defender: |
|
Actually could be a bug in Hex1b. |
Update Hex1b, Hex1b.McpServer, and Hex1b.Tool from 0.97.0 to 0.105.0 in Directory.Packages.props. Also update .vscode/mcp.json MCP server reference from 0.47.0 to 0.105.0. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Uses Hex1b's WithDockerContainer API to run the test inside an isolated Docker container. The test installs the GA release of Aspire CLI from aspire.dev/install.sh and verifies it reports a version. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Introduce a multi-stage Dockerfile and environment-adaptive test helpers for portable E2E CLI testing across local dev, CI, and PR environments. - Dockerfile.e2e: Build stage compiles CLI from source (local dev), runtime stage provides the test environment. SKIP_SOURCE_BUILD arg allows CI to skip source compilation and use pre-built artifacts. - CliE2ETestHelpers: New DockerInstallMode enum, DetectDockerInstallMode, CreateDockerTestTerminal, PrepareDockerEnvironment, and InstallAspireCliInDocker helpers. - DockerSmokeTests: Rewritten to use the Dockerfile-based approach with automatic environment detection. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Rewrite Dockerfile.e2e to copy the entire repo and run the full build pipeline (restore, build, pack, bundle) including native AOT compilation. Install clang and zlib1g-dev for the AOT toolchain. Add .dockerignore to exclude artifacts/, .dotnet/, .git/, and IDE files from the build context. Update DetectDockerInstallMode to check for bundle archives. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Bake get-aspire-cli.sh and get-aspire-cli-pr.sh into the runtime stage at /opt/aspire-scripts/ so tests can install the CLI from GA or PR artifacts without downloading scripts from raw.githubusercontent.com. Update InstallAspireCliInDocker to use the local scripts. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Port StagingChannelTests to run inside a Docker container using the Hex1b WithDockerContainer API. The test exercises aspire config set/get/delete for staging channel settings entirely within an ephemeral container, eliminating cleanup concerns. Also extract GetRepoRoot() to CliE2ETestHelpers for reuse across Docker-based test classes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The get-aspire-cli-pr.sh script needs gh CLI to download PR artifacts from GitHub Actions. Install gh in the Dockerfile runtime stage and forward GH_TOKEN, GITHUB_PR_NUMBER, GITHUB_PR_HEAD_SHA to the container when running in PullRequest mode. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Dockerfile.e2e: Add Node.js 22, Python 3, docker.io CLI to runtime stage - Dockerfile.e2e-polyglot: New polyglot variant without .NET SDK - Add DockerfileVariant enum (DotNet, Polyglot) - CreateDockerTestTerminal: Accept variant, mountDockerSocket, workspace params - PrepareDockerEnvironment: Accept optional workspace for volume mount - Both Dockerfiles create /workspace mount point with WORKDIR set Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
All non-deployment CLI E2E tests now run inside Docker containers via Hex1b's WithDockerContainer API. This makes tests portable across local dev machines and CI environments. Changes: - Group A (CLI-only): BannerTests, DoctorCommandTests, StagingChannelTests, EmptyAppHostTemplateTests, SecretDotNetAppHostTests, PlaywrightCliInstallTests, AgentCommandTests, CentralPackageManagementTests - Group B (need Docker socket): SmokeTests, BundleSmokeTests, DescribeCommandTests, LogsCommandTests, PsCommandTests, StartStopTests, StopNonInteractiveTests, WaitCommandTests, MultipleAppHostTests - Group C (Node.js): JsReactTemplateTests, PythonReactTemplateTests - Group D (polyglot): SecretTypeScriptAppHostTests, TypeScriptPolyglotTests, TypeScriptStarterTemplateTests, ProjectReferenceTests - McpDocsE2ETests: Updated to use CliE2ETestHelpers.GetRepoRoot() - Added ToContainerPath helper for host-to-container path mapping Excluded: DockerDeploymentTests, KubernetesPublishTests (deployment tests) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
DockerSmokeTests and DockerStagingChannelTests were early proof-of-concept tests. Now that SmokeTests and StagingChannelTests are themselves Docker-based, these standalone files are redundant. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Two fixes for CI test failures:
1. Mount workspace at /workspace/{guid} instead of /workspace so that
the container directory name matches workspace.WorkspaceRoot.Name.
This fixes tests that match 'Enter the project name ({name}):' patterns.
2. Add 'dotnet dev-certs https' to DoctorCommandTests since Docker
containers don't have dev certs pre-generated.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1. Add libicu74 to polyglot Dockerfile - native AOT CLI requires ICU for globalization support (fixes TypeScript/Polyglot test crashes) 2. Fix DoctorCommandTests pattern - match 'is not trusted' instead of 'partially trusted' (matches actual aspire doctor output wording) 3. Set umask 000 in PrepareDockerEnvironment - files created inside the container (as root) need to be writable by host-side ExecuteCallback (fixes DescribeCommandTests permission denied) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1. DoctorCommandTests: Use 'dotnet dev-certs https --trust' to generate AND trust the dev cert, enabling the SSL_CERT_DIR trust detection test. 2. ProjectReferenceTests, TypeScriptStarterTemplateTests: Switch from Polyglot to DotNet variant since these tests reference .NET projects or use templates that require the .NET SDK. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1. DoctorCommandTests: Revert pattern to 'partially trusted' — with --trust the cert IS trusted, so without SSL_CERT_DIR the expected output is 'partially trusted' (not 'is not trusted'). 2. WaitCommandTests: Increase timeout from 120s to 300s since Docker-in-Docker container startup is significantly slower. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The aspire wait command times out waiting for resources in nested Docker environments. The inner containers (redis, apiservice, webfrontend) are too slow to start in Docker-in-Docker on CI. Filed tracking issue: #14993 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The agent init chaining feature (aspire new → agent init prompt) is only present in source-built CLI, not GA releases. DeclineAgentInitPrompt now accepts a SequenceCounter and handles both cases: - CLI with agent init: waits for prompt, declines, waits for success - CLI without agent init: detects success prompt directly, handles the harmless 'n' command at bash, adjusts counter accordingly This replaces the separate .DeclineAgentInitPrompt().WaitForSuccessPrompt() pattern across all 28 callsites with .DeclineAgentInitPrompt(counter). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Convert TypeScriptCodegenValidationTests to use Docker container (CreateDockerTestTerminal with Polyglot variant) like all other CLI E2E tests - Fix Dockerfile COPY wildcard that fails when SKIP_SOURCE_BUILD=true: copy the bundle directory instead and gate extraction conditionally Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Move Dockerfile.e2e and Dockerfile.e2e-polyglot to tests/Shared/Docker/ for use by both CLI E2E and deployment E2E tests - Add ITestOutputHelper to CreateDockerTestTerminal and log configuration (install mode, variant, workspace, dimensions, recording path) - Convert AspireAddPackageVersionToDirectoryPackagesProps test to Docker - Set ASPIRE_PLAYGROUND/TERM/dotnet env vars in PrepareDockerEnvironment for all Docker install modes (not just PullRequest) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add nuget.org as a package source for Hex1b packages and bump from 0.105.0 to 0.111.0. This version includes a fix for Docker build deadlocks on Windows (mitchdenny/hex1b#225). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Switch Hex1b package source to GitHub Packages (mitchdenny/hex1b PR #226) to test the cmd.exe wrapper fix for Docker's 'not a TTY' error on Windows. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Updated from the cmd.exe wrapper version to the WSL2 bridge version which correctly resolves the Docker 'not a TTY' error on Windows. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Includes WSL2 Docker TTY bridge and 5-minute docker build timeout. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The Docker image build can take over a minute, so 60s may not be enough for the container to start and show the bash prompt. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Revert the 0.111.0/0.112.0 preview package updates and NuGet.config changes. The Windows Docker TTY fix needs more work; keeping the existing version for now. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
mitchdenny
force-pushed
the
aspire-hex1b-update
branch
from
fcc3a9b to
1d73faa
Compare
|
@JamesNK sadly I don't think that we are going to be able to get past the windows issues. I just tried a whole host of approaches, none of which worked. Ultimately, we are slaves to the ConPty limitations on Windows. |
|
🥀 |
The good news is that WSL will work locally. |
| /// <param name="height">Terminal height in rows.</param> | ||
| /// <param name="testName">The test name for the recording file path.</param> | ||
| /// <returns>A configured <see cref="Hex1bTerminal"/>. Caller is responsible for disposal.</returns> | ||
| internal static Hex1bTerminal CreateDockerTestTerminal( |
| /// <param name="builder">The sequence builder.</param> | ||
| /// <param name="counter">The sequence counter.</param> | ||
| /// <param name="workspace">Optional workspace — when provided, cd into /workspace.</param> | ||
| internal static Hex1bTerminalInputSequenceBuilder PrepareDockerEnvironment( |
| /// For <see cref="DockerInstallMode.GaRelease"/>, downloads and installs from aspire.dev. | ||
| /// For <see cref="DockerInstallMode.PullRequest"/>, uses the PR install script. | ||
| /// </summary> | ||
| internal static Hex1bTerminalInputSequenceBuilder InstallAspireCliInDocker( |
Description
Adds Docker-based E2E CLI test infrastructure using Hex1b's
WithDockerContainerAPI. This makes E2E CLI tests portable across local dev machines and CI by running them inside ephemeral Docker containers built from a multi-stage Dockerfile.What's included
WithDockerContainerAPI)docker/Dockerfile.e2e: Multi-stage Dockerfile that can build the CLI from source (full AOT build with./build.sh --pack && --bundle) or skip source build for CI (where pre-built artifacts are available).dockerignore: Keeps Docker build context small by excludingartifacts/,.git/,.dotnet/, IDE filesDetectDockerInstallMode(),CreateDockerTestTerminal(),PrepareDockerEnvironment(),InstallAspireCliInDocker()inCliE2ETestHelpers.csDockerSmokeTests.cs: Basic smoke test that installs the Aspire CLI and verifiesaspire --versionDockerStagingChannelTests.cs: Port ofStagingChannelTestsexercisingaspire config set/get/deletefor staging channel settings — no cleanup needed since the container is ephemeralEnvironment detection
The test infrastructure detects the environment and adjusts installation accordingly:
get-aspire-cli-pr.shto install from PR workflow artifactsartifacts/bundle/aspire-*-linux-x64.tar.gzexists, uses the source-built CLIGoal
This is Phase 1 of making CLI E2E tests more portable. The Docker approach eliminates host environment pollution and makes tests reproducible. Future work will expand this pattern to more CLI E2E tests and integrate with CI workflows to pass pre-built artifacts.
Fixes # (issue)
Checklist