Skip to content

Add rebuild command and change detection for project resources#15133

Merged
mitchdenny merged 35 commits intorelease/13.2from
feature/14970-rebuild-command
Mar 13, 2026
Merged

Add rebuild command and change detection for project resources#15133
mitchdenny merged 35 commits intorelease/13.2from
feature/14970-rebuild-command

Conversation

@mitchdenny
Copy link
Copy Markdown
Member

@mitchdenny mitchdenny commented Mar 11, 2026

Description

Adds a rebuild command to .NET project resources and a change detection service that monitors source files and prompts users when a rebuild is needed.

Fixes #14970

Rebuild Command

  • Adds a "Rebuild" button to ProjectResource instances in the Aspire dashboard
  • Creates a hidden ProjectRebuilderResource (an ExecutableResource subclass) that runs dotnet build on demand
  • The rebuild flow: Stop resource → Run dotnet build → Restart resource
  • Build output is streamed into the main resource's console logs with a [build] prefix so users can see progress and errors inline
  • Build failures leave the resource stopped with an error state showing the exit code
  • The rebuilder uses ExplicitStartupAnnotation (only starts on demand) and is hidden from the dashboard

Change Detection Service

  • ProjectChangeDetectionService: a BackgroundService that monitors source files for running project resources
  • Queries MSBuild for the project's file closure (Compile + ProjectReference items) via dotnet msbuild -getItem JSON output
  • Periodically checks file timestamps (default 10s, configurable via ASPIRE_PROJECT_CHANGE_DETECTION_INTERVAL)
  • Shows a dashboard notification banner via IInteractionService when changes are detected, with a Rebuild button that triggers the rebuild directly
  • Also logs to the resource console when changes are detected
  • Includes 2-second debounce and single-notification-per-cycle logic
  • Runs automatically in run mode (no opt-in required)

Follow-up items

  • --no-build optimization after rebuild to avoid double-build on restart
  • Interaction with dotnet watch mode (potentially hide rebuild when watch is active)

Checklist

  • Is this feature complete?
    • Yes. Ready to ship.
    • No. Follow-up changes expected.
  • Are you including unit tests for the changes and scenario tests if relevant?
    • Yes
    • No
  • Did you add public API?
    • Yes
    • No
  • Does the change make any security assumptions or guarantees?
    • Yes
    • No
  • Does the change require an update in our Aspire docs?

Copilot AI review requested due to automatic review settings March 11, 2026 11:19
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 11, 2026

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/dotnet/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 15133

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://raw.githubusercontent.com/dotnet/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 15133"

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds a rebuild workflow for .NET ProjectResource instances (via a hidden rebuilder executable + dashboard command) and introduces an opt-in background service that detects source changes and notifies users that a rebuild is needed.

Changes:

  • Adds a rebuild lifecycle command for ProjectResource that stops the resource, runs dotnet build, forwards build logs, and restarts on success.
  • Introduces ProjectChangeDetectionService + supporting build/closure utilities to periodically detect file changes and notify via resource logs + dashboard notification (opt-in).
  • Adds localized command strings and test coverage for the new command and closure/build helper behavior.

Reviewed changes

Copilot reviewed 25 out of 26 changed files in this pull request and generated 7 comments.

Show a summary per file
File Description
tests/Aspire.Hosting.Tests/ResourceCommandAnnotationTests.cs Adds rebuild command state/description tests and verifies it’s only on project resources.
tests/Aspire.Hosting.Tests/Build/ProjectFileClosureTests.cs Adds tests for detecting changed files via captured timestamps.
tests/Aspire.Hosting.Tests/Build/ProjectBuildHelperTests.cs Adds basic tests around GetProjectFileClosureAsync error/cancellation behavior.
src/Aspire.Hosting/Resources/xlf/CommandStrings.zh-Hant.xlf Adds Rebuild localized string entries (new/untranslated).
src/Aspire.Hosting/Resources/xlf/CommandStrings.zh-Hans.xlf Adds Rebuild localized string entries (new/untranslated).
src/Aspire.Hosting/Resources/xlf/CommandStrings.tr.xlf Adds Rebuild localized string entries (new/untranslated).
src/Aspire.Hosting/Resources/xlf/CommandStrings.ru.xlf Adds Rebuild localized string entries (new/untranslated).
src/Aspire.Hosting/Resources/xlf/CommandStrings.pt-BR.xlf Adds Rebuild localized string entries (new/untranslated).
src/Aspire.Hosting/Resources/xlf/CommandStrings.pl.xlf Adds Rebuild localized string entries (new/untranslated).
src/Aspire.Hosting/Resources/xlf/CommandStrings.ko.xlf Adds Rebuild localized string entries (new/untranslated).
src/Aspire.Hosting/Resources/xlf/CommandStrings.ja.xlf Adds Rebuild localized string entries (new/untranslated).
src/Aspire.Hosting/Resources/xlf/CommandStrings.it.xlf Adds Rebuild localized string entries (new/untranslated).
src/Aspire.Hosting/Resources/xlf/CommandStrings.fr.xlf Adds Rebuild localized string entries (new/untranslated).
src/Aspire.Hosting/Resources/xlf/CommandStrings.es.xlf Adds Rebuild localized string entries (new/untranslated).
src/Aspire.Hosting/Resources/xlf/CommandStrings.de.xlf Adds Rebuild localized string entries (new/untranslated).
src/Aspire.Hosting/Resources/xlf/CommandStrings.cs.xlf Adds Rebuild localized string entries (new/untranslated).
src/Aspire.Hosting/Resources/CommandStrings.resx Adds RebuildName and RebuildDescription strings.
src/Aspire.Hosting/Resources/CommandStrings.Designer.cs Updates the generated resource accessors for rebuild strings.
src/Aspire.Hosting/ProjectResourceBuilderExtensions.cs Creates a hidden ProjectRebuilderResource for project resources in run mode.
src/Aspire.Hosting/DistributedApplicationBuilder.cs Registers the new change detection hosted service in run mode.
src/Aspire.Hosting/Build/ProjectFileClosure.cs Implements timestamp-based closure and change detection helper.
src/Aspire.Hosting/Build/ProjectChangeDetectionService.cs Adds opt-in background monitoring of running projects and user notifications.
src/Aspire.Hosting/Build/ProjectBuildHelper.cs Adds MSBuild -getItem JSON parsing + directory-scan fallback to build a closure.
src/Aspire.Hosting/ApplicationModel/ProjectRebuilderResource.cs Adds the hidden executable resource type used for rebuilds.
src/Aspire.Hosting/ApplicationModel/KnownResourceCommands.cs Introduces the rebuild command name constant.
src/Aspire.Hosting/ApplicationModel/CommandsConfigurationExtensions.cs Wires the rebuild command into lifecycle commands and forwards build logs.
Files not reviewed (1)
  • src/Aspire.Hosting/Resources/CommandStrings.Designer.cs: Language not supported

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 11, 2026

🎬 CLI E2E Test Recordings

The following terminal recordings are available for commit 7d51c9c:

Test Recording
AddPackageInteractiveWhileAppHostRunningDetached ▶️ View Recording
AddPackageWhileAppHostRunningDetached ▶️ View Recording
AgentCommands_AllHelpOutputs_AreCorrect ▶️ View Recording
AgentInitCommand_DefaultSelection_InstallsSkillOnly ▶️ View Recording
AgentInitCommand_MigratesDeprecatedConfig ▶️ View Recording
AspireAddPackageVersionToDirectoryPackagesProps ▶️ View Recording
AspireUpdateRemovesAppHostPackageVersionFromDirectoryPackagesProps ▶️ View Recording
Banner_DisplayedOnFirstRun ▶️ View Recording
Banner_DisplayedWithExplicitFlag ▶️ View Recording
CertificatesClean_RemovesCertificates ▶️ View Recording
CertificatesTrust_WithNoCert_CreatesAndTrustsCertificate ▶️ View Recording
CertificatesTrust_WithUntrustedCert_TrustsCertificate ▶️ View Recording
CreateAndDeployToDockerCompose ▶️ View Recording
CreateAndDeployToDockerComposeInteractive ▶️ View Recording
CreateAndPublishToKubernetes ▶️ View Recording
CreateAndRunAspireStarterProject ▶️ View Recording
CreateAndRunAspireStarterProjectWithBundle ▶️ View Recording
CreateAndRunJsReactProject ▶️ View Recording
CreateAndRunPythonReactProject ▶️ View Recording
CreateAndRunTypeScriptStarterProject ▶️ View Recording
CreateEmptyAppHostProject ▶️ View Recording
CreateStartAndStopAspireProject ▶️ View Recording
CreateTypeScriptAppHostWithViteApp ▶️ View Recording
DescribeCommandResolvesReplicaNames ▶️ View Recording
DescribeCommandShowsRunningResources ▶️ View Recording
DetachFormatJsonProducesValidJson ▶️ View Recording
DoctorCommand_DetectsDeprecatedAgentConfig ▶️ View Recording
DoctorCommand_WithSslCertDir_ShowsTrusted ▶️ View Recording
DoctorCommand_WithoutSslCertDir_ShowsPartiallyTrusted ▶️ View Recording
LogsCommandShowsResourceLogs ❌ Upload failed
PsCommandListsRunningAppHost ❌ Upload failed
PsFormatJsonOutputsOnlyJsonToStdout ▶️ View Recording
RestoreGeneratesSdkFiles ▶️ View Recording
RunWithMissingAwaitShowsHelpfulError ▶️ View Recording
SecretCrudOnDotNetAppHost ▶️ View Recording
SecretCrudOnTypeScriptAppHost ▶️ View Recording
StagingChannel_ConfigureAndVerifySettings_ThenSwitchChannels ▶️ View Recording
StopAllAppHostsFromAppHostDirectory ▶️ View Recording
StopAllAppHostsFromUnrelatedDirectory ▶️ View Recording
StopNonInteractiveMultipleAppHostsShowsError ▶️ View Recording
StopNonInteractiveSingleAppHost ▶️ View Recording
StopWithNoRunningAppHostExitsSuccessfully ▶️ View Recording
TypeScriptAppHostWithProjectReferenceIntegration ▶️ View Recording

📹 Recordings uploaded automatically from CI run #23032713812

Mitch Denny and others added 16 commits March 12, 2026 12:24
Implements the core rebuild command feature from issue #14970.

Key changes:
- Add 'rebuild' to KnownResourceCommands
- Create ProjectRebuilderResource (hidden ExecutableResource that runs dotnet build)
- Register rebuilder resource per ProjectResource in WithProjectDefaults (run mode only)
  - Uses ExplicitStartupAnnotation so it doesn't auto-start
  - Hidden from dashboard, excluded from manifest
- Add rebuild command to CommandsConfigurationExtensions for ProjectResource
  - Stops main resource, starts rebuilder, funnels build logs to main resource console
  - Waits for build completion, restarts main resource on success
- Add command string resources and localization entries
- Add tests for rebuild command state transitions and registration

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds a background service that monitors source files for project resources
and notifies users when a rebuild is needed.

Key changes:
- ProjectBuildHelper: queries MSBuild for project file closure (Compile items,
  ProjectReference items) and records timestamps. Falls back to directory scan
  if MSBuild JSON parsing fails.
- ProjectFileClosure: model for the file set with GetChangedFiles() method.
- ProjectChangeDetectionService: BackgroundService that watches for project
  resources reaching Running state, captures their file closure, and
  periodically checks for timestamp changes. Shows notification via
  IInteractionService and logs to the resource console.
- Opt-in via ASPIRE_PROJECT_CHANGE_DETECTION=true env var.
- Configurable interval via ASPIRE_PROJECT_CHANGE_DETECTION_INTERVAL (default 10s).
- Includes debouncing (2s after last change before notifying).
- Registered in DistributedApplicationBuilder (run mode only).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Unit tests for ProjectFileClosure.GetChangedFiles() covering:
- No changes detected
- Single file modified
- File deleted (graceful handling)
- Multiple files with partial changes
- Empty closure
- Non-existent file paths

Integration tests for ProjectBuildHelper:
- Non-existent project path (graceful null return)
- Cancellation handling

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The Aspire.Hosting.Build namespace collided with the Build type in
Aspire.Hosting.Docker/Resources/ComposeNodes/Service.cs, causing
CS0118: 'Build' is a namespace but is used like a type.

Renamed to Aspire.Hosting.Rebuild to avoid the conflict.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
No need for a separate sub-namespace for these internal types.
Removes the Rebuild/ subdirectory and places files directly in
the Aspire.Hosting namespace.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Capture the ProjectResource from the closure directly instead of
looking it up by context.ResourceName, which is the instance name
(e.g. 'myproject-0') rather than the resource name ('myproject').

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The orchestrator's StartResourceAsync expects DCP instance names
(from GetResolvedResourceNames), not model resource names. Using
the model name caused 'resource not found' errors.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The --project flag is a dotnet CLI flag, not an MSBuild flag.
When DCP invokes the executable, it passes args directly to
MSBuild which doesn't recognize --project. Pass the project
path as a positional argument instead.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Rewrite ProjectBuildHelper to follow the CLI's DotNetCliRunner pattern:
- Use 'dotnet msbuild' for evaluation (not 'dotnet build')
- Add -getProperty:MSBuildVersion workaround for MSBuild bug #12490
  (single property query doesn't return valid JSON)
- Add retry logic (3 attempts) for MSBuild server contention that
  can produce exit code 0 but empty output
- Prefer FullPath over Identity for item paths when available
- Better separation: EvaluateMSBuildAsync handles process/retry,
  ParseEvaluationOutput handles JSON parsing

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Three fixes:
- Use configuration.GetBool() instead of GetValue<bool>() to match
  codebase conventions and properly parse env var string values
- Check interactionService.IsAvailable before calling
  PromptNotificationAsync (throws if dashboard not connected)
- Pass NotificationInteractionOptions with Intent=Information so
  the banner renders with proper styling
- Wrap notification call in try/catch to prevent background task crash

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The AddsDefaultsCommandsToResources test expected 3 commands (start,
stop, restart) for all resource types, but project resources now also
have a rebuild command. Added HasKnownProjectCommandAnnotations helper
to validate the 4-command set for project resources.

Also improved notification logging in ProjectChangeDetectionService.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Change detection should always run in run mode, not require an
explicit ASPIRE_PROJECT_CHANGE_DETECTION=true opt-in.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Raise key log messages from Debug to Information level so users can
see change detection activity in the AppHost console output:
- File closure capture start and result
- Change detection events
- Notification dispatch

This helps diagnose whether the service is running, detecting changes,
and sending notifications.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The notification banner now includes a 'Rebuild' button that triggers
the rebuild command directly when clicked, instead of just telling the
user to use the command manually.

Uses ResourceCommandService.ExecuteCommandAsync to invoke the rebuild
command on the project resource when the user clicks the button.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… test assertions

- Clamp change detection interval to > 0 to prevent zero/negative delay
- Use KnownResourceStates.FailedToStart instead of custom 'Build failed' string
  so dashboard command enablement works correctly
- Replace always-true test assertions with explicit Assert.Null checks

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add feature flag: features:enableDotNetProjectChangeDetection (default true)
  Set via: aspire config set -g features.enableDotNetProjectChangeDetection false
- Add interval setting: dotNetProjectChangeDetectionIntervalSeconds (default 5s)
  Set via: aspire config set -g dotNetProjectChangeDetectionIntervalSeconds 100
- Add constants to KnownConfigNames for both settings
- Extract DefaultCheckIntervalSeconds constant (5 seconds)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mitchdenny mitchdenny force-pushed the feature/14970-rebuild-command branch from 186179d to 4ff4e29 Compare March 12, 2026 01:24
Mitch Denny and others added 4 commits March 12, 2026 13:04
Remove the auto-detection of source file changes (banner notifications,
dotnet msbuild evaluation, file closure tracking). Keep only the rebuild
command itself for now.

Removed files:
- ProjectChangeDetectionService.cs
- ProjectBuildHelper.cs
- ProjectFileClosure.cs
- ProjectBuildHelperTests.cs
- ProjectFileClosureTests.cs

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Move the rebuild command logic out of AddLifeCycleCommands into its own
AddRebuildCommand method. Reorder RebuildCommand constant to follow
RestartCommand in KnownResourceCommands.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Reword RebuildDescription to be shorter
- Localize 'rebuilder not found' error message via CommandStrings.resx
- Add KnownResourceStates.Building constant, replace magic string
- Set Building state BEFORE stopping to prevent double-rebuild race
- Add 10-minute timeout for build completion (hung build protection)
- Handle cancellation gracefully without logging 'Build failed'
- Extract StopLogForwardingAsync helper

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@JamesNK
Copy link
Copy Markdown
Member

JamesNK commented Mar 12, 2026

The status goes to Finished during rebuild. It would be better if there was a build status displayed in the dashboard.

rebuild-status

@mitchdenny
Copy link
Copy Markdown
Member Author

Summary of additional fixes beyond the rebuild command

While implementing and testing the rebuild command with replicated resources (.WithReplicas(2) + .WaitFor()), we discovered two pre-existing bugs in the hosting infrastructure that became visible during rebuild-triggered restarts. These bugs exist on release/13.2 today and affect any scenario where individual replicas are started or stopped independently — rebuild just makes them easy to reproduce.


Bug 1: WaitForDependenciesAsync clobbers Running replicas to "Waiting"

Files changed: ResourceNotificationService.cs

Root cause: WaitUntilStateAsync and WaitUntilCompletionAsync both call model-level PublishUpdateAsync(resource, s => s with { State = "Waiting" }) to indicate a resource is waiting for its dependencies. The model-level PublishUpdateAsync overload iterates ALL resolved resource names (all replicas) and applies the state factory to each one. This means starting replica N sets ALL replicas to "Waiting" — including replica N-1 which may already be Running.

Why this is permanent: The DCP status watcher (OnResourceChanged) publishes per-replica state updates when DCP resources change state. Once a replica reaches Running and the watcher has published that event, no further DCP events arrive unless the DCP resource state changes again. When a subsequent replica's WaitForDependenciesAsync clobbers the first replica back to "Waiting", there is no DCP event to correct the stale state. The replica is stuck in "Waiting" forever.

The event chain (traced through the code):

  1. orchestrator.StartResourceAsync(replicaName) checks if state is "Waiting" — if not, calls DcpExecutor.StartResourceAsync
  2. DcpExecutor.StartResourceAsync publishes OnResourceStartingContext
  3. ApplicationOrchestrator.OnResourceStarting sets THIS replica to "Starting" (per-replica update), then publishes BeforeResourceStartedEvent
  4. WaitForInBeforeResourceStartedEvent calls WaitForDependenciesAsyncWaitUntilStateAsync
  5. THE BUG: WaitUntilStateAsync calls PublishUpdateAsync(resource, s => s with { State = "Waiting" }) — model-level broadcast to ALL replicas

Repro without rebuild: Start an AppHost with a resource that has .WithReplicas(2) and .WaitFor(someResource). Stop one replica. Start it again. The other replica may get clobbered to "Waiting".

Fix: Guard the state factory to only transition replicas that are actually in the startup phase (state is null, Starting, or Waiting). Replicas already in Running or terminal states keep their current state. This is a targeted fix at the two call sites (lines 278 and 342) — no signature changes, no new overloads.

await PublishUpdateAsync(resource, s =>
    s.State?.Text is null
    || s.State?.Text == KnownResourceStates.Starting
    || s.State?.Text == KnownResourceStates.Waiting
        ? s with { State = KnownResourceStates.Waiting }
        : s).ConfigureAwait(false);

Bug 2: Health monitor torn down when one replica stops

Files changed: ResourceHealthCheckService.cs

Root cause: ResourceHealthCheckService maintains a dictionary of health monitors keyed by resourceEvent.Resource.Name (the model name, e.g. "api"). However, WatchAsync emits per-replica events (e.g. "api-0", "api-1"). When ANY replica enters a terminal state, the service tears down the single health monitor for the model name — stopping health evaluation for ALL replicas, including those still Running.

Why this causes oscillation:

  1. api-0 enters Exited (terminal) → monitor for "api" torn down → health checks stop
  2. Next event for api-1 (still Running) → monitor recreated → health checks resume
  3. Health check publishes results → triggers state updates → api-0 event arrives (still Exited)
  4. Monitor torn down again → cycle repeats

On release/13.2, this bug was masked by Bug 1: the "Waiting" clobber set the stopped replica back to "Waiting" (which is non-terminal), so the monitor was never torn down. Our fix for Bug 1 exposed this bug because the stopped replica now correctly stays in its terminal state.

Fix: Before tearing down the health monitor, check if ALL replicas of the resource are in terminal states. Only tear down when none are still running. Health checks continue through the shared DCP port as long as any replica is alive.

var allReplicasTerminal = resource.GetResolvedResourceNames().All(name =>
    resourceNotificationService.TryGetCurrentState(name, out var evt)
    && KnownResourceStates.TerminalStates.Contains(evt.Snapshot.State?.Text));

if (allReplicasTerminal)
{
    state.StopResourceMonitor();
    // ...
}

Why these fixes are in this PR

Both bugs are pre-existing on release/13.2 and affect any replicated resource with health checks or .WaitFor() dependencies. The rebuild command exercises the restart-replica-after-build path, which reliably triggers both bugs. Without these fixes, rebuild with replicas is broken. The fixes are small, targeted, and improve the overall replica experience beyond just rebuild.

… rebuilds

- Start, stop, and restart commands are now disabled when the resource
  is in Building state, preventing interference during a rebuild.
- When a rebuild succeeds but no replicas were running (resource was
  stopped), restore each replica to its pre-build state instead of
  leaving it stuck in Building.
- Added test cases for Building state on start, stop, and restart
  command state callbacks.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@JamesNK
Copy link
Copy Markdown
Member

JamesNK commented Mar 12, 2026

I forgot to mention, projects in a waiting state should probably be rebuildable.

Resources stuck in Waiting (e.g. dependency not healthy) should be
rebuildable. Added Waiting to BuildableStates and treat Waiting
replicas as active for post-rebuild restart.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copy link
Copy Markdown
Member

@JamesNK JamesNK left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few issues found.

@dotnet-policy-service dotnet-policy-service bot added the needs-author-action An issue or pull request that requires more info or actions from the author. label Mar 12, 2026
@JamesNK
Copy link
Copy Markdown
Member

JamesNK commented Mar 12, 2026

Rebuilding waiting resources fails:

Waiting for resource 'messaging' to enter the 'Running' state.
Executing command 'rebuild'.
[build] Stopping resource for rebuild...
 Error executing command 'rebuild'.
k8s.Autorest.HttpOperationException: Operation returned an invalid status code 'NotFound', response body {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"executables.usvc-dev.developer.microsoft.com \"orderprocessor-wuyntwbf\" not found","reason":"NotFound","details":{"name":"orderprocessor-wuyntwbf","group":"usvc-dev.developer.microsoft.com","kind":"executables"},"code":404}

   at k8s.Kubernetes.SendRequestRaw(String requestContent, HttpRequestMessage httpRequest, CancellationToken cancellationToken)
   at k8s.AbstractKubernetes.ICustomObjectsOperations_PatchClusterCustomObjectWithHttpMessagesAsync[T](Object body, String group, String version, String plural, String name, String dryRun, String fieldManager, String fieldValidation, Nullable`1 force, IReadOnlyDictionary`2 customHeaders, CancellationToken cancellationToken)
   at k8s.AbstractKubernetes.k8s.ICustomObjectsOperations.PatchClusterCustomObjectWithHttpMessagesAsync(Object body, String group, String version, String plural, String name, String dryRun, String fieldManager, String fieldValidation, Nullable`1 force, IReadOnlyDictionary`2 customHeaders, CancellationToken cancellationToken)
   at Aspire.Hosting.Dcp.KubernetesService.<>c__DisplayClass18_0`1.<<PatchAsync>b__0>d.MoveNext() in C:\Development\Source\aspire\src\Aspire.Hosting\Dcp\KubernetesService.cs:line 170
--- End of stack trace from previous location ---
   at Aspire.Hosting.Dcp.KubernetesService.<>c__DisplayClass28_0`1.<<ExecuteWithRetry>b__0>d.MoveNext() in C:\Development\Source\aspire\src\Aspire.Hosting\Dcp\KubernetesService.cs:line 483
--- End of stack trace from previous location ---
   at Polly.ResiliencePipeline.<>c__10`1.<<ExecuteAsync>b__10_0>d.MoveNext()
--- End of stack trace from previous location ---
   at Polly.Outcome`1.GetResultOrRethrow()
   at Polly.ResiliencePipeline.ExecuteAsync[TResult](Func`2 callback, CancellationToken cancellationToken)
   at Aspire.Hosting.Dcp.KubernetesService.ExecuteWithRetry[TResult](DcpApiOperationType operationType, String resourceType, Func`2 operation, Func`2 isRetryable, CancellationToken cancellationToken) in C:\Development\Source\aspire\src\Aspire.Hosting\Dcp\KubernetesService.cs:line 480
   at Aspire.Hosting.Dcp.DcpExecutor.<>c__DisplayClass88_0.<<StopResourceAsync>b__0>d.MoveNext() in C:\Development\Source\aspire\src\Aspire.Hosting\Dcp\DcpExecutor.cs:line 2493
--- End of stack trace from previous location ---
   at Polly.ResiliencePipeline.<>c__9`2.<<ExecuteAsync>b__9_0>d.MoveNext()
--- End of stack trace from previous location ---
   at Polly.Outcome`1.GetResultOrRethrow()
   at Polly.ResiliencePipeline.ExecuteAsync[TResult,TState](Func`3 callback, TState state, CancellationToken cancellationToken)
   at Aspire.Hosting.Dcp.DcpExecutor.StopResourceAsync(IResourceReference resourceReference, CancellationToken cancellationToken) in C:\Development\Source\aspire\src\Aspire.Hosting\Dcp\DcpExecutor.cs:line 2471
   at Aspire.Hosting.Orchestrator.ApplicationOrchestrator.StopResourceAsync(String resourceName, CancellationToken cancellationToken) in C:\Development\Source\aspire\src\Aspire.Hosting\Orchestrator\ApplicationOrchestrator.cs:line 593
   at Aspire.Hosting.ApplicationModel.CommandsConfigurationExtensions.ExecuteRebuildAsync(ExecuteCommandContext context, ProjectResource projectResource) in C:\Development\Source\aspire\src\Aspire.Hosting\ApplicationModel\CommandsConfigurationExtensions.cs:line 206
   at Aspire.Hosting.ApplicationModel.ResourceCommandService.ExecuteCommandCoreAsync(String resourceId, IResource resource, String commandName, CancellationToken cancellationToken) in C:\Development\Source\aspire\src\Aspire.Hosting\ApplicationModel\ResourceCommandService.cs:line 161

Repo tests:

  1. Stop docker
  2. Run TestShop
  3. orderprocessor should be in a waiting state because messaging isn't running
  4. Rebuild orderprocessor
  5. See error

Mitch Denny and others added 2 commits March 13, 2026 09:41
File-based apps (.cs files via AddCSharpApp) rebuild automatically on
restart, so the explicit Rebuild command is unnecessary and confusing.

The guard in AddRebuilderResource already skipped creating the hidden
rebuilder resource for file-based apps, but AddRebuildCommand in
CommandsConfigurationExtensions was called unconditionally for all
ProjectResource types. This adds the same IsFileBasedApp check to
prevent the command annotation from being added.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add outer catch for context cancellation in ExecuteRebuildAsync so the
  resource state is restored to Exited instead of being stuck in Building
  forever with all commands disabled.
- Remove dead code (unreachable context.CancellationToken check).
- Fix RebuilderResourceNotFound XLF entries: correct indentation from
  8-space to 6-space, remove extra translate/xml:space attributes.
- Restore trailing newlines on resx and all 13 XLF files.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mitchdenny
Copy link
Copy Markdown
Member Author

Addressed all three feedback items in c7c0cb9:

1. Building state stuck on cancellation — Added an outer catch (OperationCanceledException) when (context.CancellationToken.IsCancellationRequested) block that sets the resource state to Exited and returns an error result. This handles cancellation at any point in the rebuild flow (during StartResourceAsync, WatchAsync, or replica restart). Also removed the dead code (if (context.CancellationToken.IsCancellationRequested) check that was unreachable).

2. XLF formatting — Fixed all 13 XLF files: corrected indentation from 8-space to 6-space for the RebuilderResourceNotFound entry, and removed the extra translate="yes" xml:space="preserve" attributes that were artifacts of manual editing.

3. Trailing newlines — Restored trailing newlines on the .resx file and all 13 XLF files.

@dotnet-policy-service dotnet-policy-service bot removed the needs-author-action An issue or pull request that requires more info or actions from the author. label Mar 12, 2026
@mitchdenny
Copy link
Copy Markdown
Member Author

Re: Should this be Finished?

Good question. The docs (polyglot-apphost-testing.md) define the convention as: Exited is for executables/projects, Finished is for containers. Since this is a project resource, Exited is the correct one by convention.

That said, looking at the codebase, the two states are functionally identical — same dashboard rendering (stop icon, info color), same terminal-state behavior, no branching logic differentiates them. Could be worth considering consolidating them in a future PR, but that would touch containers, projects, dashboard, and docs so probably not in scope here.

Happy to switch to Finished if you prefer though — it would behave identically.

Mitch Denny and others added 3 commits March 13, 2026 10:46
Change cancellation handler to use KnownResourceStates.Finished
instead of Exited per PR feedback.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Skip stopping and restarting replicas that are in Waiting state during
rebuild. Waiting replicas have no DCP Executable to stop (their lifecycle
is blocked at WaitForInBeforeResourceStartedEvent). The build output on
disk is updated, so when dependencies become ready the new binary is
launched automatically.

Also avoid setting Building state on Waiting replicas, which would
unblock WaitForInBeforeResourceStartedEvent and launch the old binary
while the build is in progress.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
When a resource is rebuilt from a stopped/terminal state, the stop
command was transitioning from Hidden to Disabled, causing it to
briefly appear (pulsing) in the dashboard. Since there is no process
to stop during a build, the stop button should remain Hidden.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mitchdenny mitchdenny requested a review from JamesNK March 13, 2026 01:24
@JamesNK
Copy link
Copy Markdown
Member

JamesNK commented Mar 13, 2026

Everything is working. However, I see this error in apphost:

image

basketservice-rebuilder-udhatzzq resource is not found.

@JamesNK
Copy link
Copy Markdown
Member

JamesNK commented Mar 13, 2026

fail: Aspire.Hosting.Dcp.DcpExecutor[0]
      Error streaming logs for basketservice-rebuilder-udhatzzq.
      k8s.Autorest.HttpOperationException: Operation returned an invalid status code 'NotFound', response body {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"executables.usvc-dev.developer.microsoft.com \"basketservice-rebuilder-udhatzzq\" not found","reason":"NotFound","details":{"name":"basketservice-rebuilder-udhatzzq","group":"usvc-dev.developer.microsoft.com","kind":"executables"},"code":404}

         at k8s.Kubernetes.SendRequestRaw(String requestContent, HttpRequestMessage httpRequest, CancellationToken cancellationToken)
         at Aspire.Hosting.Dcp.DcpKubernetesClient.ReadSubResourceAsStreamAsync(String group, String version, String plural, String name, String subResource, String namespaceParameter, IReadOnlyCollection`1 queryParams, CancellationToken cancellationToken) in C:\Development\Source\aspire\src\Aspire.Hosting\Dcp\DcpKubernetesClient.cs:line 76
         at Aspire.Hosting.Dcp.KubernetesService.<>c__DisplayClass22_0`1.<<GetLogStreamAsync>b__0>d.MoveNext() in C:\Development\Source\aspire\src\Aspire.Hosting\Dcp\KubernetesService.cs:line 345
      --- End of stack trace from previous location ---
         at Aspire.Hosting.Dcp.KubernetesService.<>c__DisplayClass28_0`1.<<ExecuteWithRetry>b__0>d.MoveNext() in C:\Development\Source\aspire\src\Aspire.Hosting\Dcp\KubernetesService.cs:line 483
      --- End of stack trace from previous location ---
         at Polly.ResiliencePipeline.<>c__10`1.<<ExecuteAsync>b__10_0>d.MoveNext()
      --- End of stack trace from previous location ---
         at Polly.Outcome`1.GetResultOrRethrow()
         at Polly.ResiliencePipeline.ExecuteAsync[TResult](Func`2 callback, CancellationToken cancellationToken)
         at Aspire.Hosting.Dcp.KubernetesService.ExecuteWithRetry[TResult](DcpApiOperationType operationType, String resourceType, Func`2 operation, Func`2 isRetryable, CancellationToken cancellationToken) in C:\Development\Source\aspire\src\Aspire.Hosting\Dcp\KubernetesService.cs:line 480
         at Aspire.Hosting.Dcp.ResourceLogSource`1.GetAsyncEnumerator(CancellationToken cancellationToken)+MoveNext() in C:\Development\Source\aspire\src\Aspire.Hosting\Dcp\ResourceLogSource.cs:line 91
         at Aspire.Hosting.Dcp.ResourceLogSource`1.GetAsyncEnumerator(CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
         at Aspire.Hosting.Dcp.DcpExecutor.<>c__DisplayClass54_1`1.<<StartLogStream>b__1>d.MoveNext() in C:\Development\Source\aspire\src\Aspire.Hosting\Dcp\DcpExecutor.cs:line 661
      --- End of stack trace from previous location ---
         at Aspire.Hosting.Dcp.DcpExecutor.<>c__DisplayClass54_1`1.<<StartLogStream>b__1>d.MoveNext() in C:\Development\Source\aspire\src\Aspire.Hosting\Dcp\DcpExecutor.cs:line 661

When a short-lived resource (like the rebuilder) is deleted while its log
stream is still active with follow: true, the DCP API returns 404 NotFound.
Previously this was caught by the generic Exception handler and logged as
an error. Now it's caught specifically and logged at Debug level, since
resource deletion during log streaming is a normal lifecycle event.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mitchdenny
Copy link
Copy Markdown
Member Author

Re: error output in apphost logs (basketservice-rebuilder 404 NotFound)

Root cause: The rebuilder is a short-lived DCP executable. When dotnet build completes, the process exits but the DCP resource persists with follow: true log streaming still active. When the resource is later deleted (on the next rebuild via EnsureResourceDeletedAsync, or during app shutdown), the follow stream's HTTP connection gets a 404 NotFound. Previously this hit the generic catch (Exception) in StartLogStream and was logged as an error.

Fix: Added a specific catch for HttpOperationException with NotFound status in DcpExecutor.StartLogStream — downgrades it from LogError to LogDebug since resource deletion during log streaming is a normal lifecycle event for short-lived resources. This is the same pattern already used elsewhere in DcpExecutor (e.g., EnsureResourceDeletedAsync).

Pushed in 7d51c9c.

@mitchdenny mitchdenny merged commit fddaee5 into release/13.2 Mar 13, 2026
252 checks passed
@mitchdenny mitchdenny deleted the feature/14970-rebuild-command branch March 13, 2026 02:33
@dotnet-policy-service dotnet-policy-service bot added this to the 13.2 milestone Mar 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Expose a rebuild command on project resources

5 participants