The Operation will be canceled. The next steps may not contain expected logs.

[tarekgh]

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=696281

Build leg reported

Build / browser-wasm linux Release LibraryTests / Send to Helix

Pull Request

dotnet/runtime#103009

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "The Operation will be canceled. The next steps may not contain expected logs.",
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=696281
Error message validated: [The Operation will be canceled. The next steps may not contain expected logs.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/4/2024 8:08:48 PM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
2980402	dotnet-dotnet	Run Tests	Log	#61373
2980401	dotnet-dotnet	Build	Log	#61225
1429091	dotnet/runtime	Download unified artifacts	Log	dotnet/runtime#128351
2980383	dotnet-sdk	Run Tests in Helix and non Helix in parallel	Log	#61229
1428998	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#54394
2980375	dotnet-sdk	Auto-baselining mode and SDL pre-checks (1ES PT)	Log	#61229
1429077	dotnet/arcade	Run Helix Tests	Log	dotnet/arcade#16856
1427864	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#66726
1429066	dotnet/arcade	Run Helix Tests	Log
1428438	dotnet/sdk	🟣 Run AoT Tests	Log	dotnet/sdk#54386
1427700	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#66748
2980193	dotnet-sdk	Run Tests in Helix and non Helix in parallel	Log	#61229
1429028	dotnet/aspnetcore	Run build.cmd helix target	Log
1428952	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#54393
1429027	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#58995
1429025	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#58995
1428453	dotnet/maui	Controls Border,BoxView,Brush,Button	Log
1429009	dotnet/roslyn	Test	Log
1428704	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#127818
1428940	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#66768
1428941	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#66768
1428869	dotnet/sdk	🟣 Run AoT Tests	Log	dotnet/sdk#54391
1428689	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#128414
1428659	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#128412
1428839	dotnet/roslyn	Test	Log
2980215	dotnet-dotnet	Build (centOSStreamContainer)	Log
2980174	dotnet-dotnet-docker	🟪 Test Images	Log
1428877	dotnet/arcade	Run Helix Tests	Log
1428821	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#54392
2980173	dotnet-maui	Run Integration Tests - RunOniOS_BlazorDebug	Log
1428864	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#66767
1428862	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#66767
1428238	dotnet/sdk	🟣 Run TestBuild Tests	Log
2980237	dotnet-roslyn	Test	Log
1428853	dotnet/maui	Controls (API 30) Border,BoxView,Brush,Button	Log
2980123	dotnet-maui	Run Integration Tests - RunOniOS_MauiReleaseTrimFull	Log
1428769	dotnet/aspnetcore	Run build.cmd helix target	Log
1428402	dotnet/runtime	Send to Helix	Log
1428713	dotnet/roslyn	Test	Log
1428527	dotnet/roslyn	Test	Log
1428766	dotnet/aspnetcore	Run build.cmd helix target	Log
1428794	dotnet/aspnetcore	Run build.cmd helix target	Log
1428767	dotnet/aspnetcore	Run build.cmd helix target	Log
1428759	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#66766
1427357	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#54132
1428761	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#66766
1428443	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#126947
1428187	dotnet/maui	Controls Layout	Log	dotnet/maui#34598
1428668	dotnet/roslyn	Test	Log
1428751	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#66765
1428753	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#66765
1428615	dotnet/sdk	🟣 Run TestBuild Tests	Log
1428393	dotnet/runtime	Send to Helix	Log	dotnet/runtime#128409
1428397	dotnet/runtime	Send tests to Helix (Unix)	Log
1428396	dotnet/runtime	Send tests to Helix (Unix)	Log
1428746	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#66443
1427870	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#127762
1428747	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#66443
2979871	dotnet-performance	Send job to Helix (Windows)	Log
1422496	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#66270
1428596	dotnet/roslyn	Test	Log
1428400	dotnet/runtime	Send to Helix	Log
1428578	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#54297
1428610	dotnet/roslyn	Test	Log	dotnet/roslyn#83807
1428224	dotnet/sdk	🟣 Run AoT Tests	Log	dotnet/sdk#54380
1428720	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#66759
1428263	dotnet/runtime	Send to Helix	Log	dotnet/runtime#127894
1428575	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#54336
1428721	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#66759
1428571	dotnet/roslyn	Test	Log
1428360	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#128147
2979855	dotnet-performance	Send job to Helix (Windows)	Log
1428739	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#54320
1428553	dotnet/sdk	🟣 Run AoT Tests	Log	dotnet/sdk#54390
1428388	dotnet/runtime	Publish Build Assets	Log	dotnet/runtime#128409
1428640	dotnet/maui	DeviceTestsAndroid (Windows)	Log
1428506	dotnet/aspnetcore	Run Quarantined Tests	Log	dotnet/aspnetcore#66658
1428618	dotnet/performance	Send job to Helix (Windows)	Log
1428458	dotnet/aspnetcore	Run Quarantined Tests	Log
1428605	dotnet/performance	Send job to Helix (Windows)	Log	dotnet/performance#5227
1428601	dotnet/performance	Send job to Helix (Windows)	Log
1428598	dotnet/performance	Send job to Helix (Windows)	Log
1428227	dotnet/runtime	Send tests to Helix (Unix)	Log
1428415	dotnet/roslyn	Test	Log
1428404	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#53971
1428061	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#66585
1428300	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#128384
1428341	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#124770
2980065	dotnet-dotnet	Reclaim disk space from unused simulator runtimes	Log
1428543	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#66756
1428540	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#66756
1428186	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#128403
1428541	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#66756
1428024	dotnet/maui	Controls Shell	Log
1428230	dotnet/runtime	Zip Build Assets	Log	dotnet/runtime#128252
1428144	dotnet/runtime	Send to Helix	Log	dotnet/runtime#125437
1428164	dotnet/runtime	Send to Helix	Log	dotnet/runtime#128347
1428508	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#66658
1428504	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#66658
1428505	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#66442
Displaying 100 of 3440 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
218	1016	3440

[dkurepa]

These appear to be timeout cancellations, and most of them are happening on the same runtime leg that uses OSX.1200.ARM64.Open, looking at grafana, they do indeed appear to be quite high. Will check with the team on what we can do about this

[dkurepa]

For pipelines experiencing timeouts on osx.1200.arm64.open we recommend using osx.14.arm64.open, as this queue has a lot more machines

[dkurepa]

There is also a number of Publish Using Darc timeouts, these are in 99% of cases AzDo Api issues. During the Publish Using Darc job, we queue up a Maestro Promotion pipeline, that downloads the artifacts from the build calling it, and publishes them where they need to go. While publishing is happening, the build pipeline keeps busy waiting for the promotion pipeline to be done, checking the status of it every 60 seconds. In cases where this times out, the build most likely did get promoted, the AzDo API just kept telling us the wrong thing

[ivanpovazan]

@dkurepa for context, the same timeouts are also caught with this tracking #1883
where I asked about potentially bumping the queues to osx.13.amd/arm64, would that suffice?
(bumping to osx.14 is not an option right now as we still want testing on osx.13 which is the minimum supported OS version)

[CarnaViire]

For the "known issues" folks: the timeout here is reported twice, making BA red despite the tracking issue. E.g.:

1️⃣ dotnet/runtime#117255

2️⃣ dotnet/runtime#117800

cc @AlitzelMendez