[✨ Triage] dotnet/runtime#125739 by richlander - macOS 26 queues are stale

[MihuBot]

Triage for dotnet/runtime#125739.
Repo filter: All networking issues.
MihuBot version: 246635.
Ping MihaZupan for any issues.

This is a test triage report generated by AI, aimed at helping the triage team quickly identify past issues/PRs that may be related.
Take any conclusions with a large grain of salt.

Tool logs

dotnet/runtime#125739: macOS 26 queues are stale by richlander
Extracted 5 search queries: osx.26.arm64.open queue running macOS-26.0.1 outdated image, macOS 26 CI queues stale missing security fixes need monthly updates, System.Net.NetworkInformation.Functional.Tests on macOS-26 failing due to old runner image, dci-macm1-build-003 macOS-26.0.1 CI machine stale image, update macOS hosted queues to newer macOS 26 builds pipeline security patches
Found 25 candidate issues

• PR #122990 (Jan 2026) - "Bump MacOS helix queues to latest"

  • Updated helix queue references in runtime pipelines from macOS 13 -> 14 and then moved to macOS 26 in follow-up edits. Discussion notes:
    • Richlander questioned whether testing macOS 14 is sufficient and argued macOS 26 is the higher priority.
    • Agocke noted there are no 26.Amd64 queues, only ARM64, which affects coverage.
    • Several legs initially failed after bumps (osx failures), pointing to missing backports and environmental differences; fixes/adjustments and darc/xharness updates were needed.
  • Relevance: shows the repo-level effort to bump queues to newer macOS versions and highlights the ARM64-only limitation for macOS 26 and that bumping can reveal test/environment gaps.

• PR #123766 (Jan 2026) - "[release/9.0] Bump macos helix queues"

  • Backport/servicing branch bump from retired 13 queues to newer macOS queue versions (OSX.13 -> OSX.15 in that PR).
  • Relevance: demonstrates servicing-branch queue updates are applied across branches; underscores the need to keep servicing queues current to avoid CI failures when older queues are retired.

• PR #119816 (Sept 2025) - "Bump OSX.12 helix queues to OSX.13"

  • Infra-only change to move OSX.12 queue references to OSX.13.
  • Discussion surfaced immediate test failures on the new images (notably OpenSSL-related and MsQuic/QUIC tests). Team discussion concluded:
    • Some tests depend on OpenSSL being available; on newer macOS / Apple Silicon environments OpenSSL is problematic.
    • Main branch had dropped OpenSSL-on-mac support; backport decisions were required (disable tests or backport changes).
  • Relevance: shows that bumping macOS queues can cause test regressions and that there are non-trivial dependencies (OpenSSL, MsQuic) and platform-specific issues to consider when updating queues.

• PR #119814 (Sept 2025) - (same timeframe) "Bump OSX.12 helix queues to OSX.13" (paired infra change)

  • Complementary change which triggered the same discussions about OpenSSL and QUIC. Team debated backporting fixes vs. disabling tests.
  • Relevance: further evidence that queue bumps surface platform/test dependency issues and that changes across branches need coordination.

• PR #113399 (Mar 2025) - "Update Mac versions in helix queues"

  • Earlier attempt to update macOS queue versions; conversation called out cryptography test failures on macOS and the difficulty of making OpenSSL work on newer macOS images.
  • Note in thread: "Never mind, this is stale" — some attempts were abandoned or superseded.
  • Relevance: historical context for macOS queue updates and the recurring OpenSSL problem.

• PR #109454 (Nov 2024) - "Upgrade our macOS build machines to the latest image"

  • Changed hosted images used by CI (macOS images) and encountered linker/toolset issues (Xcode-related), leading to reverting to macOS 13 in that change. Discussion emphasized tradeoffs (Xcode/toolset compatibility vs. staying latest).
  • Relevance: shows that moving to newer macOS images can require toolchain adjustments and may break builds; underscores necessity of measured cadence and testing when changing macOS images.

• Issue #122709 (Dec 2025) - "Build failures: macOS-13 based runner images are now retired"

  • Known-build-error report: macOS-13 runner images retired; CI failures produced a known-issue entry. The automation tracked hit counts and guidance for builds that used retired images.
  • Relevance: confirms that macOS queue retirement happens and requires repos to update queue references to avoid CI breakage.

• PRs #52327 / #52336 (Jan 2026) - "Update osx arm64 helix images" and backport to main

  • SDK repo changes to move away from osx.13.arm64.open (Helix queue retirement notice). Team noted this needed to be changed on all branches using that helix queue and that the change should be on main as well.
  • Relevance: another repo-level fix showing urgency when a specific macOS queue (osx.13.arm64.open) is scheduled/removed; mirrors the runtime repo problem.

• PR #24612 (Feb 2026) - dotnet/macios: "[CI] Bump to use macOS Tahoe"

  • macios CI moved to macOS Tahoe (macOS 26) and ran tests; multiple test runs and reimage activity followed. Machine re-imaging to Xcode/macOS 26 was noted; later comment indicates machines were re-imaged and more work to stabilize was in flight.
  • Relevance: shows platform team's effort to adopt macOS 26 on internal CI and that re-imaging and validation are part of keeping queues current.

• PR #24169 (Nov 2025) - dotnet/macios: "[xcode26.1] Update to stable bits"

  • Updates and test runs across macOS 26 on macios CI; intermittent failures surfaced but ultimately machines were re-imaged and later runs passed.
  • Relevance: shows operational work (re-imaging, Xcode updates) needed to keep macOS 26-based queues healthy.

Takeaways relevant to the new issue ("macOS 26 queues are stale"):

• Teams have been actively bumping helix queues (13 -> 14/15 -> 26) across runtime and other repos, but:
  • Queue bumps frequently expose test or dependency problems (OpenSSL, MsQuic, Xcode/toolchain differences).
  • macOS 26 currently has ARM64 availability but limited/no Amd64 queues in some pools — affects coverage.
  • CI machines sometimes need re-imaging and manual work (installing dependencies or changing setup) when images are updated.
  • Several repos reacted to retired macOS-13 queues with swift merges; coordination across branches is necessary to avoid broken PRs.
• The historical discussions support the new report's ask: macOS Helix queues should be maintained and updated regularly (monthly was suggested) because images/versions and security fixes change and retire queues. The work to bump queues is non-trivial and can require addressing test/dep breakages and machine re-imaging, so a regular update cadence + verification plan (including ARM64 vs Amd64 coverage) is warranted.