Avoid long busy-waiting between hijack retries.

[VSadov]

Fixes: #102832

• refactors the pause routine into minipal_microdelay - to have the same implementation for CoreCLR and NativeAOT
• use sub-millisecond sleep API, when available, for long(ish) pauses.
• couple minor fixes around sending suspension signals.
  (not responsible for the regression, just possible corner cases noticed while investigating)

[VSadov]

This is the actual fix. When we need to pause for longer than 10 microseconds, we sleep instead of spinning, thus freeing compute resources to other threads (including ones that we want to progress towards suspending).

[VSadov]

10 is a somewhat arbitrary number in the order of context switch cost. (2-10 microseconds).
We are not very sensitive to the exact value. It just means "too short for the sleep to have any advantages".

[VSadov]

The other reason for 10 is that the default timer slack on Linux is 50 microseconds. So the shortest sleep could be on average 25 microseconds anyways if our wakeups are bunched up with other sleepers.
Thus 10 seems like a reasonable threshold to use sleep.

[VSadov]

Performance impact on Stage1 ASP.Net benchmark.
The change recovers the loss form regression.

======================================================== arm64

======= before regression

| First Request (ms)      | 171        |
| Requests/sec            | 760,970    |
| Requests                | 11,490,582 |
| Mean latency (ms)       | 0.78       |
| Max latency (ms)        | 38.00      |


======== after regression

| First Request (ms)      | 164       |
| Requests/sec            | 627,518   |
| Requests                | 9,458,579 |
| Mean latency (ms)       | 1.65      |
| Max latency (ms)        | 54.40     |


======== after the fix

| First Request (ms)      | 160        |
| Requests/sec            | 770,416    |
| Requests                | 11,632,411 |
| Mean latency (ms)       | 0.78       |
| Max latency (ms)        | 40.54      |

======================================================= x64

======= before regression

| First Request (ms)      | 181        |
| Requests/sec            | 813,308    |
| Requests                | 12,280,780 |
| Mean latency (ms)       | 0.46       |
| Max latency (ms)        | 44.08      |

======== after regression

| First Request (ms)      | 179        |
| Requests/sec            | 771,124    |
| Requests                | 11,643,639 |
| Mean latency (ms)       | 0.56       |
| Max latency (ms)        | 43.84      |

======== after the fix

| Start Time (ms)         | 0          |
| First Request (ms)      | 185        |
| Requests/sec            | 803,472    |
| Requests                | 12,132,099 |
| Mean latency (ms)       | 0.44       |
| Max latency (ms)        | 36.27      |

commands used:

===================  arm64

crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/goldilocks.benchmarks.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/steadystate.profile.yml --scenario basicminimalapivanilla --profile arm-lin-28-app --profile amd-lin-load --profile amd-lin2-db --application.environmentVariables DOTNET_GCDynamicAdaptationMode=1 --application.framework net9.0 --application.options.collectCounters true --load.options.reuseBuild true --application.aspNetCoreVersion 9.0.0-preview.5.24256.2 --application.sdkVersion 9.0.100-preview.5.24267.1 --application.runtimeVersion 9.0.0-preview.5.24259.7


crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/goldilocks.benchmarks.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/steadystate.profile.yml --scenario basicminimalapivanilla --profile arm-lin-28-app --profile amd-lin-load --profile amd-lin2-db --application.environmentVariables DOTNET_GCDynamicAdaptationMode=1 --application.framework net9.0 --application.options.collectCounters true --load.options.reuseBuild true --application.aspNetCoreVersion 9.0.0-preview.5.24256.2 --application.sdkVersion 9.0.100-preview.5.24267.1 --application.runtimeVersion 9.0.0-preview.5.24260.2


crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/goldilocks.benchmarks.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/steadystate.profile.yml --scenario basicminimalapivanilla --profile arm-lin-28-app --profile amd-lin-load --profile amd-lin2-db --application.environmentVariables DOTNET_GCDynamicAdaptationMode=1 --application.framework net9.0 --application.options.collectCounters true --load.options.reuseBuild true --application.aspNetCoreVersion 9.0.0-preview.5.24256.2 --application.sdkVersion 9.0.100-preview.5.24267.1 --application.runtimeVersion 9.0.0-preview.5.24260.2   --application.options.outputFiles armFix\libcoreclr.so


=================== x64

crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/goldilocks.benchmarks.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/steadystate.profile.yml --scenario basicminimalapivanilla --profile intel-lin-app --profile intel-load-load --profile intel-db-db --application.environmentVariables DOTNET_GCDynamicAdaptationMode=1 --application.framework net9.0 --application.options.collectCounters true --load.options.reuseBuild true --application.aspNetCoreVersion 9.0.0-preview.5.24256.2 --application.sdkVersion 9.0.100-preview.5.24267.1 --application.runtimeVersion 9.0.0-preview.5.24259.7


crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/goldilocks.benchmarks.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/steadystate.profile.yml --scenario basicminimalapivanilla --profile intel-lin-app --profile intel-load-load --profile intel-db-db --application.environmentVariables DOTNET_GCDynamicAdaptationMode=1 --application.framework net9.0 --application.options.collectCounters true --load.options.reuseBuild true --application.aspNetCoreVersion 9.0.0-preview.5.24256.2 --application.sdkVersion 9.0.100-preview.5.24267.1 --application.runtimeVersion 9.0.0-preview.5.24260.2


crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/goldilocks.benchmarks.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/steadystate.profile.yml --scenario basicminimalapivanilla --profile intel-lin-app --profile intel-load-load --profile intel-db-db --application.environmentVariables DOTNET_GCDynamicAdaptationMode=1 --application.framework net9.0 --application.options.collectCounters true --load.options.reuseBuild true --application.aspNetCoreVersion 9.0.0-preview.5.24256.2 --application.sdkVersion 9.0.100-preview.5.24267.1 --application.runtimeVersion 9.0.0-preview.5.24260.2   --application.options.outputFiles x64Fix\libcoreclr.so

[dotnet-policy-service]

Tagging subscribers to this area: @mangod9
See info in area-owners.md if you want to be subscribed.

[jkotas]

Thanks

[VSadov]

Thanks!