Flaky test - MSBuildServerTest

[JanKrivanek]

Context

System.TimeoutException : Test failed due to timeout: process 18281 is active for more than 30 sec.

   at Microsoft.Build.UnitTests.Shared.RunnerUtilities.RunProcessAndGetOutput(String process, String parameters, Boolean& successfulExit, Boolean shellExecute, ITestOutputHelper outputHelper) in /Users/runner/work/1/s/src/UnitTests.Shared/RunnerUtilities.cs:line 110
   at Microsoft.Build.UnitTests.Shared.RunnerUtilities.ExecMSBuild(String pathToMsBuildExe, String msbuildParameters, Boolean& successfulExit, Boolean shellExecute, ITestOutputHelper outputHelper) in /Users/runner/work/1/s/src/UnitTests.Shared/RunnerUtilities.cs:line 42
   at Microsoft.Build.Engine.UnitTests.MSBuildServer_Tests.MSBuildServerTest() in /Users/runner/work/1/s/src/MSBuild.UnitTests/MSBuildServer_Tests.cs:line 121
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)

e.g.: https://dev.azure.com/dnceng-public/public/_build/results?buildId=320229&view=ms.vss-test-web.build-test-results-tab&runId=6566008&resultId=100002&paneView=debug

Looking on the history of the test it seems to fail almost 1 PR a day

[donJoseLuis]

@JanKrivanek should flaky tests be offlined until their reliability is stable?

[JanKrivanek]

@JanKrivanek should flaky tests be offlined until their reliability is stable?

Ideally yes.
For now I'd leave the decision up on whoever will be investigating (as they migh need some diag data from CI). If the test still keeps hitting PRs in ~2 weeks, I'll disable it

[AR-May]

Team triage: Let's disable the test if it hits a lot of PRs per week.

[GangWang01]

For the following step, executing the build task that sleeps 100 seconds randonly timed out.

msbuild/src/MSBuild.UnitTests/MSBuildServer_Tests.cs

Lines 108 to 123 in 93d8322

	// Prep to kill the long-lived task we're about to start.
	Task t = Task.Run(() =>
	{
	// Wait for the long-lived task to start
	// If this test seems to fail randomly, increase this time.
	Thread.Sleep(1000);
	// Kill the server
	Process.GetProcessById(pidOfServerProcess).KillTree(1000);
	});
	// Start long-lived task execution
	TransientTestFile sleepProject = _env.CreateFile("napProject.proj", sleepingTaskContents);
	RunnerUtilities.ExecMSBuild(BuildEnvironmentHelper.Instance.CurrentMSBuildExePath, sleepProject.Path, out _);
	t.Wait();

It intended to let the build task (that made the started msbuild process definitely timeout after 30 seconds) use the old server process and terminate the build task through killing the old server process.
Most likely it didn't sleep enough time waiting for the build task to start and killed the old server process too fast. This resulted in the build started a new server process, and then the started msbuild process timed out after 30 seconds becuase of the build task sleeping 100 seconds.