Can't run UWP tests from vstest.console.exe in VS 15.6.2

[jsiemens]

Description

I am unable to run UWP tests from the VS test console (vstest.console.exe). I get the error:

No suitable test runtime provider found for this run.

Steps to reproduce

1. Create new unit test project: Add new project > Windows Universal > Unit Test App (Universal Windows)
2. Click run all from Test Explorer. Test is found and executes successfully.
3. Run from command line:

C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TestWindow
λ vstest.console.exe "[SolutionRoot]\UnitTestProject\bin\x86\Debug\UnitTestProject1.exe"

I also tried different versions of UWP and it seems to make no difference.

My OS is Fall Creator's Update. I have UWP 16299 installed.

Also tried with:

--TestAdapterPath:[SolutionRoot]\UnitTestProject1\bin\x86\Debug

Either way, I get the error:
No suitable test runtime provider found for this run.

Confirmed that Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.dll exists in the output directory.

Expected behavior

Tests are run.

Actual behavior

Error: No suitable test runtime provider found for this run.

Diagnostic logs

TpTrace Verbose: 0 : 20120, 1, 2018/03/14, 12:54:04.408, 559803623423, vstest.console.exe, TestPluginCache: Discovering the extensions using extension path.
TpTrace Verbose: 0 : 20120, 1, 2018/03/14, 12:54:04.534, 559804031982, vstest.console.exe, TestPluginCache: Discovered the extensions using extension path ''.
TpTrace Verbose: 0 : 20120, 1, 2018/03/14, 12:54:04.536, 559804037001, vstest.console.exe, TestPluginCache: Discoverers are ''.
TpTrace Verbose: 0 : 20120, 1, 2018/03/14, 12:54:04.536, 559804037533, vstest.console.exe, TestPluginCache: Executors are ''.
TpTrace Verbose: 0 : 20120, 1, 2018/03/14, 12:54:04.536, 559804037879, vstest.console.exe, TestPluginCache: Setting providers are ''.
TpTrace Verbose: 0 : 20120, 1, 2018/03/14, 12:54:04.536, 559804038232, vstest.console.exe, TestPluginCache: Loggers are ''.
TpTrace Information: 0 : 20120, 1, 2018/03/14, 12:54:04.573, 559804165664, vstest.console.exe, RunTestsArgumentProcessor:Execute: Test run is starting.
TpTrace Verbose: 0 : 20120, 1, 2018/03/14, 12:54:04.573, 559804166322, vstest.console.exe, RunTestsArgumentProcessor:Execute: Queuing Test run.
TpTrace Information: 0 : 20120, 1, 2018/03/14, 12:54:04.575, 559804174493, vstest.console.exe, TestRequestManager.RunTests: run tests started.
TpTrace Information: 0 : 20120, 1, 2018/03/14, 12:54:04.668, 559804498596, vstest.console.exe, AssemblyMetadataProvider.GetFrameWork: Determined framework:'.NETCore,Version=v5.0' for source: 'C:\Users\jsiemens.LATITUDEGEO\Source\Repos\UnitTestProject1\UnitTestProject1\bin\x86\Debug\UnitTestProject1.exe'
TpTrace Information: 0 : 20120, 1, 2018/03/14, 12:54:04.668, 559804499564, vstest.console.exe, Determined framework for all sources: .NETCore,Version=v5.0
TpTrace Information: 0 : 20120, 1, 2018/03/14, 12:54:04.671, 559804508255, vstest.console.exe, AssemblyMetadataProvider.GetArchitecture: Determined architecture:X86 info for assembly: C:\Users\jsiemens.LATITUDEGEO\Source\Repos\UnitTestProject1\UnitTestProject1\bin\x86\Debug\UnitTestProject1.exe
TpTrace Information: 0 : 20120, 1, 2018/03/14, 12:54:04.672, 559804514554, vstest.console.exe, Determined platform for all sources: X86
TpTrace Information: 0 : 20120, 1, 2018/03/14, 12:54:04.677, 559804532175, vstest.console.exe, Compatible sources list :
TpTrace Information: 0 : 20120, 1, 2018/03/14, 12:54:04.677, 559804532897, vstest.console.exe, C:\Users\jsiemens.LATITUDEGEO\Source\Repos\UnitTestProject1\UnitTestProject1\bin\x86\Debug\UnitTestProject1.exe
TpTrace Information: 0 : 20120, 1, 2018/03/14, 12:54:04.680, 559804543157, vstest.console.exe, InferRunSettingsHelper.IsTestSettingsEnabled: Unable to navigate to RunSettings/MSTest. Current node: RunSettings
TpTrace Information: 0 : 20120, 1, 2018/03/14, 12:54:04.701, 559804617260, vstest.console.exe, TestPluginManager.CreateTestExtension: Attempting to load test extension: Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Hosting.DefaultTestHostManager
TpTrace Information: 0 : 20120, 1, 2018/03/14, 12:54:04.702, 559804620413, vstest.console.exe, TestPluginManager.CreateTestExtension: Attempting to load test extension: Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Hosting.DotnetTestHostManager
TpTrace Information: 0 : 20120, 1, 2018/03/14, 12:54:04.703, 559804622526, vstest.console.exe, TestPluginManager.CreateTestExtension: Attempting to load test extension: Microsoft.VisualStudio.UwpTestHostRuntimeProvider.UwpTestHostManager
TpTrace Error: 0 : 20120, 1, 2018/03/14, 12:54:04.704, 559804625740, vstest.console.exe, TestPlatform.CreateTestRunRequest: No suitable testHostProvider found for runsettings :

C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TestWindow\TestResults
X86
.NETCore,Version=v5.0
False
False


TpTrace Error: 0 : 20120, 1, 2018/03/14, 12:54:04.711, 559804649929, vstest.console.exe, TestRequestManager.RunTests: failed to run tests: Microsoft.VisualStudio.TestPlatform.ObjectModel.TestPlatformException: No suitable test runtime provider found for this run.
at Microsoft.VisualStudio.TestPlatform.Client.TestPlatform.ThrowExceptionIfTestHostManagerIsNull(ITestRuntimeProvider testHostManager, String settingXml)
at Microsoft.VisualStudio.TestPlatform.Client.TestPlatform.CreateTestRunRequest(IRequestData requestData, TestRunCriteria testRunCriteria)
at Microsoft.VisualStudio.TestPlatform.CommandLine.TestPlatformHelpers.TestRequestManager.RunTests(IRequestData requestData, TestRunCriteria testRunCriteria, ITestRunEventsRegistrar testRunEventsRegistrar)
TpTrace Information: 0 : 20120, 1, 2018/03/14, 12:54:04.715, 559804664656, vstest.console.exe, TestRequestManager.RunTests: run tests completed, sucessful: False.
TpTrace Information: 0 : 20120, 1, 2018/03/14, 12:54:04.715, 559804666186, vstest.console.exe, RunTestsArgumentProcessor:Execute: Test run is completed.
TpTrace Verbose: 0 : 20120, 1, 2018/03/14, 12:54:04.715, 559804666542, vstest.console.exe, Executor.Execute: Exiting with exit code of 1

Environment

Microsoft (R) Test Execution Command Line Tool Version 15.6.1
Windows 10, Fall Creator's Update

[singhsarab]

@jsiemens
The commandline that you have used seems incorrect. The commandline for running UWP tests is different from classic way.

For running UWP tests you can use the following commandline
vstest.console <path_to_appxrecipe> /framework:FrameworkUap10

Hope it helps.

[jsiemens]

@singhsarab

Thanks for the help. That did indeed get it running locally.

I notice this pops up a test window and the tests run in that window. The pop up windows has some interesting implications. #1, I can't actually run this at all on my build server because the build server is Windows Server 2012. Can UWP apps run on Windows Server 2016, or will we need to provision a Windows 10 build server? #2, will it work to pop up a new application window when vstest.console is running inside a VSTS build agent service?

Also, it would be nice if this information was in the docs (command line help + https://msdn.microsoft.com/en-CA/library/jj155796.aspx). Presently, the framework is not listed as one of the options in the help, and the mention of targetting the .appxrecipe file is also not mentioned.

Jeff

[pvlakshm]

Will fix up the docs. In the meantime, this blogpost might help: DevOps for IoT with Win10 IoT Core, UWP, and VSTS.

[pvlakshm]

Doc updated: VSTest.Console.exe command-line options (see under valid values for frameworks).

[premaram]

@pvlakshm I am still facing the issue even after adding the frameworkUAP10 argument to the vstest.console.exe.
May I know where the Test Adapter, "Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.dll" should be present?
I do not find it after the test agent update.

[nohwnd]

FWIW I see the same error when running from vstest.console.exe in .nuget TestPlatform, but not when I run from Developer Commandline which ships with VS.

[ArchieCoder]

Same issue as @premaram about the Adapter.