Avoid guid at the end of outputDirectory

[MarcoRossignoli]

On Coverlet we have some users complain about the dynamic folder name generated for reports file.
I did some check on code and seem that guid is "mandatory" and not removable.

https://github.com/microsoft/vstest/blob/master/src/Microsoft.TestPlatform.Common/DataCollection/DataCollectionAttachmentManager.cs#L101-L125

Could be useful add a parameter, maybe in runsetting like <DeterministicOutputdirectory>true/false</DeterministicOutputdirectory> and in that case avoid to concat guid.ToString().

Dynamic folder makes CI more complex.

Issues refs coverlet-coverage/coverlet#500 coverlet-coverage/coverlet#767

cc: @DaleyKD

[nohwnd]

I don't know the reasons for the guid but I would rather avoid yet another option. I see the coverage file described in the TRX output, but even thought there is a href to it, it does not contain the guid. I also don't see the guid anywhere else in the repo. Imho the best course of action is to provide a way that will allow the test result to be programatically processed.

When you run with CC in azure pipeline, do you see the coverage file automatically uploaded? maybe they have this already figured out.

[nohwnd]

Admittedly the trx, and xml in general is not the easiest to process, but it's a start, and it is generic format that we don't have to re-invent.

[MarcoRossignoli]

When you run with CC in azure pipeline, do you see the coverage file automatically uploaded? maybe they have this already figured out.

Looking at publish result task https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/test/publish-test-results?view=azure-devops&tabs=yaml seem that a glob filter is used

- task: PublishTestResults@2
  inputs:
    testRunner: VSTest
    testResultsFiles: '**/*.trx'
    failTaskOnFailedTests: true

[MarcoRossignoli]

Admittedly the trx, and xml in general is not the easiest to process, but it's a start, and it is generic format that we don't have to re-invent.

I'm ok with file format, but non deterministic path add some work to do in some scenario, tipical "find right file and move to".

[nohwnd]

I meant it more in the direction of returning a simple object that can be programatically processed to get to the info, instead of forcing the naming conventions to be deterministic. For example in Pester I simply return a result object that the caller can process. Here we could do something similar. Say you'd want to upload your CC file:

$result = dotnet test --collect="Code Coverage" ....params.... --json-result 
($result | ConvertFrom-Json).Attachements | where Type -eq "Code Coverage" | Invoke-RestMethod -Method PUSH

This would be the end product where we would have to come up with the result etc... So a more achievable goal would be to hookup the trx file and read the path to the CC file from there if it was correct.

[MarcoRossignoli]

$result = dotnet test --collect="Code Coverage" ....params.... --json-result

If I don't use powershell what does --json-result produce?Is it a json on standard output?

[nohwnd]

Yes, it was just an exam

…

________________________________ Od: Marco Rossignoli <notifications@github.com> Odesláno: Wednesday, April 1, 2020 8:05:48 PM Komu: microsoft/vstest <vstest@noreply.github.com> Kopie: Jakub Jareš <me@jakubjares.com>; Comment <comment@noreply.github.com> Předmět: Re: [microsoft/vstest] Avoid guid at the end of outputDirectory (#2378) $result = dotnet test --collect="Code Coverage" ....params.... --json-result Are you forcing powershell usage?If I don't use powershell what does --json-result produce?Is it a json on standard output? — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#2378 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABLYLYIFL7VICTTBJS7R7ILRKN67ZANCNFSM4LVPYFMA>.

[nohwnd]

Yes json, it was just an example to show a concept. I am not forcing powershell on anyone.

…

________________________________ Od: Marco Rossignoli <notifications@github.com> Odesláno: Wednesday, April 1, 2020 8:05:48 PM Komu: microsoft/vstest <vstest@noreply.github.com> Kopie: Jakub Jareš <me@jakubjares.com>; Comment <comment@noreply.github.com> Předmět: Re: [microsoft/vstest] Avoid guid at the end of outputDirectory (#2378) $result = dotnet test --collect="Code Coverage" ....params.... --json-result Are you forcing powershell usage?If I don't use powershell what does --json-result produce?Is it a json on standard output? — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#2378 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABLYLYIFL7VICTTBJS7R7ILRKN67ZANCNFSM4LVPYFMA>.

[MarcoRossignoli]

What if we have more than one collector https://github.com/Microsoft/vstest-docs/blob/master/docs/analyze.md#enabledisable-a-datacollector?
A json array with different Type(collector friendly name)?

[ViktorHofer]

Just stumbled upon this.

I don't know the reasons for the guid but I would rather avoid yet another option.

I disagree. VSTest should respect the specified ResultsDirectory and CoverageFileName for data collectors and not create intermediate directories between. I can do that for loggers already so I would argue DataCollectors to behave the same:

<RunSettings>
  <RunConfiguration>
     <!-- Directory for test run reports. E.g. trx, coverage etc. -->
    <ResultsDirectory>.\</ResultsDirectory>
  </RunConfiguration>
  <LoggerRunSettings>
    <Loggers>
      <Logger uri="logger://Microsoft/TestPlatform/TrxLogger/v1">
        <Configuration>
          <LogFileName>testResults.trx</LogFileName>
        </Configuration>
      </Logger>
      <Logger uri="logger://Microsoft/TestPlatform/HtmlLogger/v1">
        <Configuration>
          <LogFileName>testResults.html</LogFileName>
        </Configuration>
      </Logger>
    </Loggers>
  </LoggerRunSettings>

I would expect the following behavior:

• LogFileName inside a DataCollector configuration node is respected:

<DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="XPlat code coverage">
        <Configuration>
          <CoverageFileName>coverage.xml</CoverageFileName>
        </Configuration>
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>

• The specified ResultsDirectory is respected without any intermediate folders in it.

  <RunConfiguration>
     <!-- Directory for test run reports. E.g. trx, coverage etc. -->
    <ResultsDirectory>.\</ResultsDirectory>
  </RunConfiguration>

In this example you can even see that coverage is explicitly listed in the comment (which I took from the docs). Specifying the log file name seems to have worked in the past with the pre-coverlet code coverage collector: #1957.

[ViktorHofer]

I just observed another misbehavior:

The coverage report is created twice:

• vihofer_DESKTOP-803H9R0_2020-04-22_06_59_42\In\DESKTOP-803H9R0\coverage.opencover.xml
• 4a408eed-656b-435d-a540-30c285d1c57a\coverage.opencover.xml

[MarcoRossignoli]

The coverage report is created twice:

@ViktorHofer I think it's a dup of #2334

[ViktorHofer]

You are right :)