Skip to content

Latest commit

 

History

History
628 lines (452 loc) · 22.8 KB

File metadata and controls

628 lines (452 loc) · 22.8 KB

Coverlet Microsoft Testing Platform Integration

Microsoft.Testing.Platform and Microsoft Test Framework is a lightweight alternative for VSTest.

More information is available here:

coverlet.MTP implements coverlet.collector functionality for Microsoft.Testing.Platform.

Supported Runtime Versions

Since version 8.0.0:

  • .NET Core >= 8.0

Quick Start

Installation

Add the coverlet.MTP package to your test project:

dotnet add package coverlet.MTP

A sample project file looks like:

<Project Sdk="Microsoft.NET.Sdk">
   <PropertyGroup>
      <TargetFramework>net8.0</TargetFramework>
      <OutputType>Exe</OutputType>
      <!-- Enable Microsoft Testing Platform -->
      <UseMicrosoftTestingPlatformRunner>true</UseMicrosoftTestingPlatformRunner>
      <TestingPlatformDotnetTestSupport>true</TestingPlatformDotnetTestSupport> <!-- not required for .NET SDK 10 or later -->
   </PropertyGroup>
      <ItemGroup>
      <!-- Use xunit.v3.mtp-v2 for MTP v2.x compatibility -->
      <PackageReference Include="xunit.v3.mtp-v2" Version="3.2.1" />
      <PackageReference Include="Microsoft.Testing.Platform" Version="2.2.1" />
      <PackageReference Include="coverlet.MTP" Version="8.0.0" />
   </ItemGroup>
</Project>

Basic Usage

To collect code coverage, run your test executable with the --coverlet flag:

dotnet exec <test-assembly.dll> --coverlet

Or using dotnet test with MTP enabled projects:

dotnet test --coverlet

After the test run, a coverage.json file containing the results will be generated in the current directory.

Command Line Options

The coverlet.MTP extension provides the following command line options. To see all available options, run:

dotnet exec <test-assembly.dll> --help

Coverage Options

Option Description
--coverlet Enable code coverage data collection.
--coverlet-output-format <format> Output format(s) for coverage report. Supported formats: json, lcov, opencover, cobertura, teamcity. Can be specified multiple times. (default: json, cobertura)
--coverlet-file-prefix <prefix> Prefix for coverage report filenames to prevent overwrites when multiple test projects write to the same directory. When specified, files are named <prefix>.coverage.<extension> instead of coverage.<extension>. (default: none)
--coverlet-include <filter> Include assemblies matching filters (e.g., [Assembly]Type). Can be specified multiple times. (default: none)
--coverlet-include-directory <path> Include additional directories for sources. Can be specified multiple times. (default: none)
--coverlet-exclude <filter> Exclude assemblies matching filters (e.g., [Assembly]Type). Can be specified multiple times. User-specified filters are merged with defaults. (default: [coverlet.*]*, [xunit.*]*, [NUnit3.*]*, [nunit.*]*, [Microsoft.Testing.*]*, [Microsoft.Testplatform.*]*, [Microsoft.VisualStudio.TestPlatform.*]*)
--coverlet-exclude-by-file <pattern> Exclude source files matching glob patterns. Can be specified multiple times. (default: none)
--coverlet-exclude-by-attribute <attribute> Exclude methods/classes decorated with attributes. Can be specified multiple times. User-specified attributes are merged with defaults. (default: ExcludeFromCodeCoverage, ExcludeFromCodeCoverageAttribute, GeneratedCodeAttribute, CompilerGeneratedAttribute)
--coverlet-include-test-assembly Include test assembly in coverage. (default: false)
--coverlet-single-hit Limit the number of hits to one for each location. (default: false)
--coverlet-skip-auto-props Skip auto-implemented properties. (default: false)
--coverlet-does-not-return-attribute <attribute> Attributes that mark methods as not returning. Can be specified multiple times. (default: none)
--coverlet-exclude-assemblies-without-sources <value> Exclude assemblies without source code. Values: MissingAll, MissingAny, None. (default: None)

Note

Coverage report files will be stored in --results-directory folder and a time stamp {DateTime.UtcNow:ddMMyyHHmmssfff} is used for generated code coverage files name e.g. coverage.280326122803612.json or coverage.cobertura.280326122803612.xml

Default Exclusions Behavior

Coverlet.MTP applies sensible default exclusions to reduce noise in coverage reports. The behavior differs based on whether you use command line options or a configuration file.

Command Line Defaults:

When using command line options without a configuration file, the following defaults are automatically merged with user-specified exclusions:

  • Default Exclude Filters: [coverlet.*]*, [xunit.*]*, [NUnit3.*]*, [nunit.*]*, [Microsoft.Testing.*]*, [Microsoft.Testplatform.*]*, [Microsoft.VisualStudio.TestPlatform.*]*
  • Default Exclude by Attributes: ExcludeFromCodeCoverage, ExcludeFromCodeCoverageAttribute, GeneratedCodeAttribute, CompilerGeneratedAttribute

Configuration File Behavior (Authoritative Mode):

When a configuration file is present (testconfig.json or coverlet.mtp.appsettings.json), the configuration file becomes authoritative:

  • No defaults are injected for Format, Exclude, ExcludeByAttribute, or other settings
  • You are responsible for defining all options you need
  • Only the minimal [coverlet.*]* exclude filter is always prepended to prevent self-instrumentation

This design follows the principle that providing a configuration file represents your complete intent for coverlet settings.

Important

Breaking Change Notice: In previous versions, defaults were merged even when a configuration file existed. Starting with this version, configuration files are authoritative. If you relied on default exclusions being applied automatically, you must now explicitly add them to your configuration file.

Configuration Files

Coverlet.MTP supports two configuration file formats:

  1. testconfig.json - Microsoft Testing Platform standard format (recommended)
  2. coverlet.mtp.appsettings.json - Legacy format (for backward compatibility)

testconfig.json (Recommended)

The testconfig.json format is the standard configuration file for Microsoft Testing Platform extensions. Coverlet settings are placed under platformOptions.Coverlet:

File Location Priority:

  • [appname].testconfig.json - App-specific configuration (e.g., MyTests.testconfig.json)
  • testconfig.json - Generic configuration

Example testconfig.json:

{
  "platformOptions": {
    "Coverlet": {
      "include": "[MyApp.*]*",
      "exclude": "[*.Tests]*,[*.Generated]*",
      "excludeByAttribute": "GeneratedCode,ExcludeFromCodeCoverage,ExcludeFromCodeCoverageAttribute,CompilerGeneratedAttribute",
      "excludeByFile": "**/Migrations/*.cs",
      "format": "cobertura,json",
      "useSourceLink": false,
      "singleHit": false,
      "includeTestAssembly": false,
      "skipAutoProps": true,
      "deterministicReport": false,
      "excludeAssembliesWithoutSources": "MissingAll"
    }
  }
}

Supported Keys in platformOptions.Coverlet:

Key Type Description
include string Comma-separated include filters (e.g., [MyApp.*]*)
includeDirectory string Comma-separated additional directories for sources
exclude string Comma-separated exclude filters (e.g., [*.Tests]*)
excludeByFile string Comma-separated glob patterns for source file exclusion
excludeByAttribute string Comma-separated attributes to exclude
format string Comma-separated output formats (default: cobertura)
useSourceLink bool Enable SourceLink support
singleHit bool Limit hits to one per location
includeTestAssembly bool Include test assembly in coverage
skipAutoProps bool Skip auto-implemented properties
doesNotReturnAttribute string Comma-separated attributes marking non-returning methods
deterministicReport bool Generate deterministic reports
excludeAssembliesWithoutSources string Values: MissingAll, MissingAny, None
disableManagedInstrumentationRestore bool Disable managed instrumentation restore
mergeWith string Path to existing coverage file to merge with

Note

Keys in testconfig.json use camelCase (e.g., excludeByAttribute), following the Microsoft Testing Platform convention.

Project Setup for testconfig.json:

<ItemGroup>
  <None Update="testconfig.json">
    <CopyToOutputDirectory>Always</CopyToOutputDirectory>
  </None>
</ItemGroup>

coverlet.mtp.appsettings.json (Legacy)

The legacy coverlet.mtp.appsettings.json format is still supported for backward compatibility. If both testconfig.json and coverlet.mtp.appsettings.json exist, testconfig.json takes priority.

Project Setup:

  1. Create the configuration file in your test project
  2. Ensure it's copied to the output directory:
<ItemGroup>
  <None Update="coverlet.mtp.appsettings.json">
    <CopyToOutputDirectory>Always</CopyToOutputDirectory>
  </None>
</ItemGroup>

Important

The configuration file must be present in the output directory at runtime (next to the test assembly).

Supported Configuration Keys:

Key Type Description
Include string Comma-separated include filters (e.g., [MyApp.*]*)
IncludeDirectory string Comma-separated additional directories for sources
Exclude string Comma-separated exclude filters (e.g., [*.Tests]*,[*.Generated]*)
ExcludeByFile string Comma-separated glob patterns for source file exclusion
ExcludeByAttribute string Comma-separated attributes to exclude
Format string Comma-separated output formats (default: cobertura)
UseSourceLink bool Enable SourceLink support
SingleHit bool Limit hits to one per location
IncludeTestAssembly bool Include test assembly in coverage
SkipAutoProps bool Skip auto-implemented properties
DoesNotReturnAttribute string Comma-separated attributes marking non-returning methods
DeterministicReport bool Generate deterministic reports
ExcludeAssembliesWithoutSources string Values: MissingAll, MissingAny, None (default: MissingAll)

Example coverlet.mtp.appsettings.json:

{
  "Coverlet": {
    "Include": "[MyApp.*]*",
    "Exclude": "[*.Tests]*,[*.Generated]*",
    "ExcludeByAttribute": "GeneratedCode,ExcludeFromCodeCoverage",
    "ExcludeByFile": "**/Migrations/*.cs",
    "Format": "cobertura,json",
    "UseSourceLink": false,
    "SingleHit": false,
    "IncludeTestAssembly": false,
    "SkipAutoProps": true,
    "DeterministicReport": false,
    "ExcludeAssembliesWithoutSources": "MissingAll"
  }
}

Note

  • The configuration section is named Coverlet. With Microsoft.Extensions.Configuration, keys are case-insensitive (so Coverlet, coverlet, etc. all work).
  • Array values are specified as comma-separated strings, not JSON arrays.
  • The default exclude filter [coverlet.*]* is always prepended to the Exclude value.

Configuration Precedence

Coverlet.MTP uses the following precedence order when determining configuration values:

Priority 1: Command line options (always take precedence)
Priority 2: testconfig.json (app-specific > generic)
Priority 3: coverlet.mtp.appsettings.json (legacy)
Priority 4: Built-in defaults (only when no configuration file exists)

Key Points:

  • Command line always wins: CLI options override all configuration file settings
  • Configuration files are authoritative: When a config file exists, only its values are used (no defaults injected)
  • testconfig.json has priority: If both testconfig.json and coverlet.mtp.appsettings.json exist, only testconfig.json is used
  • Defaults apply only without config files: Built-in defaults are only used when no configuration file is present

File Priority for testconfig.json:

  1. [appname].testconfig.json (e.g., MyTests.testconfig.json)
  2. testconfig.json

Alternative: MSBuild Property for Command Line Arguments

If you prefer to define coverlet options in your .csproj file and leverage MSBuild variables (like $(AssemblyName)), you can use the TestingPlatformCommandLineArguments property:

<PropertyGroup>
  <!-- Pass coverlet options via MSBuild - allows using MSBuild variables -->
  <TestingPlatformCommandLineArguments>--coverlet --coverlet-file-prefix $(AssemblyName) --coverlet-output-format opencover</TestingPlatformCommandLineArguments>
</PropertyGroup>

This approach is useful when you want to:

  • Use MSBuild variables like $(AssemblyName), $(MSBuildProjectName), or $(Configuration)
  • Define configuration at the solution level via Directory.Build.props
  • Ensure each test project uses its assembly name as the file prefix automatically

Example Directory.Build.props for solution-wide configuration:

<Project>
  <PropertyGroup Condition="'$(IsTestProject)' == 'true'">
    <TestingPlatformCommandLineArguments>--coverlet --coverlet-file-prefix $(MSBuildProjectName)</TestingPlatformCommandLineArguments>
  </PropertyGroup>
</Project>

Examples

Generate coverage in JSON format (default):

dotnet exec TestProject.dll --coverlet

Generate coverage in Cobertura format:

dotnet exec TestProject.dll --coverlet --coverlet-output-format cobertura

Generate coverage in multiple formats:

dotnet run TestProject.dll --coverlet --coverlet-output-format json --coverlet-output-format cobertura --coverlet-output-format lcov

Include only specific assemblies:

dotnet exec TestProject.dll --coverlet --coverlet-include "[MyApp.]"

Exclude test assemblies and specific namespaces:

dotnet exec TestProject.dll --coverlet --coverlet-exclude "[.Tests]" --coverlet-exclude "[]MyApp.Generated."

Exclude by attribute:

dotnet exec TestProject.dll --coverlet --coverlet-exclude-by-attribute "Obsolete" --coverlet-exclude-by-attribute "GeneratedCode"

Use file prefix to prevent overwrites in multi-project solutions:

dotnet exec TestProject.dll --coverlet --coverlet-file-prefix "MyProject.UnitTests"

This generates files named MyProject.UnitTests.coverage.json and MyProject.UnitTests.coverage.cobertura.xml instead of overwriting the default coverage.json and coverage.cobertura.xml.

Coverage Output

Coverlet can generate coverage results in multiple formats:

  • json (default) - Coverlet's native JSON format
  • lcov - LCOV format
  • opencover - OpenCover XML format
  • cobertura - Cobertura XML format
  • teamcity - TeamCity service messages

Filter Expressions

Filter expressions allow fine-grained control over what gets included or excluded from coverage.

Syntax: [Assembly-Filter]Type-Filter

Wildcards:

  • * matches zero or more characters
  • ? makes the prefixed character optional

Examples:

  • [*]* - All types in all assemblies
  • [coverlet.*]Coverlet.Core.Coverage - Specific class in matching assemblies
  • [*]Coverlet.Core.Instrumentation.* - All types in a namespace
  • [coverlet.*.tests?]* - Assemblies ending with .test or .tests

Both --coverlet-include and --coverlet-exclude can be used together, but --coverlet-exclude takes precedence.

Excluding From Coverage

Using Attributes

Apply the ExcludeFromCodeCoverage attribute from System.Diagnostics.CodeAnalysis to exclude methods or classes:

[ExcludeFromCodeCoverage] public class NotCovered { // This class will be excluded from coverage }

Additional attributes can be specified using --coverlet-exclude-by-attribute.

Using Source File Patterns

Exclude source files using glob patterns with --coverlet-exclude-by-file:

dotnet exec TestProject.dll --coverlet --coverlet-exclude-by-file "**/Generated/*.cs"

How It Works

The coverlet.MTP extension integrates with the Microsoft Testing Platform using the extensibility model:

  1. Test Host Controller Extension: Implements ITestHostProcessLifetimeHandler to instrument assemblies before tests run and collect coverage after tests complete.

  2. Before Tests Run:

    • Locates the test assembly and referenced assemblies with PDBs
    • Instruments assemblies by inserting code to record sequence point hits
  3. After Tests Run:

    • Restores original non-instrumented assemblies
    • Reads recorded hit information
    • Generates coverage report in the specified format(s)

Comparison with coverlet.collector (VSTest)

Feature coverlet.MTP coverlet.collector
Test Platform Microsoft Testing Platform VSTest
Configuration Command line options runsettings file
coverlet.mtp.appsettings.json
Default Format JSON Cobertura

Known Limitations

  • Threshold validation is not yet supported (planned for future releases)
  • Report merging is not yet supported (use external tools like dotnet-coverage or reportgenerator)

Tip

Merging coverage files from multiple test runs:

Use dotnet-coverage tool:

dotnet-coverage merge coverage/**/coverage.cobertura.xml -f cobertura -o coverage/merged.xml

Or use reportgenerator:

reportgenerator -reports:"**/*.cobertura.xml" -targetdir:"coverage/report" -reporttypes:"HtmlInline_AzurePipelines_Dark;Cobertura"

Architecture

Coverlet.MTP uses a two-process architecture:

flowchart LR
    subgraph Controller["CONTROLLER PROCESS"]
        direction TB
        THC["builder.TestHostControllers"]
        PLH["AddProcessLifetimeHandler"]
        EVP["AddEnvironmentVariableProvider"]
        THC --> PLH
        THC --> EVP
    end

    subgraph TestHost["TEST HOST PROCESS"]
        direction TB
        TH["builder.TestHost"]
        TSLH["AddTestSessionLifetimeHandle"]
        DC["AddDataConsumer"]
        TALC["AddTestApplicationLifecycleCallbacks"]
        TH --> TSLH
        TH --> DC
        TH --> TALC
    end

    Controller -->|"Environment Variables<br/>Process Lifecycle Events"| TestHost

    style Controller fill:#e6f3ff,stroke:#0066cc,stroke-width:2px
    style TestHost fill:#e6ffe6,stroke:#009900,stroke-width:2px
Loading

And here's a sequence diagram showing the coverlet.MTP flow:

sequenceDiagram
    participant C as Controller Process<br/>(CoverletExtensionCollector)
    participant E as Environment Variables
    participant T as Test Host Process<br/>(CoverletInProcessHandler)

    Note over C: BeforeTestHostProcessStartAsync
    C->>C: Instrument assemblies
    C->>E: Set COVERLET_MTP_COVERAGE_ENABLED=true
    C->>E: Set COVERLET_MTP_COVERAGE_IDENTIFIER
    C->>E: Set COVERLET_MTP_HITS_FILE_PATH

    Note over C: OnTestHostProcessStartedAsync
    C->>T: Start test host process

    Note over T: Constructor
    T->>E: Read environment variables
    T->>T: Initialize handler

    Note over T: OnTestSessionStartingAsync
    T->>T: Tests execute...

    Note over T: OnTestSessionFinishingAsync
    T->>T: FlushCoverageData()
    T->>T: Call ModuleTrackerTemplate.UnloadModule

    Note over C: OnTestHostProcessExitedAsync
    C->>C: Read hit files
    C->>C: Generate coverage reports
Loading

Environment Variables

Variable Purpose
COVERLET_MTP_COVERAGE_ENABLED Indicates coverage is active
COVERLET_MTP_COVERAGE_IDENTIFIER Unique ID for result correlation
COVERLET_MTP_HITS_FILE_PATH Directory for hit data files
COVERLET_MTP_INPROC_DEBUG Set to "1" to debug in-process handler
COVERLET_MTP_INPROC_EXCEPTIONLOG_ENABLED Set to "1" for detailed error logging

Troubleshooting

Enable Diagnostic Output

Use the MTP diagnostic options to get detailed logs:

dotnet exec TestProject.dll --coverlet --diagnostic --diagnostic-verbosity trace --diagnostic-output-directory ./logs

Debug Coverlet Extension

Set the environment variable to attach a debugger:

set COVERLET_MTP_DEBUG=1 dotnet exec TestProject.dll --coverlet

Debugging and Troubleshooting (not tested - just defined)

Enable Debugger Launch

To launch a debugger when coverlet.MTP initializes:

Windows:

set COVERLET_MTP_DEBUG=1

Linux/macOS:

export COVERLET_MTP_DEBUG=1

Wait for Debugger Attach

To make coverlet.MTP wait for a debugger to attach (Windows):

set COVERLET_MTP_DEBUG_WAIT=1

The extension will display:

[Coverlet.MTP] CoverletExtension: Waiting for debugger to attach... [Coverlet.MTP] Process Id: 12345, Name: dotnet

Use Visual Studio's "Debug > Attach to Process" (Ctrl+Alt+P) to attach.

Enable Tracker Logging

To collect detailed logs from the injected coverage tracker (Windows):

set COVERLET_ENABLETRACKERLOG=1

Log files will be created near the instrumented module location:

  • moduleName.dll_tracker.txt - Main tracker log
  • TrackersHitsLog/ folder - Detailed hit information

Enable Instrumentation Debugging

For detailed instrumentation diagnostics (Windows):

set COVERLET_MTP_INSTRUMENTATION_DEBUG=1

Enable Exception Logging

To capture detailed exception information (Windows):

set COVERLET_MTP_EXCEPTIONLOG_ENABLED=1

Using MTP Built-in Diagnostics

Microsoft Testing Platform provides built-in diagnostic logging (Windows):

dotnet test --diagnostic --diagnostic-verbosity Trace

This creates detailed logs in the TestResults directory.

Combined Debugging Example

Enable all debugging features (Windows)

set COVERLET_MTP_DEBUG_WAIT=1 set COVERLET_ENABLETRACKERLOG=1 set COVERLET_MTP_EXCEPTIONLOG_ENABLED=1

Run tests with MTP diagnostics

dotnet test --coverlet --diagnostic --diagnostic-verbosity Trace

Requirements

  • .NET 8.0 SDK or newer
  • Microsoft.Testing.Platform 2.0.0 or newer
  • Test framework with MTP support (e.g., xUnit v3 (xunit.v3.mtp-v2), MSTest v3, NUnit with MTP adapter)

Related Documentation