LivingDoc Implementation (Fix for skipped steps and missing error messages)

[Registeel]

I have taken an implementation of @dakiers project that generates the TestExecution.json file that is required to run SpecFlow+ LivingDoc with test results. I added everything but the "Hooks" directory to a class library then added the hooks to my existing reqnroll project and it worked great for generating the TestExecution.json file.

There are issues I found that I have resolved in this implementation.

1. LivingDoc no longer records the skipped steps as grayed out and shows them as successful.
2. step.TestError?.Message is always empty so errored steps give no output.

I've resolved both of these issues. I'll start with the first problem.

Skipped Steps

The issue with skipped steps is that they are not processed in the [AfterStep] hook. So once you hit a failed step, all the skipped ones afterwards are not recorded in the ExecutionResult object that is written to in the [AfterStep] hook like so:

_result!.StepResult.Add(new(
    (DateTime.UtcNow - _stepStartTime!).Value, 
    step.Status, 
    step.TestError?.Message, 
    outputHelper.FlushOutputs(OutputLocation.AfterStep))); 

The solution for this problem is to build a custom SkippedStepHandler. This is a custom class with any name you like that inherits the reqnroll ISkippedStepHandler interface.

My implementation looks like this:

public class SkippedStepHandler : ISkippedStepHandler
{
    public void Handle(ScenarioContext scenarioContext)
    {
        var executionResult = scenarioContext.ScenarioContainer.Resolve<ExecutionResult>();

        executionResult!.StepResults.Add(new(
            new TimeSpan(),
            scenarioContext.StepContext.Status,
            scenarioContext.StepContext.TestError?.Message));
    }
}

The confusing part about this is where the ExecutionResult is coming from. This is what is used to record the step status to write to the TestExecution.json file. We need to pass this object by reference from the main reqnroll hook file that is running the tests.

I have done this in my [BeforeScenario] hook from dakiers project like so:

[BeforeScenario]
public void BeforeScenario(FeatureInfo feature, ScenarioInfo scenario)
{
    var skipHandler = new SkippedStepHandler();
    this._objectContainer.RegisterInstanceAs<ISkippedStepHandler>(skipHandler);

    _result = new(
        ExecutionContextType,
        feature.FolderPath,
        feature.Title,
        feature.Title,
        scenario.Arguments.Values.Cast<string>().ToArray());

    this._objectContainer.RegisterInstanceAs(_result);
}

The _objectContainer is an instance of an IObjectContainer that is dependency injected into the main Hook class.

With this setup the _result object is passed to the custom handler and when skipped steps are hit they are written to the result before being written to the TestExecution.json file in the [AfterScenario] and [AfterTestRun] hooks.

Onto problem number two.

Empty Error Messages

This one was a simpler problem. Essentially step.TestError?.Message is always empty, but you do get an error message for the failure of the step in your Scenario Context.

My solution is to change step.TestError?.Message, (step coming from var step = scenario.StepContext), to scenario.TestError?.Message.
If you would like extra logging I also appended scenario.TestError?.InnerException?.Message to mine.

[gasparnagy]

@Registeel Thx for sharing!

[marcelstuder]

@Registeel Thank you for the detailed and very helpful insights!

There is a small issue in the [BeforeScenario] handler: in your example above the feature.Title is used twice as an argument when constructing the ExecutionResult instance but the scenario title should be used instead:

[BeforeScenario]
public void BeforeScenario(FeatureInfo feature, ScenarioInfo scenario)
{
    var skipHandler = new SkippedStepHandler();
    this._objectContainer.RegisterInstanceAs<ISkippedStepHandler>(skipHandler);

    _result = new(
        ExecutionContextType,
        feature.FolderPath,
        feature.Title,
        feature.Title,    // should be scenario.Title
        scenario.Arguments.Values.Cast<string>().ToArray());

    this._objectContainer.RegisterInstanceAs(_result);
}

[marcelstuder]

@Registeel FYI: there is a small issue that I ran into when upgrading to ReqNRoll v2.3.0:

As the deprecated dependency to Specflow.Internal.Json has been removed in ReqNRoll v2.3.0 this code from @dakiers project in LivingDocHooks.cs won't work anymore:

    [AfterTestRun(Order = AfterOrder)]
    public static async Task AfterTestRun()
    {
        _execution!.GenerationTime = DateTime.UtcNow;

        await File.WriteAllTextAsync(FileName, _execution.ToJson(new(false)));
    }

As an alternative System.Text.Json.JsonSerializer can be used (as discussed here). The example below uses the JsonStringEnumConverter so that enum values will be serialized using their string representations (which is what the removed Specflow.Internal.Json dependency did, otherwise the generated file won't work properly with LivingDoc):

    [AfterTestRun(Order = AfterOrder)]
    public static async Task AfterTestRun()
    {
        _execution!.GenerationTime = DateTime.UtcNow;

        // See https://github.com/orgs/reqnroll/discussions/68 - write json file to support LivingDoc
        var serializerOptions = new JsonSerializerOptions
        {
            Encoder = JavaScriptEncoder.UnsafeRelaxedJsonEscaping,
            WriteIndented = true,
            Converters = { new JsonStringEnumConverter() }  // do not use underlying enum values, use the string representation
        };
        await File.WriteAllTextAsync("TestExecution.json", JsonSerializer.Serialize(_execution, serializerOptions), Encoding.UTF8);
    }