Add code action for generating a test from rule evaluation

[anderseknert]

The Evaluate feature enables exploratory policy development, where changes to either policy, input or data immediately are reflected in the evaluation result. This goes a long way to build understanding and confidence in the policy as it's being developed. Ideally, we'd be able to transfer this confidence into something that persists past the development session! I believe one way to do this would be to provide an option to generate a test from the same state that the workspace is in when the user clicks Evaluate.

Given a (overly trivial) workspace like this:

policy.rego

package policy

allow if {
    input.foo == "foo"
    data.bar == "bar"
}

input.json

{"foo": "foo"}

data.json

{"bar": "bar"}

Clicking Evaluate would of course return true for the given workspace state. Translating the evaluation to a test from that would then render something like:

policy_test.rego

package policy_test

import data.policy

test_allow {
    policy.allow with input.foo as "foo" with data.bar as "bar"
}

Changing e.g. input.json to {"foo": "bar"} would make the result undefined, and we could easily add a test for that condition too:

policy_test.rego

package policy_test

import data.policy

test_allow {
    policy.allow with input.foo as "foo" with data.bar as "bar"
}

test_not_allow {
    not policy.allow with input.foo as "bar" with data.bar as "bar"
}

Exactly how test names should be generated will be left as an exercise to the implementor... but the name should most likely be provided / refined by the user anyway.

This would definitely not be a trivial task with all things considered (identifying and narrowing down dependencies to mock, for one), but I imagine we can do this gradually, and perhaps have a first version copy the whole input document to the test, not consider data.json files, and so on. Even in a crude form requiring some manual edits would still save time compared to writing all tests from scratch.

[srenatus]

https://github.com/open-policy-agent/eopa/tree/main/pkg/test_bootstrap Prior art or maybe a starting point

[anderseknert]

Totally! We should be able to use much of that code as a starting point. And it looks like @philipaconrad already did the hard work of narrowing down dependencies for rules that I mentioned above! The language server has the benefit of actually having eval time data at hand, so instead of trying to cover all possible cases, or to add "EXAMPLE" value placeholders, we'll use the same input/data from the workspace that the user provided for eval. An intuitive way to "record" development state in the form of tests, or so I hope!