Adds an operator to create enrich policies

[slDias]

As suggested in Issue #1796, this commit adds a new operator that combines the delete, creation and execution of enrich policies within a single operation.

I've added a new challenge to nyc_taxis track to test this it is equivalent to the esql challenge, it can be found here (I can also open a MR replacing the 'esql' one, if desired).

Happy to change anything if needed.

[craigtaverner]

This looks like a nice addition. But there are a few things to fix or change. At the very least fix the docs. The suggestions I made for supporting more flexible combinations of the delete/create/execute actions could be done here or in a followup PR.

[craigtaverner]

This meta-data section seems to have been incorrectly placed inside the esql section, instead of the create-enrich-policy section below.

[slDias]

Thanks for reviewing it. The docs were mixed during a merge, I've rebased the changes instead so its fixed now.

[craigtaverner]

This link should rather be moved into the policies list item below.

[slDias]

done

[craigtaverner]

Since the operation does more than just create, I think it should be named enrich-policy, and it can delete, create and execute the policy (or any combination of those actions).

[slDias]

renamed it, the combinations will be part of another PR

[craigtaverner]

Shouldn't the flag be inside the operation? Having it outside the operation makes it seem like it is not tied to enrich policy management at all, but part of generic challenges.

[slDias]

fixed, good catch. it was already part of the operation but I've misplace it in the docs.

[craigtaverner]

Since this can do much more than just create the policy, I think it should be renamed EnrichPolicy, and take optional boolean flags for delete, create and execute, and only create requires the policy bodies, while all the rest require at least the policy name.

[slDias]

Renamed it for now, I'll work on a separate PR for the new format.
I gave it some thought, wdyt of removing policies and then having lists on the delete and execute? that gives more control over what is done to each policy.

example:

{
  "operation-type": "enrich-policy",
  "delete": [
    "a-policy-name",
    "another-policy",
    "yet-another-one"
  ],
  "create": {
    "a-policy-name": {
      "match": {
        "indices": "nyc_rate_codes",
        "match_field": "id",
        "enrich_fields": [
          "name"
        ]
      }
    }
  },
  "execute": [
    "a-policy-name",
    "a-completely-different-policy"
  ]
}

I will work on this version if I find the time, since it's easy to change to your suggestion if you prefer later

[slDias]

Got to implement this, it's as a draft in case you decide it's no good, but it is working and ready to review: #2101

[craigtaverner]

I wonder if it would be nicer to have a list instead of an object as the value of policies. Then we could, for example, just list the names if all we want to do it re-execute existing policies:

{
  "name": "re-execute-enrich-policies",
  "operation": {
    "operation-type": "enrich-policy",
    "policies": [ "nyc_rate_codes", "nyc_payment_types" ],
    "delete": false,
    "create": false,
    "execute": true
  }
}

If we have "create": true(which can be the default), then the values inside the list of policies must be objects with a single key, the name of the policy, and it's value would be the policy body. This is one level nesting deeper than your example.

[slDias]

will do that or my suggestion in another PR

[craigtaverner]

I think it would be nicer to have three booleans: delete, create, execute. The defaults could be: false, true, true.

[slDias]

will do that or my suggestion in another PR

[craigtaverner]

If we supported the nested arrays approach I described above, we could have a normalize option here to bring the multiple formats into a single version so the functions below do not need to care about that.

[slDias]

will do that or my suggestion in another PR

[craigtaverner]

What does this mean? Should this be in this PR?

[slDias]

this is a folder to be ignored from the zed IDE, which I use. I've removed it to avoid friction

[gareth-ellis]

@slDias let us know if you have any questions regarding @craigtaverner 's review. I'm going to change this to draft, please switch back to "ready for review" when you have addressed the feedback - thanks!

[craigtaverner]

LGTM

[gareth-ellis]

@elasticmachine update branch

[gareth-ellis]

Thanks @slDias !