feat(transform): Add KV Store Set Add Transform

[jshlbrd]

Description

• Adds a new KV Store function for adding items to a set (unique list)
  • Adds examples/config/transform/enrich/kvstore_set_add
  • Updates internal/kv
• Refactors other KV Store functions to address name collision between item operations and set operations

Motivation and Context

The backend infra used by the KV store feature (primarily DynamoDB, but in the future Redis or other similar systems) support more operations than setting and getting records, so this is a starting point for adding more support to that part of the project. Stating the obvious, this makes it possible to store items as a set / unique list in a KV store; this adds another capability to track state (as indexed values). It's worth mentioning that sets are unordered lists containing unique items.

This required a rename of the existing KV store functions:

• kv_store_get is now kv_store_item_get
• kv_store_set is now kv_store_item_set

The rename isn't breaking, but references in substation.libsonnet were changed from kv_store.get() and kv_store.set() to kv_store.iget() and kv_store.iset(). This gives us more room to add more KV store functions in the future, like:

• kv_store_set_delete (delete item from a set)
• kv_store_list_add (add an item to an ordered list)
• kv_store_item_delete (delete item from store)
• kv_store_number_add (add to a number in a store)
• kv_store_number_increment (increase a number in a store)

How Has This Been Tested?

• Integration tested using the new example.
• End to end testing in AWS is in progress.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.

[shellcromancer]

Requesting changes primary for the casing, but also we have a breaking change in here with w.r.t. application configs

[shellcromancer]

Do you think we should consider changes to this set symbol from an end-user libsonnet functions as breaking changes?

My understanding is that this will break Jsonnet compilation and app loading so I think changes here will require user configurations to do a small migration which we should call out in release notes at the minimum.

[jshlbrd]

There's two scenarios that I can think of with this:

• You upgrade the entire repo (including this file) and the Jsonnet configurations fail to compile. That's an error that would be raised prior to deployment. It's annoying but it won't impact production deployments.
• You upgrade the source code (not including this file) and the Jsonnet configurations successfully compile. There's no impact since enrich_kv_store_set is still accepted by transform.New().

The configs aren't in scope for SemVer (it's mentioned in VERSIONING) partially for this reason, it's not possible to migrate from (e.g.) set() to set: {} in Jsonnet while being backwards compatible. An alternative solution is to fork the libsonnet file, but that kicks the problem down the road because this can happen again. I think the only sensible path forward with that is to make the libsonnet file smaller and bring the condition and transform functions into SemVer.

[shellcromancer]

I think the only sensible path forward with that is to make the libsonnet file smaller and bring the condition and transform functions into SemVer.

I agree with this, I forgot they were out of the projects defined SerVer scope but agree that they should be given how it's the projects primary interface across all examples and documentation. Along the SerVer principles, no matter what the breakage is, it should be clearly documented in the changelog and visible with the tagged release.

Either we should make the breaking change that this is and bump the project to 2.0, or we can use a different container name for the KV operations (such as transform.enrich.kv_store.set_set(...), transform.enrich.kv_store.set_get(...), transform.enrich.kv_store.item_set(...), and transform.enrich.kv_store.item_get(...)) to not make this breaking.

[jshlbrd]

This commit should make this backcompat with the existing config. We can rename these and make changes to the SemVer docs in a future release.