feat: Add Key-Value Store Functionality

[jshlbrd]

Description

• Adds support for read-write and read-only key-value stores
  • New interface with factory (internal/kv)
  • Methods for managing stores (Setup, Close), putting values into stores (Set, SetWithTTL), and retrieving values from stores (Get)
  • KV backends in this PR:
    • AWS DynamoDB (rw)
    • Memory (rw)
    • CSV File (ro)
    • JSON File (ro)
    • Text File (ro)
• Adds a kv_store processor for retrieving and putting items into any KV store, all features of the KV store interface are supported
• Adds two new patterns for the kv_store processor
  • cache-aside pattern: performs a cache-aside by retrieving a value from a configured KV store and running a processor plus updating the KV store if the get action missed
  • Zeek Intelligence Framework pattern: uses the CSV File KV store to load Zeek Intelligence Framework files and perform indicator matching, by default works with Critical Path Security's Intelligence Feeds
• Adds the DynamoDB API call GetItem (required for the DynamoDB KV store)

Motivation and Context

This addresses both #55 and #65 -- both use cases are abstracted into the generic KV store use case, which can be extended with support for future backends. This functionality enables this functionality / future work:

• Idempotent Producers #59
• Idempotent consumers (not ticketed)
• Idempotent processing (now possible via configurations as code)
• Refactoring internal/ip/database into KV store
• Threat intelligence feeds (now possible via configurations as code)
• Simplifying the project's caching strategies

How Has This Been Tested?

Unit tests are passing and all features were locally integration tested.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.

[julieagnessparks]

I like the add of Zeek Intelligence Framework, I was unfamiliar with it but it's a great way to bring in indicators.

Only concern is having documentation for how a user to utilize the test files you added.

[julieagnessparks]

Are there instructions for how someone should use this example test / what should be filled in (TODO)?

[jshlbrd]

Not really, this is meant as an example for developers and not users who interact through configurations. Once we've merged this the user-facing documentation would be added to substation.readme.io, both as a processor and likely in a new section that describes the different types of KV stores.

[jshlbrd]

I like the add of Zeek Intelligence Framework, I was unfamiliar with it but it's a great way to bring in indicators.

Yeah, that's a nice example of how we can create read-only KV stores from almost any file. Those files are just fancy CSV files (they use tab as the delimiter, so technically it's TSV) and this makes it relatively easy to retrieve and parse them.

[shellcromancer]

Nice way to combine the cachine with the lookup processor proposal! Left a few comments on the semantics for the JSON KV and a few other things but overall this looks great! :)

[shellcromancer]

docs: this could be more clear as a doclink

[jshlbrd]

don't think this matters since it's not exported -- check it out here https://pkg.go.dev/github.com/brexhq/substation@v0.8.0/process