Skip to content

feat!: Migrate to Meta Processors#7

Merged
jshlbrd merged 18 commits intomainfrom
jshlbrd/meta-processors
Jul 18, 2022
Merged

feat!: Migrate to Meta Processors#7
jshlbrd merged 18 commits intomainfrom
jshlbrd/meta-processors

Conversation

@jshlbrd
Copy link
Contributor

@jshlbrd jshlbrd commented Jul 18, 2022

Description

This PR introduces meta-processing and migrates all processors to the meta-processing technique. "Meta processors" are any processors that execute other processors; this PR adds two:

  • for_each: accepts JSON array as input and executes the configured processor on each element in the array
  • pipeline: executes a series of processors on JSON or unstructured data

The for_each processor replaces JSON array support in other processors, making this PR a breaking change that requires a new release.

This PR also includes these changes:

  • internal/base64: adds decoding and encoding base64 strings as its own internal package
  • internal/json: Set supports encoding non-UTF8 bytes into base64 encoded strings
  • process/expand: additional keys found in the JSON object are always copied into the expanded JSON objects
  • process/process: adds ProcessorInvalidSettings error, used across all processors

Motivation and Context

Meta-processing reduces complexity in other processors by removing direct JSON array support and opens up some new use cases. For example, it's now possible to decode complex binary structures entirely within JSON objects (e.g., a key-value that is base64 encoded gzipped data) and execute complex processors (e.g., DynamoDB, Lambda) on arrays of values.

How Has This Been Tested?

The two new processors have unit tests (and for_each has unit tests that execute every processor that used to support arrays).

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.

@jshlbrd jshlbrd requested a review from a team as a code owner July 18, 2022 03:45
@jshlbrd jshlbrd removed the request for review from a team July 18, 2022 03:47
@jshlbrd jshlbrd merged commit f0aabce into main Jul 18, 2022
@jshlbrd jshlbrd deleted the jshlbrd/meta-processors branch July 18, 2022 04:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant