feat(kinesisfirehose): add built-in data processors to decompress CloudWatch logs and extract messages

[Tietew]

Issue # (if applicable)

Closes #33691.
Closes #20242.

Reason for this change

Data Firehose supports following convenient conversions:

• Decompress CloudWatch logs
• Extract message after decompression of CloudWatch Logs
• Add a new line delimiter when delivering data to Amazon S3

This PR adds above 3 processors because:

• Decompression (without CloudWatchLogProcessing) may need AppendDelimiterToRecord
• CloudWatchLogProcessing requires Decompression

Description of changes

• introduce processors (plural) prop to specify multiple processors
• deprecate processor (singular) prop in favor of processors
• add built-in processor classes:
  • DecompressionProcessor - Decompress CloudWatch logs
  • CloudWatchLogProcessor - Extract message after decompression of CloudWatch Logs
  • AppendDelimiterToRecordProcessor - Add a new line delimiter when delivering data to Amazon S3

Describe any new or updated permissions being added

N/A - No permissions are required by these built-in processors.

Description of how you validated changes

Added unit tests and integ tests

see also #33691 (comment)

Checklist

• My code adheres to the CONTRIBUTING GUIDE and DESIGN GUIDELINES

---

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

[aws-cdk-automation]

(This review is outdated)

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 82.39%. Comparing base (be383a9) to head (a4d86ac).

Additional details and impacted files

@@           Coverage Diff           @@
##             main   #33749   +/-   ##
=======================================
  Coverage   82.39%   82.39%           
=======================================
  Files         120      120           
  Lines        6960     6960           
  Branches     1175     1175           
=======================================
  Hits         5735     5735           
  Misses       1120     1120           
  Partials      105      105           

Flag	Coverage Δ
suite.unit	82.39% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
packages/aws-cdk	∅ <ø> (∅)
packages/aws-cdk-lib/core	82.39% <ø> (ø)

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

✅ Updated pull request passes all PRLinter validations. Dismissing previous PRLinter review.

[paulhcsun]

Hi @Tietew, since this is a stable library we cannot accept any breaking changes. I think the best thing to do is deprecate the property in the interface and then just return some empty object or undefined and update the logic in the renderDataProcessor method

aws-cdk/packages/aws-cdk-lib/aws-kinesisfirehose/lib/private/helpers.ts

Line 113 in 734ca66

function renderDataProcessor(

to ignore this case if the processIdentifier is empty and ignore that case. Let me know what you think.

[Tietew]

@paulhcsun Thank you for comment.
I'll update to return empty duumy object instead of undefined.

Pull request has been modified.

Pull request has been modified.

[Abogical]

LGTM. Thanks!

[mergify]

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

[mergify]

This pull request has been removed from the queue for the following reason: pull request branch update failed.

The pull request can't be updated

rebase conflict between base and head.

You should update or rebase your pull request manually. If you do, this pull request will automatically be requeued once the queue conditions match again.
If you think this was a flaky issue, you can requeue the pull request, without updating it, by posting a @mergifyio requeue comment.

[Tietew]

@Mergifyio update

[mergify]

update

✅ Branch has been successfully updated

[mergify]

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

[mergify]

This pull request has been removed from the queue for the following reason: pull request branch update failed.

The pull request can't be updated

rebase conflict between base and head.

You should update or rebase your pull request manually. If you do, this pull request will automatically be requeued once the queue conditions match again.
If you think this was a flaky issue, you can requeue the pull request, without updating it, by posting a @mergifyio requeue comment.

[Tietew]

@Mergifyio update

[mergify]

update

✅ Branch has been successfully updated

[mergify]

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

[github-actions]

Comments on closed issues and PRs are hard for our team to see.
If you need help, please open a new issue that references this one.