Skip to content

fix(transform): Change Kinesis Payload Limit to MB#206

Merged
jshlbrd merged 1 commit intomainfrom
jshlbrd/fix/kinesis-payload-size
Jul 17, 2024
Merged

fix(transform): Change Kinesis Payload Limit to MB#206
jshlbrd merged 1 commit intomainfrom
jshlbrd/fix/kinesis-payload-size

Conversation

@jshlbrd
Copy link
Contributor

@jshlbrd jshlbrd commented Jul 17, 2024

Description

  • Changes the payload size limit for Kinesis Data Streams from MiB to MB

Motivation and Context

Discovered an edge case where the system generated a Kinesis Data Stream payload of more than 5 MB when using the PutRecords API. Changing the limit of each record from 1 MiB (1048576 bytes) to 1 MB (1000000 bytes) fixed the problem. According to the AWS docs, Firehose uses 1 MiB as its message size limit.

How Has This Been Tested?

Tested E2E in the production data pipeline the bug was discovered in.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.

@jshlbrd jshlbrd marked this pull request as ready for review July 17, 2024 14:26
@jshlbrd jshlbrd requested a review from a team as a code owner July 17, 2024 14:26
@jshlbrd jshlbrd merged commit 16baaa0 into main Jul 17, 2024
@jshlbrd jshlbrd deleted the jshlbrd/fix/kinesis-payload-size branch July 17, 2024 14:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants