feat(producer): add MaxBufferBytes to limit retry buffer size

[wanwenli]

Summary

This PR introduces the MaxBufferBytes configuration to Sarama’s producer, allowing users to limit the retry buffer size based on the total byte size of messages. This complements the existing MaxBufferLength configuration, which limits the number of messages in the retry buffer. The implementation ensures more robust control over memory usage and prevents potential OOM issues when dealing with large messages.

The changes are built upon the foundation laid in #3026, further enhancing the retry buffer’s configurability and usability.

Key Changes

• Added a new Producer.Retry.MaxBufferBytes configuration.
• The type of MaxBufferBytes is int64, as the maximum value of int32 (approximately 2 GB) is insufficient for buffers holding many messages of large average size, such as 1 MB. Using int64 ensures the buffer can handle higher limits if needed.
• Ensured that any value below the functional lower limit of 32 MB is pushed to this minimum.
• Updated the retry handler to enforce both MaxBufferBytes and MaxBufferLength constraints.
• Added relevant unit tests to validate functionality for both configurations, including edge cases and scenarios with unlimited settings.

This enhancement allows users to fine-tune producer behavior, particularly when working with large message payloads or memory-constrained environments.

[wanwenli]

@dnwe can you please review this PR? Thanks!

[puellanivis]

The only thing left to comment about is a possible alternative, that may or may not be an improvement.

Otherwise, this looks like it does what it intends.

[dnwe]

Thank you for this contribution, and for refining it based on the review feedback. The changes LGTM!