High idle CPU Bento Kinesis consumer

[danielavelez12]

Hi! We’re running a WarpStream agent that hosts Bento pipelines consuming from Kinesis, and the Kinesis consumption loop is burning through the pod’s CPU even at extremely low traffic.

Our environment:

WarpStream agent version: v765 (chart 1.0.1)
Also reproduced on: v774 (chart 1.0.12)
Agent roles: proxy, jobs, pipelines
Non-mTLS agents
Embedded Bento version: warpstreamlabs/bento@v1.14.1

The workload:

4 Kinesis CDC streams
Total traffic is extremely low: about 488 records/day across all 4 streams
This appears to be idle polling overhead, not load-related CPU usage

Observed behavior:

With GOMAXPROCS=1, it burns ~1 full core
With GOMAXPROCS=2, it expands and burns ~2 full cores

CPU profile
The CPU profile shows ~96% of CPU in:
github.com/warpstreamlabs/bento/internal/impl/aws.(*kinesisReader).runConsumer

Top contributors include:

runtime.selectgo (~57%)
time.After / time.NewTimer (~31%)
runtime.nanotime (~15%)
runtime.lock2 (~13%)
additional GC pressure from short-lived timer and channel allocations

CPU profile dumps:
ws-cpu-profile-2-top.txt
ws-cpu-profile-text.txt

Is this a known issue in the embedded Bento Kinesis consumer? Is there any config available to avoid this hot-loop behavior?

[jem-davies]

Hey 👋 - thanks for the report.

Going to get some benchmarks written for this component and take a look at the latest Bento version.

As you point out - if this is being caused by a busy select loop then we should be able to get that sorted.

[danielavelez12]

Sounds good, thank you Jem!

[jem-davies]

@danielavelez12 - are you able to share the bento config that was being run here? thanks 🙏

[jem-davies]

I have been trying to get a benchmark that is the same as reported by setting up a low-volume kinesis stream but unable to reproduce the same profile with high CPU use in runConsumer ...

[danielavelez12]

Yep! We produced the issue (99% CPU usage) with this config, and with a completely empty stream

input:
    aws_kinesis:
        batching:
            count: 100
            period: 1s
        dynamodb:
            create: true
            table: bento-kinesis-checkpoints
        region: us-east-1
        streams:
            - test-kinesis-cdc-staging
output:
    fallback:
        - kafka_franz_warpstream:
            idempotent_write: false
            key: ${! metadata("kinesis_partition_key") }
            topic: ${! metadata("kinesis_stream") }
        - kafka_franz_warpstream:
            idempotent_write: false
            topic: pipeline.test-kinesis-cdc-staging.dlq

[jem-davies]

@danielavelez12 - the stream is that provisioned etc. / how many shards does it have etc.?

[jem-davies]

@danielavelez12 - ok I am seeing :

Type: cpu
Time: 2026-03-27 17:59:15 GMT
Duration: 107.70s, Total samples = 92.99s (86.34%)
Showing nodes accounting for 81.56s, 87.71% of 92.99s total
Dropped 555 nodes (cum <= 0.46s)
      flat  flat%   sum%        cum   cum%
    14.27s 15.35% 15.35%     15.23s 16.38%  runtime.nanotime1
     8.12s  8.73% 24.08%        33s 35.49%  runtime.selectgo
     5.31s  5.71% 29.79%      5.31s  5.71%  runtime.lock2
     4.90s  5.27% 35.06%      6.25s  6.72%  runtime.unlock2
        4s  4.30% 39.36%      4.09s  4.40%  runtime.walltime
     3.85s  4.14% 43.50%      5.16s  5.55%  runtime.tryDeferToSpanScan
     3.47s  3.73% 47.23%      3.47s  3.73%  runtime.kevent
     3.28s  3.53% 50.76%     13.46s 14.47%  runtime.mallocgcSmallScanNoHeader
     2.02s  2.17% 52.93%     17.25s 18.55%  runtime.nanotime (inline)
     1.76s  1.89% 54.82%      2.82s  3.03%  runtime.scanblock
     1.73s  1.86% 56.68%      1.73s  1.86%  runtime.memclrNoHeapPointers
     1.58s  1.70% 58.38%      4.06s  4.37%  runtime.scanObjectsSmall
     1.54s  1.66% 60.04%      1.54s  1.66%  runtime.madvise
     1.32s  1.42% 61.46%      3.04s  3.27%  runtime.scanObject
     1.29s  1.39% 62.85%     76.06s 81.79%  github.com/warpstreamlabs/bento/internal/impl/aws.(*kinesisReader).runConsumer.func2
     1.24s  1.33% 64.18%      1.24s  1.33%  syscall.rawsyscalln

with that config 👍 & provisioned kinesis with 1000 shards and GOMAXPROCS = 1

[jem-davies]

@danielavelez12

Is there any config available to avoid this hot-loop behavior?

It seems removing the batching_policy in the aws_kinesis input does alter the profile ...

Will have to investigate this further still.

[danielavelez12]

Sure, let me know if I can help