cpd: Add Go/rpk based throughput benchmark by StephanDollberg · Pull Request #29772 · redpanda-data/redpanda

StephanDollberg · 2026-03-07T23:24:30Z

Adds a produce throughput benchmark to rpk and adds a CPD test for it.

Note this still uses two shards (which introduces some inherent noise due to unlucky connection shard placement). To fix that we might need to add a custom load balancer thingy to redpanda (to shuffle produce/consume connections correctly).

Backports Required

Release Notes

none

Copilot

Pull request overview

Adds a new rpk benchmark CLI command (Go/franz-go based) and integrates it into the ducktape/CPD test framework via a new RpkBenchmarkService, plus a perf test and a service smoke test.

Changes:

Add rpk benchmark command to create a topic, optionally wait for balanced leadership, run a producer workload, and emit JSON metrics.
Add RpkBenchmarkService ducktape wrapper that runs the benchmark remotely and writes a result.json for CPD.
Add a perf test and a basic service self-test for the new benchmark wrapper.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tests/rptest/tests/rpk_benchmark_service_test.py	New smoke test validating the ducktape benchmark wrapper produces usable metrics.
tests/rptest/services/rpk_benchmark_service.py	New ducktape service wrapper to execute `rpk benchmark` and collect metrics.
tests/rptest/services/redpanda.py	Add return type annotation to `find_binary`.
tests/rptest/perf/rpk_benchmark_test.py	New perf/CPD test that runs the benchmark and writes result metrics.
src/go/rpk/pkg/cli/root.go	Registers the new `benchmark` subcommand with the root CLI.
src/go/rpk/pkg/cli/benchmark/benchmark.go	Implements the benchmark CLI command, topic lifecycle, leadership check, workload, and metrics output.
src/go/rpk/pkg/cli/benchmark/BUILD	Bazel target for the new benchmark command package.
src/go/rpk/pkg/cli/BUILD	Wires the new benchmark package into the CLI build deps.

src/go/rpk/pkg/cli/benchmark/benchmark.go

tests/rptest/services/rpk_benchmark_service.py

vbotbuildovich · 2026-03-08T00:43:53Z

CI test results

test results on build#81480

test_class	test_method	test_arguments	test_kind	job_url	test_status	passed	reason	test_history
WriteCachingFailureInjectionE2ETest	test_crash_all	{"use_transactions": false}	integration	https://buildkite.com/redpanda/redpanda/builds/81480#019ccaaf-eb27-4742-b388-81011d061513	FLAKY	10/11	Test PASSES after retries.No significant increase in flaky rate(baseline=0.0907, p0=1.0000, reject_threshold=0.0100. adj_baseline=0.2482, p1=0.0577, trust_threshold=0.5000)	https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=WriteCachingFailureInjectionE2ETest&test_method=test_crash_all

test results on build#81485

test_class	test_method	test_arguments	test_kind	job_url	test_status	passed	reason	test_history
TestReadReplicaService	test_identical_lwms_after_delete_records	{"cloud_storage_type": 1, "partition_count": 5}	integration	https://buildkite.com/redpanda/redpanda/builds/81485#019ccd97-c2ae-4348-a6a3-2e4e6ca2d6b4	FLAKY	18/21	Test PASSES after retries.No significant increase in flaky rate(baseline=0.0537, p0=0.2920, reject_threshold=0.0100. adj_baseline=0.1526, p1=0.3925, trust_threshold=0.5000)	https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=TestReadReplicaService&test_method=test_identical_lwms_after_delete_records

test results on build#81486

test_class	test_method	test_arguments	test_kind	job_url	test_status	passed	reason	test_history
DirectConsumerVerifierTest	test_basic_consuming_from_topic	{"failure_mode": "RANDOM"}	integration	https://buildkite.com/redpanda/redpanda/builds/81486#019cce21-3d2a-48fa-9fdf-733cbfdf31b9	FLAKY	10/11	Test PASSES after retries.No significant increase in flaky rate(baseline=0.0000, p0=1.0000, reject_threshold=0.0100. adj_baseline=0.1000, p1=0.3487, trust_threshold=0.5000)	https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=DirectConsumerVerifierTest&test_method=test_basic_consuming_from_topic
CloudTopicsL0GCNodeFailureTest	test_node_failure_mid_gc	{"cloud_storage_type": 2}	integration	https://buildkite.com/redpanda/redpanda/builds/81486#019cce21-3d2d-4a09-8dc5-564a34f63a12	FLAKY	10/11	Test PASSES after retries.No significant increase in flaky rate(baseline=0.0269, p0=1.0000, reject_threshold=0.0100. adj_baseline=0.1000, p1=0.3487, trust_threshold=0.5000)	https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=CloudTopicsL0GCNodeFailureTest&test_method=test_node_failure_mid_gc

graham-rp

Couple of ergonomic concerns, but I think the in-the-weeds logic looks good. Back a few steps though, have you thought about a more interactive version of this? I'm thinking it could be a pretty good use of timers, spinner, etc

graham-rp · 2026-03-10T14:00:06Z

tests/rptest/perf/rpk_benchmark_test.py

+            warmup_s=20,
+            duration_s=60,


do we need the full 80s during tests?

Yes, smoothes out some of the noise.

graham-rp · 2026-03-10T14:09:09Z

src/go/rpk/pkg/cli/benchmark/benchmark.go

+
+	cmd := &cobra.Command{
+		Use:   "benchmark",
+		Short: "Run a Kafka benchmark",


we could probably use a Long: set of instructions here

graham-rp · 2026-03-10T14:10:32Z

src/go/rpk/pkg/cli/benchmark/benchmark.go

+	cmd.Flags().Int32VarP(&partitions, "partitions", "p", 18, "Number of partitions for benchmark topic creation")
+	cmd.Flags().Int16VarP(&replicas, "replicas", "r", 3, "Replication factor for benchmark topic creation")
+	cmd.Flags().IntVar(&clients, "clients", 16, "Number of producer client connections")
+	cmd.Flags().IntVar(&recordSize, "record-size", 100, "Record payload size in bytes")


out of curiosity, why these numbers?

It's a small batch size high RPS workload which generally stresses brokers the most.

graham-rp · 2026-03-10T14:19:16Z

src/go/rpk/pkg/cli/benchmark/benchmark.go

+	cmd.Flags().IntVar(&recordSize, "record-size", 100, "Record payload size in bytes")
+	cmd.Flags().IntVar(&warmupS, "warmup", 10, "Warmup duration in seconds")
+	cmd.Flags().IntVar(&durationS, "duration", 60, "Measurement duration in seconds")
+	cmd.Flags().StringVar(&metricsJSON, "metrics-json", "", "Optional path to write final metrics JSON")


I'm wondering if there's a way to get this into the regular --format type structure. Maybe suppress output if --format=json?

Let me see what that does. Having output would still be useful for debugging purposes. Only the final "stats" are really needed in json for tool usage.

Right, yeah had a look and I am not sure it's 100% fit. I prefer still having status output.

graham-rp · 2026-03-10T14:28:03Z

src/go/rpk/pkg/cli/benchmark/benchmark.go

+				}(cl)
+			}
+
+			ticker := time.NewTicker(time.Second)


Instead of having the ticker and printStats switching between phases, I'm thinking that using time.After to start the second phase may be more direct

I am not entirely following. The ticker here in the end is really just for periodic printing.

The second/after-warmup phase is started like you suggest using time.After/Before in both the printer and metrics collection in the produceloop.

graham-rp · 2026-03-10T14:30:12Z

src/go/rpk/pkg/cli/benchmark/benchmark.go

+	if final {
+		prefix = "final "
+	}
+	fmt.Printf("%srequests/s=%.2f MB/s=%.2f errors=%d\n", prefix, m.RequestsPerSec, m.MBPerSec, m.Errors)


I think the canonical way to do this would be out.NewTable

graham-rp · 2026-03-10T15:01:59Z

src/go/rpk/pkg/cli/benchmark/benchmark.go

+	topic string,
+	payload []byte,
+	measureStart time.Time,
+	stats *stats,


$0.02: I think this sort of violates the ideas of CSP. It'd probably be more idiomatic to send results down a channel, but I think it's probably fine here?

Channels are quite expensive for what is basically two atomnic increments.

StephanDollberg · 2026-03-10T17:15:37Z

Couple of ergonomic concerns, but I think the in-the-weeds logic looks good. Back a few steps though, have you thought about a more interactive version of this? I'm thinking it could be a pretty good use of timers, spinner, etc

For now this mostly an internal (hidden) tool with the benefit of it being in rpk that it's basically everywhere available where PR is deployed. We are still considering making it a separate app. Later we might decided to make it generally available at which point we might be able to tune visuals.

Adds a (hidden) benchmark subcommand. For now it's only a simple produce RPS throughput benchmark. Right now we mostly use OMB. However OMB has the disadvantage that it's basically just latency benchmark while we are really missing a throughput benchmark. Equally rdkafka_performance has the disadvantage that the code is a bit rough but more importantly it's single connection and librdkafka suffers from kafka client C10K problem. Using the go client (as is done rpk) avoids this issue.

Adds a service class for the new rpk subcommand. Note this command potentially runs for longer so can't be part of the normal rpk commands (this would run into DT timeouts otherwise).

Add a CPD produce RPS throughput test. We keep the OMB result.json result file which means the results get automatically ingested into perf db. Mostly optimized for noise. NB. reactor util is not at 100% here.

StephanDollberg requested review from ballard26 and travisdowns March 7, 2026 23:24

StephanDollberg requested review from kbatuigas and r-vasquez as code owners March 7, 2026 23:24

Copilot AI review requested due to automatic review settings March 7, 2026 23:24

StephanDollberg requested a review from a team as a code owner March 7, 2026 23:24

github-actions bot added area/rpk area/build labels Mar 7, 2026

StephanDollberg mentioned this pull request Mar 7, 2026

cpd: Add rdkafka based produce throughput bench #29732

Closed

7 tasks

Copilot started reviewing on behalf of StephanDollberg March 7, 2026 23:25 View session

Copilot AI reviewed Mar 7, 2026

View reviewed changes

src/go/rpk/pkg/cli/benchmark/benchmark.go Outdated Show resolved Hide resolved

src/go/rpk/pkg/cli/benchmark/benchmark.go Show resolved Hide resolved

src/go/rpk/pkg/cli/benchmark/benchmark.go Show resolved Hide resolved

tests/rptest/services/rpk_benchmark_service.py Show resolved Hide resolved

StephanDollberg force-pushed the stephan/bencher branch 2 times, most recently from d5dbb9c to 39e9161 Compare March 7, 2026 23:35

StephanDollberg force-pushed the stephan/bencher branch 2 times, most recently from 34cfa88 to 6e3351b Compare March 8, 2026 15:37

graham-rp reviewed Mar 10, 2026

View reviewed changes

StephanDollberg added 5 commits March 10, 2026 17:18

rpk: Use table writer in benchmark tool

d9a2b77

tests: Add DT service for rpk benchmark

d0d9c9c

Adds a service class for the new rpk subcommand. Note this command potentially runs for longer so can't be part of the normal rpk commands (this would run into DT timeouts otherwise).

tests: Add smoke test for rpk benchmark DT service

282eebc

tests: Add rpk benchmark CPD test

51a0920

Add a CPD produce RPS throughput test. We keep the OMB result.json result file which means the results get automatically ingested into perf db. Mostly optimized for noise. NB. reactor util is not at 100% here.

StephanDollberg force-pushed the stephan/bencher branch from 6e3351b to 51a0920 Compare March 10, 2026 17:28

Conversation

StephanDollberg commented Mar 7, 2026

Backports Required

Release Notes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vbotbuildovich commented Mar 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CI test results

Uh oh!

graham-rp left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

StephanDollberg commented Mar 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

vbotbuildovich commented Mar 8, 2026 •

edited

Loading