QPS–Latency benchmarking support for streaming workloads

[akhilpathivada]

Hi Team,

I’m evaluating benchmarking approaches for streaming / continuous write & read workloads and wanted to understand how this can be achieved in OpenSearch Benchmark.

I recently had a discussion in VectorDBBench regarding QPS–Latency style benchmarking for streaming workloads: zilliztech/VectorDBBench#638

Does OpenSearch Benchmark support QPS vs latency style benchmarking (e.g., fixed or controlled throughput with latency measurement) for streaming scenarios?

If not directly supported, what would be the recommended way to achieve this?

[OVI3D0]

hey @akhilpathivada ! thanks for opening the issue.

I quickly skimmed through the discussion linked and just want to ensure I understand your question. To sum it up, you're wondering if OSB can run a benchmark at different concurrency levels like 5 clients, 10, 20 etc, and measure the QPS/latency at a per-concurrency level rather than just returning overall QPS and latency?

We got a similar question recently #1008 and OSB does support specifying multiple search clients in one benchmark run: #614

There is also Redline testing https://docs.opensearch.org/latest/benchmark/reference/commands/redline-test/
Which is meant for finding the max QPS/Latency a cluster can support. So, if you want your domain to support queries with a max of N query Latency, you can set the client-timeout number and redline testing will find the max QPS supported at that latency level.