Skip to content

Reduce lock contention in SyncMetricStorage for concurrent metric recording#3959

Open
perhapsmaple wants to merge 3 commits intoopen-telemetry:mainfrom
perhapsmaple:optimize-concurrent-metric-recording
Open

Reduce lock contention in SyncMetricStorage for concurrent metric recording#3959
perhapsmaple wants to merge 3 commits intoopen-telemetry:mainfrom
perhapsmaple:optimize-concurrent-metric-recording

Conversation

@perhapsmaple
Copy link
Copy Markdown
Contributor

Fixes #3927

Changes

First set of changes to reduce lock contention: Move MetricAttributes construction outside the lock

Benchmark before:

Date: 2026-03-31
Build: Release, CXX14, AppleClang 17.0.0, arm64 (M-series, 8 cores)
Repetitions: 5
Threads: 4 (default)

Benchmark                                          Time (ns)     CPU (ns)    Iterations
---------------------------------------------------------------------------------------
BM_MeasurementsTest (1 thread)
  mean                                              2,237,371       12,722        10,000
  median                                            2,234,953       12,774        10,000
  stddev                                                6,464          215
  cv                                                    0.29%         1.69%

BM_MeasurementsThreadsShareCounterTest (4 threads, shared counter)
  mean                                              4,353,910       58,109         1,000
  median                                            4,471,432       59,356         1,000
  stddev                                              250,165        2,399
  cv                                                    5.75%         4.13%

BM_MeasurementsPerThreadCounterTest (4 threads, per-thread counters)
  mean                                              1,120,472       47,976        10,000
  median                                            1,122,614       49,069        10,000
  stddev                                               99,805        1,933
  cv                                                    8.91%         4.03%

Benchmark after:

Date: 2026-03-31
Build: Release, CXX14, AppleClang 17.0.0, arm64 (M-series, 8 cores)
Repetitions: 5
Threads: 4 (default)

Benchmark                                          Time (ns)     CPU (ns)    Iterations
---------------------------------------------------------------------------------------
BM_MeasurementsTest (1 thread)
  mean                                              2,230,225       11,958        10,000
  median                                            2,229,963       11,957        10,000
  stddev                                                  826         24.4
  cv                                                    0.04%         0.20%

BM_MeasurementsThreadsShareCounterTest (4 threads, shared counter)
  mean                                              2,904,767       51,929         1,000
  median                                            2,904,170       51,912         1,000
  stddev                                               17,484          800
  cv                                                    0.60%         1.54%

BM_MeasurementsPerThreadCounterTest (4 threads, per-thread counters)
  mean                                              1,031,435       45,674        10,000
  median                                            1,038,361       46,468        10,000
  stddev                                               64,472        1,719
  cv                                                    6.25%         3.76%

Shared-counter wall time dropped from 4,354us to 2,905us (33% reduction).

For significant contributions please make sure you have completed the following items:

  • CHANGELOG.md updated for non-trivial changes
  • Unit tests have been added
  • Changes in public API reviewed

Signed-off-by: Harish <140232061+perhapsmaple@users.noreply.github.com>
@perhapsmaple perhapsmaple requested a review from a team as a code owner March 31, 2026 16:31
@codecov
Copy link
Copy Markdown

codecov bot commented Mar 31, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.13%. Comparing base (0b1c704) to head (0eef2cc).

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #3959      +/-   ##
==========================================
+ Coverage   90.13%   90.13%   +0.01%     
==========================================
  Files         230      230              
  Lines        7293     7294       +1     
==========================================
+ Hits         6573     6574       +1     
  Misses        720      720              
Files with missing lines Coverage Δ
...ntelemetry/sdk/metrics/state/sync_metric_storage.h 89.19% <100.00%> (+0.62%) ⬆️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

std::lock_guard<opentelemetry::common::SpinLockMutex> guard(attribute_hashmap_lock_);
attributes_hashmap_
->GetOrSetDefault(attributes, attributes_processor_.get(), create_default_aggregation_)
attributes_hashmap_->GetOrSetDefault(std::move(attr), create_default_aggregation_)
Copy link
Copy Markdown
Contributor

@ThomsonTan ThomsonTan Mar 31, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The overload of GetorSetDefault which accepts KeyValueIterable could be removed with this change.

Aggregation *GetOrSetDefault(
const opentelemetry::common::KeyValueIterable &attributes,
const AttributesProcessor *attributes_processor,
nostd::function_ref<std::unique_ptr<Aggregation>()> aggregation_callback)
{
// TODO: avoid constructing MetricAttributes from KeyValueIterable for
// hash_map_.find which is a heavy operation

@perhapsmaple
Copy link
Copy Markdown
Contributor Author

Can the cppcheck errors be suppressed? It reports an “access after move” issue here, but this looks like a false positive.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Reduce lock contention in SyncMetricStorage for concurrent metric recording

4 participants