Update default exemplar reservoir size for base-2 exponential histogram buckets

[MrAlias]

Originally posted by @jsuereth in #3670 (comment)

I'd suggest limiting to a range of 5-20 exemplars (or some division of the exponential histogram's max size) purely from an overhead standpoint on tracking these points. I don't have hard numbers to back this up, just some anecdotes.

Here's some VERY lazy math with naive assumptions:

Explicit Bucket histograms

• Default 15 boundaries, 16 buckets
• 1:1 exemplar to bucket
• Likely storage per bucket count = 8 bytes
• Max, min, sum storage = 8*3 = 24 bytes
• Memory overhead without Exemplars: 168 + 83 = 152 bytes
• Likely storage per exemplar (best-case pointer to shared attributes, pointer to shared context + measurement + timestamp) = 32 bytes
• Memory head just for Exemplars = 512 bytes
• Total overhead / histogram = 664 bytes

Exponential Bucket Histogram

• Default 320 buckets + overflow (Let's assume negative measurements don't exist)
• Likely storage per bucket count = ~2 bytes (assuming we're using "adapting integer" algorithm where we scale arrays as counts grow, and we're in a sweet spot of measurement-counts)
• max, min, sum storage = 8*3 bytes
• Memory overhead without exemplars - ~640 + 24 = 664 bytes

Given the key size differential, I don't think we should be spending quite as much memory overhead on exemplars. Particularly because Exponential Buckets are intended to give us very high accuracy percentiles. I think we're better of attempting to grab lower number but higher entropy exemplars.

[reyang]

Another thing for us to consider - maybe most folks would care more about exemplars for the lowest (including the Zero Counts for base2 exponential buckets histogram) and the highest buckets?

[MrAlias]

I think we could recommend a weighted sampling algorithm for the exponential histogram exemplar reservoirs.

One of the only things we will still need to nail down is the weighting distribution.