Generic latency slo

[brancz]

This PR has 3 changes:

• Introduce notErrorSelector

The notErrorSelector allows using the latency burn function for metrics
other than those that have errors defined as not 5xx.

• Template latency rules with named fields …

Reading and understanding the code was very difficult as the parameters
had to be counted, now it's much easier to reason about.

• Add note about requirement of histogram bucket

@metalmatze

If I'm not mistaken then this should largely solve #15

[brancz]

Regarding the last one, I can't help but wonder that there must be a better way to do this than requiring the exact histogram bucket to be present.

[brancz]

Somewhat related, is it really correct to compare non-errors with all requests here?

slo-libsonnet/slo-libsonnet/latency-burn.libsonnet

Line 28 in e238df4

sum(rate(%s{%s}[%s]))

Isn't the point that we exclude any sort of errors?

[metalmatze]

Simple and straight forward! 💯
Thanks!

[metalmatze]

Regarding the last one, I can't help but wonder that there must be a better way to do this than requiring the exact histogram bucket to be present.

That's probably something that should be taken care of outside of this slo-libsonnet? Something generating rules with this library can check for different buckets and then suggest them.

[brancz]

I think that would help, but if it could be generic to the chosen buckets (and there are investigations of dynamic bucketing) then that would be even better. I'll open an issue to investigate and see if we could find something.