Improve documentation around client-side bottlenecks #1421
Description
We started discussing this in #1397 but offline discussions show that we were not fully aligned on what should happen. The relevant comments from @dliappis and @michaelbaamonde:
It occurred to me that we could at least document this best practice (among others in the future perhaps) in a section that is linked in all the warnings callouts we currently have for custom parameter sources, custom runners and custom schedulers. WDYT?
Expanding this a bit, I'd love a section in the docs that explains what we mean by the "hot path"/"performance-critical code-path", but also more generally a high-level overview of why this is so important in the first place. Reference material on client-side bottlenecks would be great: why they're so crucial to avoid, how they manifest themselves, what we do in Rally to avoid them, etc. You can piece some of this together with what's documented already, but a conceptual overview would help a lot.
As mentioned by @michaelbaamonde, this information is partly scattered across the docs, and could benefit from a dedicated section. What I'm suggesting:
- Add a new section named "Client-side performance bottlenecks" in the "Tips and tricks" page (or somewhere else more appropriate)
- Explain in that section that we need to avoid client-side bottlenecks in Rally to avoid skewing the results, possibly reusing some of the wording in https://www.elastic.co/blog/seven-tips-for-better-elasticsearch-benchmarks (sections 4. and 5.)
- In practice, mention that as a user you can introduce such bottlenecks with custom parameter sources, custom runners and custom schedulers
- Explain that if you identified a client-side bottleneck (with USE for example), you can use driver profiling to see what is slow in practice
- Update the warnings in customer parameter sources, runners and schedulers to point to this new section instead of just the driver profiling docs