Preserve pre-existing rally-metrics index templates by default

[fressi-elastic]

It handles issue #1900: Rally should not overwrite pre existing templates by default

When opening an EsMetricsStore, it creates the index template if any of following is true:

• the index template doesn't exist
• reporting/datastore.overwrite_existing_templates option is true and there are differences
  between existing and requested template.

It will preserve existing template on all the other cases.
It logs a warning when an existing index template is being replaced.
It logs index template differences between the existing one (if any) and configured one.

[gareth-ellis]

Could we get an update to the docs done too, to explain this feature? Obviously we want to have it tested too before we merge

[fressi-elastic]

Could we get an update to the docs done too, to explain this feature? Obviously we want to have it tested too before we merge

I did tested and it works as expected in all cases except one: one the overwrite_existing_templates is false it looks like it act as it is true. I guess the code is parsing it as string instead of a boolean. I am investigating on it.

[gbanasiak]

In addition to documentation I would also try adding new modules in here to enable full scope of mypy checks:

rally/pyproject.toml

Lines 200 to 206 in 2ef33eb

	[[tool.mypy.overrides]]
	module = [
	"esrally.mechanic.team",
	"esrally.utils.modules",
	"esrally.utils.io",
	"esrally.utils.process",
	]

Context: Until #1798, so recently on Rally timescale, we had no annotations. They were introduced very surgically to reduce the scope. Then in #1859 we introduced mypy overrides that restore full set of checks for selected modules - either new ones, or the ones we are revisiting. As we're planning to broaden annotations it would be best if all new modules were added there. Ultimately the overrides should be removed.

[gbanasiak]

CI failure will be addressed by elastic/rally-tracks#748.

[fressi-elastic]

I am still processing pending comments. I am going to split this PR.

[fressi-elastic]

I took out parts of this PR to some new smaller ones:

• PR Suppress a flaky internal mypy error appearing randomly when executing make lint. #1919
• PR Implements a decorator to facilitate creating table-driven unit tests. #1922
• PR Introduce tools for pretty printing JSON like objects and their differences. #1924
• PR Simplify config.Config implementation  #1925

[gbanasiak]

@fressi-elastic Many thanks for the split. As mentioned earlier, I'm happy with how this works for rally-metrics. The only remaining concern is:

I think the same logic should apply to remaining templates - rally-results, rally-races, and rally-annotations, for consistency. WDYT?

The rationale is the datastore.overwrite_existing_templates setting name is generic, and we have multiple templates, see EsResultsStore and EsRaceStore classes in metrics.py. We can address this in a follow-up PR if you prefer.

[fressi-elastic]

I already started working in a follow-up PR address @gbanasiak comments. Thank you for addressing this point I didn't catch earlier.