Add an ILM action to rollup an index

[csoulios]

Rollup should no longer be a continuous running job. Instead, it should be an action that can be triggered on ILM-managed indices. The action should iterate over all grouping tuples (dimensions) and calculate aggregate metrics (min/max/avg/value_count) generating one document per grouping tuple (dimension).

In the context of this issue we should implement the following:

• Implement a "grouping tuple" field type to generate the tuple at index time for the document.
• Implement a "rollup metric" field type to store aggregation results (min/max/count/avg etc). This metric should provide the correct information to requesting aggregator (e.g. avg agg on rollup_metric will internally fetch sum + count)
• Store document count somewhere on the document (docvalue field?)
• Create an ILM rollup action

[elasticmachine]

Pinging @elastic/es-analytics-geo (:Analytics/Rollup)

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/ILM+SLM)

[TommySedin]

Are there any news on this? We're having a soon-to-be situation with metrics indices needing to be rolled up and I'd want to know if I should build something based on the Rollup mechanic or wait for this. Or do you believe the ILM action will be "backwards compatible" with previous rollup indices?

[cmklar]

waiting for this enhancement, Thanks!

[polyfractal]

@TommySedin it's still very early in the process, so I can't really give any concrete details about how it will end up looking. We're working on the search side of things first, trying to remove the _rollup_search end point before tackling the ILM portion mentioned in this ticket. So right now the ILM side is still a bit vague in terms of implementation.

We definitely want some kind of migration path though. It's not clear how that will turn out: could be that we add logic so that both "old" and "new" rollup indices are handled internally by the search end point, or some way to migrate/reindex data to the new format that the ILM action uses, etc. But our goal is to not leave any currently rolled-up data "left behind"

That's about all I have unfortunately, like I said it's still early in the process so we don't have a lot firm details yet.

[mcascallares]

Will this feature allow for rollup jobs modification? Imagine the scenario where new fields appear in the indices, would it be possible to dynamically add new fields to the existing rollups without the need for recreating the job and losing the historical values?

Thanks

[giladgal]

Our tentative plan is to have rollup as an action in ILM. The rollup configuration will stay pretty much the same, but it will be part of the ILM action. This means to update the rollup, one will only need to update the rollup configuration in the ILM policy, and the next time the ILM policy is executed it performs the rollup with the new parameters, e.g. the new fields.

A future enhancement could be to automatically add fields that are being added to the index, e.g. based on a wildcard or some similar rule.

This is the plan, but keep in mind that we are in the design phase, so the implementation may be different.
@mcascallares