[RollupV2] Rollups of Rollups over all ILM Tiers

[PascalSenn]

I am following #42720, and really like the way the new API and integration into ILM is.

What i have not seen (maybe also just not found), is the ability to rollup based on rollups on different tiers.

Basically the following scenario:

A High frequencey API that ingests a large amount of time series data. The idea is to ingest the data and the pretty quickly roll it up into smaller Indices. <1h hot.

<index>
  |
  |
 hot (<1h)
  |\
  | \
  |  \
  |   \
  |    \
  |     \
  |      \
  |       \
  |   warm (< 1d) <rollup 5 minute> 
  |        |\
  |        | \
 delete    |  \
           |   \
           |    \
           |     \
           |      \
           |       \
           |   cold (< 6M) <rollup 12h> 
           |        |\
           |        | \
         delete     |  \
                    |   \
                    |    \
                    |     \
                    |      \
                    |       \
                    |      frozen (< 2y) <rollup 1d>
                    |        | 
                  delete     | 
                             | 
                           delete     

This would make it possible to store a lot of relevant data without the cost of excessive disk usage. Especially with tracing or network data, it's enough to have a rough idea how the service or the network performed a few months back.

To realize this each phase would need to have support for the rollup API. Also it would have to be possible to make rollups of rollups (i think #70534).

Is this something that you guys have on the radar?

[elasticmachine]

Pinging @elastic/es-analytics-geo (Team:Analytics)

[csoulios]

@PascalSenn thank you for your feedback.

The use case you are describing is the main downsampling scenario we plan to support. In fact, we decided that we must optimize rollups for the downsampling use case.

As we felt that the downsampling operation would be a better fit with our time-series database effort, we decided to rethink rollups as a tsdb operation. We think integrating tsdb and rollups (downsampling) would yield far better user experience as well as more efficient code.

So we have made rollups (downsampling) an important milestone of TSDB (#74660)

[PascalSenn]

@csoulios Awesome thanks for the feedback!