[RFC] [Remote Store] Remote Store Stats API

[linuxpi]

Is your feature request related to a problem? Please describe.
#6789 talks about adding stats related to Remote Store. This will help us identify how we are performing with Remote Store enabled indices. We need Rest API to expose these stats to provide visibility into how the Remote Store enabled indices are performing.

Scope of this issue is to add required API(s) for exposing Remote Store related stats

Describe the solution you'd like

GET /_remote_store/stats/<index>/<shardId>

{
      "shard_id" : "[my-index-1][0]",
      "local_refresh_timestamp_in_millis" : 196439653,
      "local_refresh_cumulative_count" : 0,
      "remote_refresh_timestamp_in_millis" : 196439653,
      "remote_refresh_cumulative_count" : 0,
      "bytes_lag" : 0,
      "rejection_count" : 0,
      "consecutive_failure_count" : 0,
      "total_remote_refresh" : {
        "started" : 0,
        "succeeded" : 0,
        "failed" : 0
      },
      "total_uploads_in_bytes" : {
        "started" : 0,
        "succeeded" : 0,
        "failed" : 0
      },
      "remote_refresh_size_in_bytes" : {
        "last_successful" : 0,
        "moving_avg" : 0.0
      },
      "upload_latency_in_bytes_per_sec" : {
        "moving_avg" : 0.0
      },
      "remote_refresh_latency_in_nanos" : {
        "moving_avg" : 0.0
      }
}

---

Index Level Stats

GET /_remote_store/stats/<index>

[
  {
     "shardId": <>,
     ...
  },
  {
     "shardId": <>,
     ...
  }
  ...
]

---

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[ashking94]

@linuxpi Couple of points -

• We can add a field bytes_behind which gives info on how many bytes are we lagging behind the local store.
• We could allow passing '*' as the value in the stats api - /cat/remote_store/{index". This will allow to fetch stats for all the shards in one api call.
• We could have the api return total sum across all shards present in the cluster.
• We would want the aggregated information per node basis as well. This will provide data on the outgoing traffic to remote store and be a feedback for reallocating the shard across nodes manually/programmatically.
• Currently each remote-backed index allows user to set translog and segments repository. We should also have aggregate on repository level. This will give insights on when a repository is not acting as usual.

[sachinpkale]

@linuxpi Can you please also provide details around the API permissions?

[linuxpi]

@ashking94

We can add a field bytes_behind which gives info on how many bytes are we lagging behind the local store.

Yes. update the structure

We could allow passing '*' as the value in the stats api - /cat/remote_store/{index". This will allow to fetch stats for all the shards in one api call.

Not sure if this is a good idea. if a cluster has many shards the api response become huge

We could have the api return total sum across all shards present in the cluster.

sum of all metrics? I dont think all metrics would make sense when summed

We would want the aggregated information per node basis as well. This will provide data on the outgoing traffic to remote store and be a feedback for reallocating the shard across nodes manually/programmatically.

Node level aggregation would be very useful. but i am planning to add it incrementally

Currently each remote-backed index allows user to set translog and segments repository. We should also have aggregate on repository level. This will give insights on when a repository is not acting as usual.

Thats a good point. We can implement all aggregate level metrics incrementally - cluster, node and repository level

[sachinpkale]

@linuxpi What does started signify under upload_bytes, total_uploads and total_deletes? Can we check existing stats API and use the same naming conventions?

[linuxpi]

@sachinpkale started signifies the bytes/objects sent for upload. succeeded and failed would reflect out of those how many succeeded or failed. started should be equal to succeeded + failed

[linuxpi]

Can we check existing stats API and use the same naming conventions?

@sachinpkale i checked a various stats objects part of ClusterStatsIndices . What i've noticed there is each stat is appended with _in_bytes where-ever application. i'll check more and try to comply as much as possible but if you had anything specific in mind do let me know

[sachinpkale]

@linuxpi Which metric/s will be used to determine the time it takes to complete one run of segment uploads post a refresh? Reference: #7474