[Remote Store] Fix stats reporting for multistream downloads.

[Rishikesh1159]

Description

This PR reports stats for multistream downloads. With addition of copyTo() in PR proper stats are no longer reported. This PR fixes the issue and reports proper stats.

Related Issues

Resolves #10283

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)
• GitHub issue/PR created in OpenSearch documentation repo for the required public documentation changes (#[Issue/PR number])

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[Rishikesh1159]

This PR fixes the stats reporting for multistream downloads. I have tested the working of fix by following testing:

Testing:
-> Created a cluster with remote store enabled and repository registered.
-> Ingested some data and made call to _remotestore/stats api to get download stats.

-> Without this PR fix (download stats are empty for replicas):

{
 "_shards" : {
  "total" : 2,
  "successful" : 2,
  "failed" : 0
 },
 "indices" : {
  "nyc_taxis" : {
   "shards" : {
    "0" : [
     {
      "routing" : {
       "state" : "STARTED",
       "primary" : true,
       "node" : "eGJVITx2QAmhJ0Y6ywgKvw"
      },
      "segment" : {
       "download" : { },
       "upload" : {
        "local_refresh_timestamp_in_millis" : 1696445223665,
        "remote_refresh_timestamp_in_millis" : 1696445223665,
        "refresh_time_lag_in_millis" : 0,
        "refresh_lag" : 0,
        "bytes_lag" : 0,
        "backpressure_rejection_count" : 0,
        "consecutive_failure_count" : 0,
        "total_uploads" : {
         "started" : 129,
         "succeeded" : 129,
         "failed" : 0
        },
        "total_upload_size" : {
         "started_bytes" : 3756462697,
         "succeeded_bytes" : 3756462697,
         "failed_bytes" : 0
        },
        "remote_refresh_size_in_bytes" : {
         "last_successful" : 0,
         "moving_avg" : 7.698216395E7
        },
        "upload_speed_in_bytes_per_sec" : {
         "moving_avg" : 4.59313421E7
        },
        "remote_refresh_latency_in_millis" : {
         "moving_avg" : 829.4
        }
       }
      },
      "translog" : {
       "upload" : {
        "last_successful_upload_timestamp" : 1696445164677,
        "total_uploads" : {
         "started" : 554,
         "failed" : 0,
         "succeeded" : 554
        },
        "total_upload_size" : {
         "started_bytes" : 3308030813,
         "failed_bytes" : 0,
         "succeeded_bytes" : 3308030813
        },
        "total_upload_time_in_millis" : 209010433,
        "upload_size_in_bytes" : {
         "moving_avg" : 5061984.05
        },
        "upload_speed_in_bytes_per_sec" : {
         "moving_avg" : 7328.3
        },
        "upload_time_in_millis" : {
         "moving_avg" : 715100.5
        }
       },
       "download" : { }
      }
     },
     {
      "routing" : {
       "state" : "STARTED",
       "primary" : false,
       "node" : "W7neivyKQpiIxwffoZCXrw"
      },
      "segment" : {
       "download" : { },
       "upload" : { }
      },
      "translog" : {
       "upload" : { },
       "download" : { }
      }
     }
    ]
   }
  }
 }
}

-> With this PR fix (download stats for replica are reported):

curl localhost:9200/_remotestore/stats/nyc_taxis?pretty
{
  "_shards" : {
    "total" : 2,
    "successful" : 2,
    "failed" : 0
  },
  "indices" : {
    "nyc_taxis" : {
      "shards" : {
        "0" : [
          {
            "routing" : {
              "state" : "STARTED",
              "primary" : true,
              "node" : "2C-7q8IkR4aM4N1Uj1jsJA"
            },
            "segment" : {
              "download" : { },
              "upload" : {
                "local_refresh_timestamp_in_millis" : 1696446555174,
                "remote_refresh_timestamp_in_millis" : 1696446555174,
                "refresh_time_lag_in_millis" : 0,
                "refresh_lag" : 0,
                "bytes_lag" : 0,
                "backpressure_rejection_count" : 0,
                "consecutive_failure_count" : 0,
                "total_uploads" : {
                  "started" : 103,
                  "succeeded" : 103,
                  "failed" : 0
                },
                "total_upload_size" : {
                  "started_bytes" : 2589972453,
                  "succeeded_bytes" : 2589972453,
                  "failed_bytes" : 0
                },
                "remote_refresh_size_in_bytes" : {
                  "last_successful" : 0,
                  "moving_avg" : 5.309439115E7
                },
                "upload_speed_in_bytes_per_sec" : {
                  "moving_avg" : 4.399847085E7
                },
                "remote_refresh_latency_in_millis" : {
                  "moving_avg" : 822.1
                }
              }
            },
            "translog" : {
              "upload" : {
                "last_successful_upload_timestamp" : 1696446532704,
                "total_uploads" : {
                  "started" : 431,
                  "failed" : 0,
                  "succeeded" : 431
                },
                "total_upload_size" : {
                  "started_bytes" : 2596014317,
                  "failed_bytes" : 0,
                  "succeeded_bytes" : 2596014317
                },
                "total_upload_time_in_millis" : 127155781,
                "upload_size_in_bytes" : {
                  "moving_avg" : 5638599.25
                },
                "upload_speed_in_bytes_per_sec" : {
                  "moving_avg" : 10546.45
                },
                "upload_time_in_millis" : {
                  "moving_avg" : 536076.7
                }
              },
              "download" : { }
            }
          },
          {
            "routing" : {
              "state" : "STARTED",
              "primary" : false,
              "node" : "y6DkskhCRECaNG2zq2rF5A"
            },
            "segment" : {
              "download" : {
                "last_sync_timestamp" : 1696446709535,
                "total_download_size" : {
                  "started_bytes" : 2161626590,
                  "succeeded_bytes" : 2161626590,
                  "failed_bytes" : 0
                },
                "download_size_in_bytes" : {
                  "last_successful" : 337,
                  "moving_avg" : 1.18097258E7
                },
                "download_speed_in_bytes_per_sec" : {
                  "moving_avg" : 1.31509003E7
                }
              },
              "upload" : { }
            },
            "translog" : {
              "upload" : { },
              "download" : { }
            }
          }
        ]
      }
    }
  }
}

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/27120/
• CommitID: 3b6aba3
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/27127/
• CommitID: 3b6aba3
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/27162/
• CommitID: 05f48e7
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/27170/
• CommitID: 05f48e7
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/27178/
• CommitID: 05f48e7

[codecov]

Codecov Report

Merging #10402 (05f48e7) into main (2965e69) will decrease coverage by 0.01%.
Report is 12 commits behind head on main.
The diff coverage is 79.45%.

@@             Coverage Diff              @@
##               main   #10402      +/-   ##
============================================
- Coverage     71.16%   71.15%   -0.01%     
- Complexity    58297    58375      +78     
============================================
  Files          4830     4843      +13     
  Lines        274903   275203     +300     
  Branches      40059    40068       +9     
============================================
+ Hits         195637   195830     +193     
- Misses        62882    62965      +83     
- Partials      16384    16408      +24     

Files	Coverage Δ
...arch/telemetry/metrics/DefaultMetricsRegistry.java	100.00% <100.00%> (ø)
...ch/telemetry/metrics/noop/NoopMetricsRegistry.java	100.00% <100.00%> (ø)
...va/org/opensearch/telemetry/metrics/tags/Tags.java	100.00% <100.00%> (ø)
...org/opensearch/repositories/url/URLRepository.java	72.00% <100.00%> (ø)
...g/opensearch/transport/netty4/Netty4Transport.java	73.65% <100.00%> (ø)
...opensearch/repositories/azure/AzureRepository.java	68.88% <100.00%> (-1.33%)	⬇️
...g/opensearch/repositories/hdfs/HdfsRepository.java	61.90% <100.00%> (ø)
...java/org/opensearch/repositories/s3/S3Service.java	75.52% <100.00%> (+0.52%)	⬆️
.../org/opensearch/telemetry/OTelTelemetryPlugin.java	100.00% <100.00%> (ø)
...rg/opensearch/telemetry/OTelTelemetrySettings.java	100.00% <100.00%> (ø)
... and 41 more

... and 492 files with indirect coverage changes