Revert "Use SequentialStoredFieldsLeafReader to read Lucene changes"

[dnhatn]

I benchmarked an improvement in CCR and found that changes introduced in #67190 made CCR 10 times slower (1816 seconds to 26515 seconds).

"total_read_remote_exec_time_millis" : 2651518
"total_read_remote_exec_time_millis" : 1816094

With concurrent indexing and Lucene merges, documents in segments are no longer sorted by sequence numbers. And if the index sorting is specified, documents are never sorted by sequence numbers. Using a mergeInstance stored field reader will decompress the whole block, which we will not use.

I labelled this non-issue for an unreleased bug. I am also checking the other usages of mergeInstance in Elasticsearch.

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[dnhatn]

@dliappis Is it possible to have CCR in our nightly benchmark to catch such a kind of issue? Thanks!

[romseygeek]

Well done catching this before it was released! Do we know if it did improve things for the specific case where sequence number order was preserved, or does that basically never happen?

[dliappis]

@dliappis Is it possible to have CCR in our nightly benchmark to catch such a kind of issue? Thanks!

@dnhatn as discussed offline, currently the bare metal infrastructure isn't ready to support such experiments that by definition require different regions, so we'll need to rely on the self service workflow for a bit I am afraid.

[dnhatn]

Thanks @romseygeek + @dliappis.

[dnhatn]

Do we know if it did improve things for the specific case where sequence number order was preserved, or does that basically never happen?

I will look into this.