The regressions we saw in scroll latency after upgrading to a new Lucene snapshot that more aggressively compresses stored fields made me wonder if we should use the same trick as #61806 of pretending we're doing a merge to speed up stored fields retrieval when doc IDs are adjacent.
