Checksum docvalues files when load store metadata

[dnhatn]

In CorruptedFileIT, we corrupt a random segment file then check that the corrupting store is marked as corrupted. However, if a DocValues file is flipped, we will trip an assertion in a FixedBitSet when opening a SoftDeletesDirectoryReaderWrapper. This commit makes sure that we verify the DocValues files when loading the metadata from a store.

Another option is to limit the set of corrupting files in the tests.

[elasticmachine]

Pinging @elastic/es-distributed

[dnhatn]

The stack trace:

WARNING: Uncaught exception in thread: Thread[elasticsearch[node_s1][fetch_shard_store][T#4],5,TGRP-CorruptedFileIT]
java.lang.AssertionError: index=373, numBits=283
	at __randomizedtesting.SeedInfo.seed([CFF5E7C9656807DF]:0)
	at org.apache.lucene.util.FixedBitSet.get(FixedBitSet.java:181)
	at org.apache.lucene.index.PendingSoftDeletes.applySoftDeletes(PendingSoftDeletes.java:116)
	at org.apache.lucene.index.SoftDeletesDirectoryReaderWrapper.wrap(SoftDeletesDirectoryReaderWrapper.java:115)
	at org.apache.lucene.index.SoftDeletesDirectoryReaderWrapper$SoftDeletesSubReaderWrapper.wrap(SoftDeletesDirectoryReaderWrapper.java:95)
	at org.apache.lucene.index.FilterDirectoryReader$SubReaderWrapper.wrap(FilterDirectoryReader.java:56)
	at org.apache.lucene.index.FilterDirectoryReader$SubReaderWrapper.access$000(FilterDirectoryReader.java:51)
	at org.apache.lucene.index.FilterDirectoryReader.<init>(FilterDirectoryReader.java:83)
	at org.apache.lucene.index.SoftDeletesDirectoryReaderWrapper.<init>(SoftDeletesDirectoryReaderWrapper.java:53)
	at org.apache.lucene.index.SoftDeletesDirectoryReaderWrapper.<init>(SoftDeletesDirectoryReaderWrapper.java:49)
	at org.elasticsearch.common.lucene.Lucene.getExactNumDocs(Lucene.java:159)
	at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:887)
	at org.elasticsearch.index.store.Store$MetadataSnapshot.<init>(Store.java:797)
	at org.elasticsearch.index.store.Store.getMetadata(Store.java:293)
	at org.elasticsearch.index.store.Store.getMetadata(Store.java:258)

[s1monw]

I don't think we can force the checksum on the metadata retrieval this is just too costly. We should try to find a better way of doing this.

[bleskes]

I wonder why this is an assertion and not a hard IndexOutOfBounds exception. Also, it feels like we should pre check this higher up and throw an Index corrupted exception if things don't add up?

[s1monw]

@bleskes we try to make this not needed on the lucene end. stay tuned

[dnhatn]

Thanks @s1monw and @bleskes for taking a look. This is resolved by LUCENE-8341.