ReplicationTrackerRetentionLeaseTests.testReplicaIgnoresOlderRetentionLeasesVersion fails occasionally

[AthenaEryma]

I ran into this locally, and it reproduces reliably for me on latest master at time of writing. The test appears to pass most of the time, although this particular line fails reproducibly. There aren't any failures of this test in the build stats tracker, so I'm not going to mute it (yet).

./gradlew :server:unitTest -Dtests.seed=ACF05F449C755A1 -Dtests.class=org.elasticsearch.index.seqno.ReplicationTrackerRetentionLeaseTests -Dtests.method="testReplicaIgnoresOlderRetentionLeasesVersion" -Dtests.security.manager=true -Dtests.locale=en-GI -Dtests.timezone=America/St_Thomas -Dcompiler.java=11 -Druntime.java=11

The failure is:

Expected: an empty collection
   >      but: <[RetentionLease{id='3-0', retainingSequenceNumber=979171579384517195, timestamp=5923124907862339051, source='zUdKvmZW'}, RetentionLease{id='3-1', retainingSequenceNumber=6017346890990993807, timestamp=3929571805510614766, source='OzrFxoRl'}, RetentionLease{id='3-2', retainingSequenceNumber=2725439553447783298, timestamp=3071769291030185046, source='CKtLGpoK'}, RetentionLease{id='3-3', retainingSequenceNumber=8598544668823806214, timestamp=7849418724783540755, source='vGgDkvNA'}]>
   >    at __randomizedtesting.SeedInfo.seed([ACF05F449C755A1:51C70B25254BA317]:0)
   >    at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   >    at org.elasticsearch.index.seqno.ReplicationTrackerRetentionLeaseTests.testReplicaIgnoresOlderRetentionLeasesVersion(ReplicationTrackerRetentionLeaseTests.java:362)
   >    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   >    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   >    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   >    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
   >    at java.base/java.lang.Thread.run(Thread.java:834)

[elasticmachine]

Pinging @elastic/es-distributed

[cbuescher]

Still failing on at least 6.x with very similar looking failure:

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+matrix-java-periodic/ES_BUILD_JAVA=java11,ES_RUNTIME_JAVA=zulu11,nodes=immutable&&linux&&docker/220/consoleFull

./gradlew :server:unitTest \
  -Dtests.seed=69F7C45D4D0C2BEE \
  -Dtests.class=org.elasticsearch.index.seqno.ReplicationTrackerRetentionLeaseTests \
  -Dtests.method="testAddOrRenewRetentionLease" \
  -Dtests.security.manager=true \
  -Dtests.locale=it-SM \
  -Dtests.timezone=Asia/Srednekolymsk \
  -Dcompiler.java=11 \
  -Druntime.java=11

Reproduces locally, this is on current 6.x head at 3bca1aa which includes 0169965.
@dnhatn Not sure if this build was missing somehting or this is another error?

[cbuescher]

Also reproduces on master at 3c1544d with the same above reproduce line. I don't know about the frequency of these failures yet but will mute these tests again just in case.