Fix token Invalidation when retries exhausted

[albertzaharovits]

Fixes a glitch that misses to call the index invalidation listener, when retry count is exhausted and there are still tokens to be retried.

I recommend the split diff to better visualize the change

[elasticmachine]

Pinging @elastic/es-security

[jaymode]

LGTM

[albertzaharovits]

both https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+pull-request+default-distro/8166/consoleFull and https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+pull-request-bwc/1448/consoleFull are plagued by:

java.lang.IllegalStateException: Message not fully read (request) for requestId [398], action [cluster:monitor/nodes/stats[n]], available [0]; resetting

In a multitude of independent tests. I'm going to assume a transient network failure.

[albertzaharovits]

@elasticmachine run elasticsearch-ci/default-distro

[albertzaharovits]

@elasticmachine run elasticsearch-ci/bwc

[albertzaharovits]

The failures were utterly alien and I think they were due to the #39512 missing in the PR . I have merged master in.

[jkakavas]

LGTM.

[albertzaharovits]

@elasticmachine run elasticsearch-ci/bwc

[albertzaharovits]

For https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+pull-request-bwc/1539/console

17:25:48 BUILD SUCCESSFUL in 1h 31m 28s
17:25:48 1186 actionable tasks: 917 executed, 269 up-to-date
17:25:48 
17:25:48 Publishing build scan...
17:25:48 https://gradle.com/s/vdh2woq5musmc
17:25:48 
17:25:49 runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
17:25:49 runbld>>> DURATION: 6139064ms
17:25:49 runbld>>> STDOUT: 15735728 bytes
17:25:49 runbld>>> STDERR: 100027 bytes
17:25:49 runbld>>> WRAPPED PROCESS: SUCCESS (0)
17:25:49 runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/elastic+elasticsearch+pull-request-bwc
17:25:57 runbld>>> Storing build metadata: bwc_refspec_distribution:bwc:minor=aec2db78eafa3447e86528b65f3fb296fd29e137;bwc_refspec_distribution:bwc:staged=7bb0afe5bd539b01d409144e46b089fa519c2b3f
17:25:57 runbld>>> Adding test report.
17:25:57 runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/elastic+elasticsearch+pull-request-bwc
17:26:04 runbld>>> Found 219 test output files
17:26:04 Warning: default mime table not found: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.x86_64/jre/lib/content-types.properties
17:26:04 runbld>>> Sending debug log to infra-root+runbld-debug@e***.co
17:26:05 #error {
17:26:05  :cause /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.x86_64/jre/lib/ext/dnsns.jar
17:26:05  :via
17:26:05  [{:type java.lang.InternalError
17:26:05    :message java.io.FileNotFoundException: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.x86_64/jre/lib/ext/dnsns.jar
17:26:05    :at [sun.misc.URLClassPath$JarLoader getResource URLClassPath.java 1052]}
17:26:05   {:type java.io.FileNotFoundException
17:26:05    :message /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.x86_64/jre/lib/ext/dnsns.jar
17:26:05    :at [sun.misc.URLClassPath$JarLoader getJarFile URLClassPath.java 938]}]
17:26:05  :trace
17:26:05  [[sun.misc.URLClassPath$JarLoader getJarFile URLClassPath.java 938]
17:26:05   [sun.misc.URLClassPath$JarLoader access$800 URLClassPath.java 801]
17:26:05   [sun.misc.URLClassPath$JarLoader$1 run URLClassPath.java 886]
17:26:05   [sun.misc.URLClassPath$JarLoader$1 run URLClassPath.java 879]
17:26:05   [java.security.AccessController doPrivileged AccessController.java -2]
17:26:05   [sun.misc.URLClassPath$JarLoader ensureOpen URLClassPath.java 878]
17:26:05   [sun.misc.URLClassPath$JarLoader getResource URLClassPath.java 1050]
17:26:05   [sun.misc.URLClassPath getResource URLClassPath.java 249]
17:26:05   [java.net.URLClassLoader$1 run URLClassLoader.java 366]
17:26:05   [java.net.URLClassLoader$1 run URLClassLoader.java 363]
17:26:05   [java.security.AccessController doPrivileged AccessController.java -2]
17:26:05   [java.net.URLClassLoader findClass URLClassLoader.java 362]
17:26:05   [java.lang.ClassLoader loadClass ClassLoader.java 424]

I have raised https://github.com/elastic/infra/issues/9993