[CI] TestClustersPluginIT.testMultiNode failure

[henningandersen]

This test failed on my master PR build. I have not been able to reproduce it locally. Unsure if it relates to #37889.

This seems to be the primary problem:

[2019-04-16T09:36:36,076][WARN ][o.e.c.NodeConnectionsService] [multiNode-3] failed to connect to {multiNode-1}{76UnrzNcSQCGO2byg8l0-g}{pWe_gZtjS_22EyAtKnzMDA}{127.0.0.1}{127.0.0.1:33513}{testattr=test, ml.machine_memory=63323983872, ml.max_open_jobs=20, xpack.installed=true} (tried [1] times)
  org.elasticsearch.transport.ConnectTransportException: [multiNode-1][127.0.0.1:33513] connect_exception
  	at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:956) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  	at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$3(ActionListener.java:161) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]

leading to:

`cluster{::multiNode}` failed to wait for cluster health yellow after 40 SECONDS

testMultiNode.zip

[elasticmachine]

Pinging @elastic/es-core-infra

[alpar-t]

I tried to reproduce this both locally and on an identical worker but it won't reproduce.
There's not much to go by from the logs., I don't see why the nodes wouldn't be able to talk to each-other. Will try some more to reproduce with the entire suite.

[henningandersen]

This test failed again on my PR build: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+pull-request-1/12570/testReport/org.elasticsearch.gradle.testclusters/TestClustersPluginIT/testMultiProject/

Looks like a similar but not identical error:

> `cluster{:bravo:myTestCluster}` failed to wait for cluster health yellow after 40 SECONDS

Will leave test unmuted for now and retry PR.

[henningandersen]

Once more, seems it fails frequently on my PRs:

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+pull-request-1/12727/testReport/

though not on all PR builds:

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+pull-request-1/12703/

[henningandersen]

Once again here: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+pull-request-1/12965/testReport/

[henningandersen]

Turns out the last failures were for testMultiProject. testMultiNode is still muted.

Muted testMultiProject in 40216b4