Update Cluster Health checking logic

[DJRickyB]

As of elastic/elasticsearch#78968, Elasticsearch does not return a 408 for cluster health timeouts.

We now need to parse the HTTP 200 response for the timed_out field in case of timeouts while a wait_for_* parameter is invoked.

Note that currently the operation relies on a combination of default operation parameters (request-params and retry-until-success) in order to execute, but this may be a good opportunity to abstract the client-level behavior away from the user and provide some opaque wait-for parameters to the operation, then a top-level timeout. This would allow us to abort tests which will never pass the cluster state assertion, rather than the current behavior of hanging indefinitely.

[j-bennet]

It looks like elastic/elasticsearch#78968 was reverted: elastic/elasticsearch#80821.

As of 8.2.2, Elasticsearch still returns 408:

> curl -D - -k -u elastic:changeme "https://localhost:9200/_cluster/health?wait_for_nodes=2&timeout=1ms&pretty"
HTTP/1.1 408 Request Timeout
X-elastic-product: Elasticsearch
content-type: application/json
content-length: 465

{
  "cluster_name" : "elasticsearch",
  "status" : "green",
  "timed_out" : true,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 2,
  "active_shards" : 2,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

Elasticsearch version:

> curl -k -u elastic:changeme "https://localhost:9200"
{
  "name" : "eli.local",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "iYa9R7SbRmqbW1bwWGs1aA",
  "version" : {
    "number" : "8.2.2",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "9876968ef3c745186b94fdabd4483e01499224ef",
    "build_date" : "2022-05-25T15:47:06.259735307Z",
    "build_snapshot" : false,
    "lucene_version" : "9.1.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

Not sure what's up with that.

[arteam]

Unfortunately, the needed reverted due to backward compatibility concerns :( It landed quite late to 8.0 and there was a concern that we didn't provide a proper migration path for ES 7.x users.

[pquentin]

@j-bennet great find! And since it's not even deprecated now we can continue what we're currently doing and close this issue.