[FR/Bug] Improve logstash error messages when trying to index to a red index

[m-a-leclercq]

At the moment, logstash logs a misleading error message when trying to write to a red index (Tested on 9.3)

This is easily reproduced by creating an index with a routing requirement that cannot be satisfied :

PUT /red_index?timeout=5s
{
  "settings": {
    "index.routing.allocation.require._name": "non_existent_node"
  }
}

Here's what logstash is logging :

{"level":"ERROR","loggerName":"logstash.outputs.elasticsearch","timeMillis":1774265617972,"thread":"[twitter.conf]>worker0","logEvent":{"message":"Attempted to send a bulk request but Elasticsearch appears to be unreachable or down","message":"Elasticsearch Unreachable: [[https://test-logger-d4f449.es.us-central1.gcp.cloud.es.io:443/_bulk?filter_path=errors,items.*.error,items.*.status][Manticore::SocketTimeout](https://test-logger-d4f449.es.us-central1.gcp.cloud.es.io/_bulk?filter_path=errors,items.*.error,items.*.status][Manticore::SocketTimeout)] Read timed out","exception":"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError","will_retry_in_seconds":2}}
{"level":"ERROR","loggerName":"logstash.outputs.elasticsearch","timeMillis":1774265620028,"thread":"[twitter.conf]>worker0","logEvent":{"message":"Attempted to send a bulk request but there are no living connections in the pool (perhaps Elasticsearch is unreachable or down?)","message":"No Available connections","exception":"LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError","will_retry_in_seconds":4}}

The bulk response returns a 200 but errors is true and red_index documents return a 503. For example :

    {
      "index": {
        "_index": "red_index",
        "_id": "A",
        "status": 503,
        "error": {
          "type": "unavailable_shards_exception",
          "reason": """[red_index][1] primary shard is not active Timeout: [5s], request: [BulkShardRequest [[red_index][1]] containing [index {[red_index][A], source[{ "title": "Doc A — should FAIL (red index)", "status": "fail" }]}]]"""
        }
      }

This looks like a regression / non propagation of the actual error message that should be available to Logstash and should be logged rather than logging a dead ES instance error message.