INTERNAL_SERVER_ERROR when calling _stats

[cwurm]

Elasticsearch version: 5.4.0 (on Elastic Cloud)

Steps to reproduce:

1. Call GET it_ops_logs/_stats
2. Output:

{
  "_shards": {
    "total": 2,
    "successful": 0,
    "failed": 2,
    "failures": [
      {
        "shard": 0,
        "index": "it_ops_logs",
        "status": "INTERNAL_SERVER_ERROR",
        "reason": {
          "type": "failed_node_exception",
          "reason": "Failed node [ee8ER9CZQZaSSKkdcLJkzQ]",
          "caused_by": {
            "type": "illegal_state_exception",
            "reason": "Negative longs unsupported, use writeLong or writeZLong for negative numbers [-3]"
          }
        }
      },
      {
        "shard": 0,
        "index": "it_ops_logs",
        "status": "INTERNAL_SERVER_ERROR",
        "reason": {
          "type": "failed_node_exception",
          "reason": "Failed node [YAc58bLUTuiYAZy943nTAA]",
          "caused_by": {
            "type": "illegal_state_exception",
            "reason": "Negative longs unsupported, use writeLong or writeZLong for negative numbers [-3]"
          }
        }
      }
    ]
  },
  "_all": {
    "primaries": {},
    "total": {}
  },
  "indices": {}
}

Logs: Nothing unusual (no errors or such).

[nik9000]

If it is possible to reproduce It'd be helpful to have error_trace turned on.

[abeyad]

I can't reproduce this with the above steps, I'm sure that there were some node failures in a particular manner that allowed this error to manifest.

@cwurm Any other steps that you can add for the reproduction?

[cwurm]

@nik9000 It's a Cloud cluster, I don't think I can turn this on. :-(

@abeyad It only happens on this index as far as I can tell. I know we resized that Cloud cluster (doubled it in size) earlier today. Cloud console shows all nodes as up and running. No errors in the log.

I'm not sure what I can do. You can access the cluster if you want - ping me.

[jasontedor]

It's a Cloud cluster, I don't think I can turn this on. :-(

It's a request parameter, you can set it (?error_trace=true).

No errors in the log.

Are you sure? We warn log all requests that are responded to with a 500 (unless error_trace is set to true).

[cwurm]

@jasontedor Oh sorry, my bad. Running GET it_ops_logs/_stats?error_trace=true doesn't change the output though.

I'm searching furiously through the logs UI in Cloud, but can't find anything related. Unfortunately, it seems next to impossible to go through ES logs in Cloud sequentially (new log lines get added all the time and screw up the pagination). What would the exception look like? (I searched for exception, error, failed - anything I could think of).

[jasontedor]

I obtained the logs from this instance and I know what the issue is, we have a double decrement bug when handling certain queries that fail in the fetch phase. This double decrement leads to the number of outstanding queries on a shard falling negative and that leads to the serialization issue here.

[jasontedor]

I opened #24922.