Date field mapping "epoch_millis" errors on date aggregations with timezones other than UTC

[cbuescher]

As demonstrated in elastic/kibana#19858, having a date field with "format": "epoch_millis" can cause
errors when running date histogram aggregations with a time zone other than UTC. I think this problem was introduced as a side effect of #30534. While throwing an error for parsing time values with "format": "epoch_millis" and timezones other than UTC looks like the intented behaviour (see #22621), it should be able to overwrite this by specifying format in the aggregation. This however doesn't seem to work currently on 6.x.

[elasticmachine]

Pinging @elastic/es-search-aggs

[cbuescher]

Specifying format like this in the aggregation:

DELETE test_epochmilli_1

PUT test_epochmilli_1
{
  "settings" : {
        "number_of_shards" : 1,
        "number_of_replicas" : 0
  },
  "mappings": {
    "doc": {
      "properties": {
        "@timestamp": {
          "type": "date",
          "format": "epoch_millis"
        },
        "name": {
          "type": "keyword"
        }
      }
    }
  }
}

PUT test_epochmilli_1/doc/1
{
  "@timestamp": 1477954800000,
  "name": "some_name"
}

GET test_epochmilli_1/_search
{
	"aggs": {
		"2": {
			"date_histogram": {
				"field": "@timestamp",
				"interval": "1M",
				"time_zone": "Europe/Berlin",
                                "format" : "yyyy-MM-dd",   <===== should overwrite "epoch_millis" from the mapping
				"min_doc_count": 1
			}
		}
	}
	
}

Throws

{
  "error": {
    "root_cause": [
      {
        "type": "parse_exception",
        "reason": "failed to parse date field [1477954800000] with format [epoch_millis]"
      }
    ],
    "type": "search_phase_execution_exception",
    "reason": "all shards failed",
    "phase": "query",
    "grouped": true,
    "failed_shards": [
      {
        "shard": 0,
        "index": "test_epochmilli_1",
        "node": "gOhPXLDVS4GRO2M1gHWXpA",
        "reason": {
          "type": "parse_exception",
          "reason": "failed to parse date field [1477954800000] with format [epoch_millis]",
          "caused_by": {
            "type": "illegal_argument_exception",
            "reason": "time_zone must be UTC for format [epoch_millis]"
          }
        }
      }
    ]
  },
  "status": 400
}

I would have expected it to use the specified format instead.

[cbuescher]

I might have a fix, still working on tests though before opening a PR.