Remove deprecated use of types in ES queries

[epixa]

We still have places in Kibana that are querying Elasticsearch using types, and this is logging deprecation warnings in ES. In addition to not wanting to have Kibana spamming deprecation logs for behaviors that users cannot control, the ES team is moving forward with their plans to reject types in future major versions, so we need to sort this out now.

The proposed first step is to add a configurable check (disabled by default) in our elasticsearch callWithRequest, callWithInternalUser, and search/console proxies that will throw an error when a response is received with deprecation warning headers that match the type error. We'll then set up a dedicated CI environment that runs with this configuration on. @jakelandis is going to provide the relevant warning messages. Then we'll track down each CI failure and remove the deprecated behavior one by one.

After that, a manual QA effort is likely necessary due to areas with low or missing test coverage, and they can use the same flags to help make identifying these requests easier.

@rayafratkina @jinmu03 @timroes

[epixa]

#30546 is an example PR that will fix one specific occurrence of a typed query to ES.

[jakelandis]

I counted 41 different messages that Elasticsearch can emit. All 41 messages start with the following: [types removal]

The harder ones that may require type changes also has the text include_type_name and are likely more difficult to fix.

I would suggest to start with something like (pseudo code):

if(header["Warning"].includes("[types removal]")){    
	if(header["Warning"].includes("include_type_name")){
		warning
	}else{
		error
	}
}

If we can get that to pass, it will be a great start and then can move on to the "include_type_name" ones.

Example of an deprecation HTTP header:

Warning: 299 Elasticsearch-7.0.0-SNAPSHOT-Unknown "[types removal] Specifying include_type_name in put index template requests is deprecated. The parameter will be removed in the next major version."

Example of the deprecation logs:

[2019-02-08T09:39:05,723][WARN ][o.e.d.r.a.d.RestGetAction] [node-0] [types removal] Specifying types in document get requests is deprecated, use the /{index}/_doc/{id} endpoint instead.
[2019-02-08T09:39:10,788][WARN ][o.e.d.a.b.BulkRequest    ] [node-0] [types removal] Specifying types in bulk requests is deprecated.
[2019-02-08T09:39:12,004][WARN ][o.e.d.r.a.s.RestSearchAction] [node-0] [types removal] Specifying types in search requests is deprecated.
[2019-02-08T09:42:32,969][WARN ][o.e.d.r.a.d.RestDeleteAction] [node-0] [types removal] Specifying types in document index requests is deprecated, use the typeless endpoints instead (/{index}/_doc/{id}, /{index}/_doc, or /{index}/_create/{id}).
[2019-02-08T09:42:37,053][WARN ][o.e.d.r.a.a.i.RestPutMappingAction] [node-0] [types removal] Using include_type_name in put mapping requests is deprecated. The parameter will be removed in the next major version.
[2019-02-08T09:43:00,468][WARN ][o.e.d.r.a.d.RestDeleteAction] [node-0] [types removal] Specifying types in document index requests is deprecated, use the /{index}/_doc/{id} endpoint instead.

Note the deprecation logs are throttled to only show up one time per ES instance, but the headers are sent every time.

[joshdover]

@jakelandis Thanks for the info.

Unfortunately, elasticsearch-js doesn't expose any response details besides the request body, but I've been able to hack this into its source locally for debugging purposes:

// node_modules/elasticsearch/src/lib/transport.js:280
if (headers && headers.warning) {
  var e = new Error(`Elasticsearch warning: ${headers.warning}, request: ${params.req.method} ${params.req.path}?${Object.keys(params.req.query).map(key => key + '=' + params.req.query[key]).join('&')}`);
  console.log(e.stack);
}

Sharing for anyone who might want to do something similar.

[joshdover]

This PR should fix the next most common issue: #30452

[rayafratkina]

@alexwizp can you please confirm this can be closed?
cc @tylersmalley

[epixa]

I think this is finished, so I'm closing this. Feel free to reopen if I'm wrong.