Query validation doesn't work consistently on inner queries

[darinakl]

Elasticsearch version: 5.6.9

When using validate API with inner queries returned results are not consistent:
The following queries, according to the API, are invalid:

1. {"query":{"query_string":{"query":"abc AND"}}}
2. {"query":{"bool":{"must":[{"query_string":{"query":"abc AND"}}]}}}

Although this one(when filter is added) is valid:
3. {"query":{"bool":{"filter":[{"range":{"@timestamp":{"lt":1549993646000,"gte":1549992746000}}}],"must":[{"query_string":{"query":"abc AND"}}]}}}

Steps to reproduce:

1. curl -X POST
   'localhost:9200/my_index/_validate/query?explain=true&ignore_unavailable=true'
   -H 'Content-Type: application/json'
   -d '{"query":{"query_string":{"query":"abc AND"}}}'

2. curl -X POST
   'localhost:9200/my_index/_validate/query?explain=true&ignore_unavailable=true'
   -H 'Content-Type: application/json'
   -d '{"query":{"bool":{"must":[{"query_string":{"query":"abc AND"}}]}}}'

3. curl -X POST
   'localhost:9200/my_index/_validate/query?explain=true&ignore_unavailable=true'
   -H 'Content-Type: application/json'
   -d '{"query":{"bool":{"filter":[{"range":{"@timestamp":{"lt":1549993646000,"gte":1549992746000}}}],"must":[{"query_string":{"query":"abc AND"}}]}}}'

[elasticmachine]

Pinging @elastic/es-search

[javanna]

This happens because there is a range query that gets parsed first, which causes the whole query to be rewritten to a match_none query in case there are no shards that hold any document within the specified range. In that case the invalid query_string does not even get parsed. You can see that from the validate query response as well:

{
	"_shards": {
		"total": 1,
		"successful": 1,
		"failed": 0
	},
	"valid": true,
	"explanations": [{
		"index": "my_index",
		"valid": true,
		"explanation": "MatchNoDocsQuery(\"User requested \"match_none\" query.\")"
	}]
}

On one hand the query does contain some invalid portion so it should not pass validation, though the second query is ignored with the current data hence the query runs fine and returns 0 results when fired using the _search endpoint. I do find it odd that query validation depends on indexed data though, I would prefer it to throw an error regardless in this case, although with some specific data the second query never runs. As soon as you index at least a document that is within the range, this query will fail, which is also reflected in the validate query output.

[javanna]

It seems like we started rewriting queries in the validate query API as part of #29483 , which addressed an issue around async actions in terms queries, for which it does make sense to call rewrite. Though it makes query validation dependent on data indexed at validation time which I think is a bit controversial. I will mark this team-discuss .

[javanna]

We discussed this with the team, we said that we would like to try and address this, as it's not consistent that a query may fail or not depending on what some of its queries rewrite to. I will have a look at whether it is possible to still parse every query as a lucene query and fail if that does not work.

[elasticsearchmachine]

Pinging @elastic/es-search (Team:Search)