Docker images failing build

[ywelsch]

CI is tripping all over the newly added Docker images build.

> Task :distribution:docker:buildDockerImage FAILED
invalid reference format
Task ':distribution:docker:buildDockerImage' is not up-to-date because:

FAILURE: Build failed with an exception.
  Task has not declared any outputs despite executing actions.
Starting process 'command 'docker''. Working directory: /var/lib/jenkins/workspace/elastic+elasticsearch+6.x+multijob-unix-compatibility/os/opensuse/distribution/docker Command: docker build build/docker --pull --tag elasticsearch:6.6.0-SNAPSHOT --tag docker.elastic.co/elasticsearch/elasticsearch:6.6.0-SNAPSHOT --tag docker.elastic.co/elasticsearch/elasticsearch-full:6.6.0-SNAPSHOT
Successfully started process 'command 'docker''
Sending build context to Docker daemon  147.1MB


* What went wrong:
Step 1/30 : FROM centos:7 AS builder
Execution failed for task ':distribution:docker:buildDockerImage'.
> Process 'command 'docker'' finished with non-zero exit value 1
:distribution:docker:buildDockerImage (Thread[Execution worker for ':' Thread 11,5,main]) completed. Took 0.652 secs.

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+multijob-unix-compatibility/os=opensuse/108/console

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+release-tests/195/console

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+multijob-darwin-compatibility/108/console

and many many more.

Relates to #36246

[elasticmachine]

Pinging @elastic/es-core-infra

[jasontedor]

Half of these are addressed by an internal infra PR that I opened. I am looking into the other failures.

The salient error message is:

:distribution:docker:buildDockerImage FAILED
--
Sending build context to Docker daemon  147.1MB
 
Step 1/30 : FROM centos:7 AS builder
invalid reference format

[jasontedor]

This invalid reference format issue appears to only be occurring on the openSUSE CI workers, so something is off there.

[nknize]

Out of the blue I randomly started getting the following on #35320:

15:02:09 Caused by: java.lang.IllegalArgumentException: expected build.docker to be unset or one of "true" or "false" but was [$(]

Link to Jenkins build: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+pull-request-2/1814/console

[jasontedor]

We are actively working on this internally, sorry for the issues.

[jasontedor]

A multi-stage Docker build as we use requires Docker 17.05 or higher, and the openSUSE CI workers only have Docker 17.04.

[jasontedor]

I opened an internal issue for this.

[jasontedor]

I opened #36497 so that the issue is more apparent.

[pgomulka]

seems like thanks to #36497 we are now getting a bit more details about the problem - Possibly the docker version on opensuse and sles is incorrect.

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-unix-compatibility/os=opensuse/124/
and all other builds on that distro since 12th of December (day after the fix from #36497 was merged.

Successfully started process 'command '/usr/bin/docker''

FAILURE: Build failed with an exception.


* What went wrong:
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.0/userguide/command_line_interface.html#sec:command_line_warnings

Publishing build scan...
Failed to notify task execution graph listener.
> Docker version 18.06.1-ce, build e68fc7a215d7. Expression: matcher.matches()

Same for SLES
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-unix-compatibility/os=sles/124/console