[TEST-FAILURE] org.elasticsearch.client.documentation.LicensingDocumentationIT fails on release-tests CI

[colings86]

Build URL: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+release-tests/855/console

I have tried to reproduce this locally without any luck, although I was not able to run with -Dbuild.snapshot=false locally so this bug might be specific to release builds.

Reproduce command (as lifted from CI failure, locally I removed the -Dbuild.snapshot=false):

REPRODUCE WITH: ./gradlew :client:rest-high-level:integTestRunner \
  -Dtests.seed=6B866F493C3B7898 \
  -Dtests.class=org.elasticsearch.client.documentation.LicensingDocumentationIT \
  -Dtests.method="testPutLicense" \
  -Dtests.security.manager=true \
  -Dbuild.snapshot=false \
  -Dtests.jvm.argline="-Dbuild.snapshot=false" \
  -Dtests.locale=el-GR \
  -Dtests.timezone=Africa/Khartoum

It looks like the test is trying to put an invalid license and expecting ES to return an exception saying the license is valid but the server is returning that the licence is invalid. Maybe the way the test is creating an valid licence might be wrong:

  1> [2018-08-02T10:45:24,332][INFO ][o.e.c.d.LicensingDocumentationIT] [testPutLicense] before test
  2> NOTE: leaving temporary files on disk at: /var/lib/jenkins/workspace/elastic+elasticsearch+master+release-tests/client/rest-high-level/build/testrun/integTestRunner/J0/temp/org.elasticsearch.client.documentation.LicensingDocumentationIT_6B866F493C3B7898-001
  2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=1403, maxMBSortInHeap=7.8940949346565645, sim=RandomSimilarity(queryNorm=false): {}, locale=el-GR, timezone=Africa/Khartoum
  2> NOTE: Linux 4.4.0-1061-aws amd64/Oracle Corporation 1.8.0_181 (64-bit)/cpus=16,threads=1,free=360450152,total=516947968
  1> [2018-08-02T10:45:24,333][INFO ][o.e.c.d.LicensingDocumentationIT] [testPutLicense] initializing REST clients against [http://[::1]:39337]
  2> NOTE: All tests run in this JVM: [SnapshotClientDocumentationIT, TasksClientDocumentationIT, WatcherDocumentationIT, TasksIT, PingAndInfoIT, IndicesClientDocumentationIT, MigrationDocumentationIT, SearchDocumentationIT, WatcherIT, CRUDDocumentationIT, ClusterClientDocumentationIT, IngestClientIT, IndicesClientIT, SnapshotIT, BulkProcessorIT, IngestClientDocumentationIT, RankEvalIT, StoredScriptsIT, StoredScriptsDocumentationIT, BulkProcessorRetryIT, ClusterClientIT, CrudIT, SearchIT, LicensingDocumentationIT]
  1> [2018-08-02T10:45:24,352][INFO ][o.e.c.d.LicensingDocumentationIT] [testPutLicense] after test
FAILURE 0.03s | LicensingDocumentationIT.testPutLicense <<< FAILURES!
   > Throwable #1: java.lang.AssertionError: expected:<invalid> but was:<valid>
   > 	at __randomizedtesting.SeedInfo.seed([6B866F493C3B7898:8EDA9690B7D4DA9F]:0)
   > 	at org.elasticsearch.client.documentation.LicensingDocumentationIT.testPutLicense(LicensingDocumentationIT.java:68)
   > 	at java.lang.Thread.run(Thread.java:748)

[elasticmachine]

Pinging @elastic/es-core-infra

[colings86]

It looks like the test inlines a licence in raw json which I assume will not have a correct signature for release builds

[colings86]

Test muted on master in 097c428

[colings86]

Also muted on 6.x in f45e954

[jasontedor]

We should blacklist this test for release builds, but not completely mute the test for all builds.

[colings86]

@jasontedor Thats a better idea, I'd be happy to make that change so its only muted for release tests, but could you explain what I need to change because I'm not sure how to add tests to the blacklist in this way?