Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. [2018-08-01T15:25:12,764][INFO ][o.e.e.NodeEnvironment ] [node-2] using [1] data paths, mounts [[/ (/dev/disk1s1)]], net usable_space [215.1gb], net total_space [465.6gb], types [apfs] [2018-08-01T15:25:12,769][INFO ][o.e.e.NodeEnvironment ] [node-2] heap size [494.9mb], compressed ordinary object pointers [true] [2018-08-01T15:25:12,770][INFO ][o.e.n.Node ] [node-2] node name [node-2], node ID [CVngT_QBQB2XrJf2cLIfHA] [2018-08-01T15:25:12,771][INFO ][o.e.n.Node ] [node-2] version[6.5.0-SNAPSHOT], pid[3092], build[default/zip/4ea97bb/2018-08-01T22:22:49.979385Z], OS[Mac OS X/10.13.4/x86_64], JVM["Oracle Corporation"/Java HotSpot(TM) 64-Bit Server VM/10/10+46] [2018-08-01T15:25:12,771][INFO ][o.e.n.Node ] [node-2] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/var/folders/gx/83z_5d1d10b0xpp82xf7_tz00000gn/T/elasticsearch.EOOaNT3c, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -XX:UseAVX=2, -Xms512m, -Xmx512m, -ea, -esa, -Des.path.home=/Users/andy/code/es/elasticsearch/x-pack/qa/rolling-upgrade/with-system-key/build/cluster/v6.5.0-SNAPSHOT#oldClusterTestCluster node2/elasticsearch-6.5.0-SNAPSHOT, -Des.path.conf=/Users/andy/code/es/elasticsearch/x-pack/qa/rolling-upgrade/with-system-key/build/cluster/v6.5.0-SNAPSHOT#oldClusterTestCluster node2/elasticsearch-6.5.0-SNAPSHOT/config, -Des.distribution.flavor=default, -Des.distribution.type=zip] [2018-08-01T15:25:12,771][WARN ][o.e.n.Node ] [node-2] version [6.5.0-SNAPSHOT] is a pre-release version of Elasticsearch and is not suitable for production [2018-08-01T15:25:14,182][INFO ][o.e.p.PluginsService ] [node-2] loaded module [aggs-matrix-stats] [2018-08-01T15:25:14,182][INFO ][o.e.p.PluginsService ] [node-2] loaded module [analysis-common] [2018-08-01T15:25:14,182][INFO ][o.e.p.PluginsService ] [node-2] loaded module [ingest-common] [2018-08-01T15:25:14,182][INFO ][o.e.p.PluginsService ] [node-2] loaded module [lang-expression] [2018-08-01T15:25:14,183][INFO ][o.e.p.PluginsService ] [node-2] loaded module [lang-mustache] [2018-08-01T15:25:14,183][INFO ][o.e.p.PluginsService ] [node-2] loaded module [lang-painless] [2018-08-01T15:25:14,183][INFO ][o.e.p.PluginsService ] [node-2] loaded module [mapper-extras] [2018-08-01T15:25:14,183][INFO ][o.e.p.PluginsService ] [node-2] loaded module [parent-join] [2018-08-01T15:25:14,183][INFO ][o.e.p.PluginsService ] [node-2] loaded module [percolator] [2018-08-01T15:25:14,183][INFO ][o.e.p.PluginsService ] [node-2] loaded module [rank-eval] [2018-08-01T15:25:14,184][INFO ][o.e.p.PluginsService ] [node-2] loaded module [reindex] [2018-08-01T15:25:14,184][INFO ][o.e.p.PluginsService ] [node-2] loaded module [repository-url] [2018-08-01T15:25:14,184][INFO ][o.e.p.PluginsService ] [node-2] loaded module [transport-netty4] [2018-08-01T15:25:14,184][INFO ][o.e.p.PluginsService ] [node-2] loaded module [tribe] [2018-08-01T15:25:14,184][INFO ][o.e.p.PluginsService ] [node-2] loaded module [x-pack-core] [2018-08-01T15:25:14,184][INFO ][o.e.p.PluginsService ] [node-2] loaded module [x-pack-deprecation] [2018-08-01T15:25:14,185][INFO ][o.e.p.PluginsService ] [node-2] loaded module [x-pack-graph] [2018-08-01T15:25:14,185][INFO ][o.e.p.PluginsService ] [node-2] loaded module [x-pack-logstash] [2018-08-01T15:25:14,185][INFO ][o.e.p.PluginsService ] [node-2] loaded module [x-pack-ml] [2018-08-01T15:25:14,185][INFO ][o.e.p.PluginsService ] [node-2] loaded module [x-pack-monitoring] [2018-08-01T15:25:14,185][INFO ][o.e.p.PluginsService ] [node-2] loaded module [x-pack-rollup] [2018-08-01T15:25:14,185][INFO ][o.e.p.PluginsService ] [node-2] loaded module [x-pack-security] [2018-08-01T15:25:14,185][INFO ][o.e.p.PluginsService ] [node-2] loaded module [x-pack-sql] [2018-08-01T15:25:14,185][INFO ][o.e.p.PluginsService ] [node-2] loaded module [x-pack-upgrade] [2018-08-01T15:25:14,186][INFO ][o.e.p.PluginsService ] [node-2] loaded module [x-pack-watcher] [2018-08-01T15:25:14,186][INFO ][o.e.p.PluginsService ] [node-2] no plugins loaded [2018-08-01T15:25:17,677][INFO ][o.e.x.s.a.s.FileRolesStore] [node-2] parsed [0] roles from file [/Users/andy/code/es/elasticsearch/x-pack/qa/rolling-upgrade/with-system-key/build/cluster/v6.5.0-SNAPSHOT#oldClusterTestCluster node2/elasticsearch-6.5.0-SNAPSHOT/config/roles.yml] [2018-08-01T15:25:18,098][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [node-2] [controller/3110] [Main.cc@109] controller (64 bit): Version 6.5.0-SNAPSHOT (Build 33db7ba593fc91) Copyright (c) 2018 Elasticsearch BV [2018-08-01T15:25:18,405][DEBUG][o.e.a.ActionModule ] [node-2] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security [2018-08-01T15:25:18,603][INFO ][o.e.d.DiscoveryModule ] [node-2] using discovery type [zen] [2018-08-01T15:25:19,245][INFO ][o.e.n.Node ] [node-2] initialized [2018-08-01T15:25:19,245][INFO ][o.e.n.Node ] [node-2] starting ... [2018-08-01T15:25:19,405][INFO ][o.e.t.TransportService ] [node-2] publish_address {127.0.0.1:51678}, bound_addresses {[::1]:51677}, {127.0.0.1:51678} [2018-08-01T15:25:19,419][WARN ][o.e.b.BootstrapChecks ] [node-2] HTTPS is required in order to use the token service; please enable HTTPS using the [xpack.security.http.ssl.enabled] setting or disable the token service using the [xpack.security.authc.token.enabled] setting [2018-08-01T15:25:19,445][INFO ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [node-2] publish_address {127.0.0.1:51680}, bound_addresses {[::1]:51679}, {127.0.0.1:51680} [2018-08-01T15:25:19,448][INFO ][o.e.n.Node ] [node-2] started [2018-08-01T15:25:22,891][INFO ][o.e.c.s.ClusterApplierService] [node-2] detected_master {node-1}{1S6_8hxRR4WaytXQW-rvyg}{RnRBOBr3Ql-swWQAPQ6MzA}{127.0.0.1}{127.0.0.1:51642}{testattr=test, ml.machine_memory=17179869184, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}, added {{node-1}{1S6_8hxRR4WaytXQW-rvyg}{RnRBOBr3Ql-swWQAPQ6MzA}{127.0.0.1}{127.0.0.1:51642}{testattr=test, ml.machine_memory=17179869184, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true},{node-0}{EQ5jSYrnSEOL6m8Yu7AyaQ}{e41Pg_fHTPKZ4ulopvdeaw}{127.0.0.1}{127.0.0.1:51608}{testattr=test, ml.machine_memory=17179869184, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true},}, reason: apply cluster state (from master [master {node-1}{1S6_8hxRR4WaytXQW-rvyg}{RnRBOBr3Ql-swWQAPQ6MzA}{127.0.0.1}{127.0.0.1:51642}{testattr=test, ml.machine_memory=17179869184, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true} committed version [1]]) [2018-08-01T15:25:23,135][INFO ][o.e.x.s.a.TokenService ] [node-2] refresh keys [2018-08-01T15:25:23,371][INFO ][o.e.x.s.a.TokenService ] [node-2] refreshed keys [2018-08-01T15:25:23,866][INFO ][o.e.l.LicenseService ] [node-2] license [05d010af-71fd-469f-990b-642b2b6221c6] mode [trial] - valid [2018-08-01T15:25:34,025][INFO ][o.e.x.s.a.r.TransportPutRoleAction] [node-2] added role [native_role] [2018-08-01T15:25:40,092][INFO ][o.e.x.s.a.u.TransportPutUserAction] [node-2] added user [token_user] [2018-08-01T15:25:55,027][INFO ][o.e.d.z.ZenDiscovery ] [node-2] master_left [{node-1}{1S6_8hxRR4WaytXQW-rvyg}{RnRBOBr3Ql-swWQAPQ6MzA}{127.0.0.1}{127.0.0.1:51642}{testattr=test, ml.machine_memory=17179869184, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}], reason [no longer master] org.elasticsearch.transport.RemoteTransportException: [node-1][127.0.0.1:51642][internal:discovery/zen/fd/master_ping] Caused by: org.elasticsearch.cluster.NotMasterException: local node is not master [2018-08-01T15:25:55,033][WARN ][o.e.d.z.ZenDiscovery ] [node-2] master left (reason = no longer master), current nodes: nodes: {node-2}{CVngT_QBQB2XrJf2cLIfHA}{XTB8QzlsRuqwLaOqFDEdbQ}{127.0.0.1}{127.0.0.1:51678}{ml.machine_memory=17179869184, xpack.installed=true, testattr=test, ml.max_open_jobs=20, ml.enabled=true}, local {node-1}{1S6_8hxRR4WaytXQW-rvyg}{RnRBOBr3Ql-swWQAPQ6MzA}{127.0.0.1}{127.0.0.1:51642}{testattr=test, ml.machine_memory=17179869184, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}, master {node-0}{EQ5jSYrnSEOL6m8Yu7AyaQ}{e41Pg_fHTPKZ4ulopvdeaw}{127.0.0.1}{127.0.0.1:51608}{testattr=test, ml.machine_memory=17179869184, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true} [2018-08-01T15:25:55,042][WARN ][o.e.c.NodeConnectionsService] [node-2] failed to connect to node {node-0}{EQ5jSYrnSEOL6m8Yu7AyaQ}{e41Pg_fHTPKZ4ulopvdeaw}{127.0.0.1}{127.0.0.1:51608}{testattr=test, ml.machine_memory=17179869184, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true} (tried [1] times) org.elasticsearch.transport.ConnectTransportException: [node-0][127.0.0.1:51608] connect_exception at org.elasticsearch.transport.TcpChannel.awaitConnected(TcpChannel.java:165) ~[elasticsearch-6.5.0-SNAPSHOT.jar:6.5.0-SNAPSHOT] at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:643) ~[elasticsearch-6.5.0-SNAPSHOT.jar:6.5.0-SNAPSHOT] at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:542) ~[elasticsearch-6.5.0-SNAPSHOT.jar:6.5.0-SNAPSHOT] at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:329) ~[elasticsearch-6.5.0-SNAPSHOT.jar:6.5.0-SNAPSHOT] at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:316) ~[elasticsearch-6.5.0-SNAPSHOT.jar:6.5.0-SNAPSHOT] at org.elasticsearch.cluster.NodeConnectionsService.validateAndConnectIfNeeded(NodeConnectionsService.java:153) [elasticsearch-6.5.0-SNAPSHOT.jar:6.5.0-SNAPSHOT] at org.elasticsearch.cluster.NodeConnectionsService$1.doRun(NodeConnectionsService.java:106) [elasticsearch-6.5.0-SNAPSHOT.jar:6.5.0-SNAPSHOT] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723) [elasticsearch-6.5.0-SNAPSHOT.jar:6.5.0-SNAPSHOT] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.5.0-SNAPSHOT.jar:6.5.0-SNAPSHOT] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?] at java.lang.Thread.run(Thread.java:844) [?:?] Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: 127.0.0.1/127.0.0.1:51608 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?] at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?] at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:323) ~[?:?] at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:633) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?] ... 1 more Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?] at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?] at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:323) ~[?:?] at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:633) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?] ... 1 more [2018-08-01T15:25:58,038][WARN ][o.e.d.z.ZenDiscovery ] [node-2] not enough master nodes discovered during pinging (found [[Candidate{node={node-1}{1S6_8hxRR4WaytXQW-rvyg}{RnRBOBr3Ql-swWQAPQ6MzA}{127.0.0.1}{127.0.0.1:51642}{testattr=test, ml.machine_memory=17179869184, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}, clusterStateVersion=98}, Candidate{node={node-2}{CVngT_QBQB2XrJf2cLIfHA}{XTB8QzlsRuqwLaOqFDEdbQ}{127.0.0.1}{127.0.0.1:51678}{ml.machine_memory=17179869184, xpack.installed=true, testattr=test, ml.max_open_jobs=20, ml.enabled=true}, clusterStateVersion=98}]], but needed [3]), pinging again [2018-08-01T15:26:01,041][WARN ][o.e.d.z.ZenDiscovery ] [node-2] not enough master nodes discovered during pinging (found [[Candidate{node={node-1}{1S6_8hxRR4WaytXQW-rvyg}{RnRBOBr3Ql-swWQAPQ6MzA}{127.0.0.1}{127.0.0.1:51642}{testattr=test, ml.machine_memory=17179869184, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}, clusterStateVersion=98}, Candidate{node={node-2}{CVngT_QBQB2XrJf2cLIfHA}{XTB8QzlsRuqwLaOqFDEdbQ}{127.0.0.1}{127.0.0.1:51678}{ml.machine_memory=17179869184, xpack.installed=true, testattr=test, ml.max_open_jobs=20, ml.enabled=true}, clusterStateVersion=98}]], but needed [3]), pinging again [2018-08-01T15:26:04,043][WARN ][o.e.d.z.ZenDiscovery ] [node-2] not enough master nodes discovered during pinging (found [[Candidate{node={node-1}{1S6_8hxRR4WaytXQW-rvyg}{RnRBOBr3Ql-swWQAPQ6MzA}{127.0.0.1}{127.0.0.1:51642}{testattr=test, ml.machine_memory=17179869184, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}, clusterStateVersion=98}, Candidate{node={node-2}{CVngT_QBQB2XrJf2cLIfHA}{XTB8QzlsRuqwLaOqFDEdbQ}{127.0.0.1}{127.0.0.1:51678}{ml.machine_memory=17179869184, xpack.installed=true, testattr=test, ml.max_open_jobs=20, ml.enabled=true}, clusterStateVersion=98}]], but needed [3]), pinging again [2018-08-01T15:26:07,044][WARN ][o.e.d.z.ZenDiscovery ] [node-2] not enough master nodes discovered during pinging (found [[Candidate{node={node-1}{1S6_8hxRR4WaytXQW-rvyg}{RnRBOBr3Ql-swWQAPQ6MzA}{127.0.0.1}{127.0.0.1:51642}{testattr=test, ml.machine_memory=17179869184, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}, clusterStateVersion=98}, Candidate{node={node-2}{CVngT_QBQB2XrJf2cLIfHA}{XTB8QzlsRuqwLaOqFDEdbQ}{127.0.0.1}{127.0.0.1:51678}{ml.machine_memory=17179869184, xpack.installed=true, testattr=test, ml.max_open_jobs=20, ml.enabled=true}, clusterStateVersion=98}]], but needed [3]), pinging again [2018-08-01T15:26:10,048][WARN ][o.e.d.z.ZenDiscovery ] [node-2] not enough master nodes discovered during pinging (found [[Candidate{node={node-1}{1S6_8hxRR4WaytXQW-rvyg}{RnRBOBr3Ql-swWQAPQ6MzA}{127.0.0.1}{127.0.0.1:51642}{testattr=test, ml.machine_memory=17179869184, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}, clusterStateVersion=98}, Candidate{node={node-2}{CVngT_QBQB2XrJf2cLIfHA}{XTB8QzlsRuqwLaOqFDEdbQ}{127.0.0.1}{127.0.0.1:51678}{ml.machine_memory=17179869184, xpack.installed=true, testattr=test, ml.max_open_jobs=20, ml.enabled=true}, clusterStateVersion=98}]], but needed [3]), pinging again