[🐛 Bug]: Router can't start

[Doofus100500]

What happened?

Try to update to the 0.25.3 chart version, with 0.25.1 all worked fine

Command used to start Selenium Grid with Docker (or Kubernetes)

helm

Relevant log output

2023-11-30 07:22:49,417 INFO Included extra file "/etc/supervisor/conf.d/selenium-grid-router.conf" during parsing
2023-11-30 07:22:49,421 INFO RPC interface 'supervisor' initialized
2023-11-30 07:22:49,422 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2023-11-30 07:22:49,422 INFO supervisord started with pid 8
2023-11-30 07:22:50,425 INFO spawned: 'selenium-grid-router' with pid 9
Starting Selenium Grid Router...
2023-11-30 07:22:50,435 INFO success: selenium-grid-router entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
Using SE_SUB_PATH: /staging
Tracing is enabled
Classpath will be enriched with these external jars :  --ext :/external_jars/https/repo1.maven.org/maven2/io/opentelemetry/opentelemetry-exporter-otlp/1.32.0/opentelemetry-exporter-otlp-1.32.0.jar:/external_jars/https/repo1.maven.org/maven2/io/opentelemetry/opentelemetry-exporter-jaeger/1.32.0/opentelemetry-exporter-jaeger-1.32.0.jar:/external_jars/https/repo1.maven.org/maven2/io/grpc/grpc-netty/1.59.0/grpc-netty-1.59.0.jar:/external_jars/https/repo1.maven.org/maven2/io/opentelemetry/opentelemetry-sdk-trace/1.32.0/opentelemetry-sdk-trace-1.32.0.jar:/external_jars/https/repo1.maven.org/maven2/io/opentelemetry/opentelemetry-sdk-metrics/1.32.0/opentelemetry-sdk-metrics-1.32.0.jar:/external_jars/https/repo1.maven.org/maven2/io/opentelemetry/opentelemetry-sdk-logs/1.32.0/opentelemetry-sdk-logs-1.32.0.jar:/external_jars/https/repo1.maven.org/maven2/io/opentelemetry/opentelemetry-exporter-otlp-common/1.32.0/opentelemetry-exporter-otlp-common-1.32.0.jar:/external_jars/https/repo1.maven.org/maven2/io/opentelemetry/opentelemetry-exporter-sender-okhttp/1.32.0/opentelemetry-exporter-sender-okhttp-1.32.0.jar:/external_jars/https/repo1.maven.org/maven2/io/opentelemetry/opentelemetry-sdk-extension-autoconfigure-spi/1.32.0/opentelemetry-sdk-extension-autoconfigure-spi-1.32.0.jar:/external_jars/https/repo1.maven.org/maven2/io/opentelemetry/opentelemetry-sdk/1.32.0/opentelemetry-sdk-1.32.0.jar:/external_jars/https/repo1.maven.org/maven2/io/opentelemetry/opentelemetry-exporter-common/1.32.0/opentelemetry-exporter-common-1.32.0.jar:/external_jars/https/repo1.maven.org/maven2/com/fasterxml/jackson/jr/jackson-jr-objects/2.15.3/jackson-jr-objects-2.15.3.jar:/external_jars/https/repo1.maven.org/maven2/io/grpc/grpc-core/1.59.0/grpc-core-1.59.0.jar:/external_jars/https/repo1.maven.org/maven2/io/netty/netty-codec-http2/4.1.97.Final/netty-codec-http2-4.1.97.Final.jar:/external_jars/https/repo1.maven.org/maven2/io/netty/netty-handler-proxy/4.1.97.Final/netty-handler-proxy-4.1.97.Final.jar:/external_jars/https/repo1.maven.org/maven2/com/google/guava/guava/32.0.1-android/guava-32.0.1-android.jar:/external_jars/https/repo1.maven.org/maven2/com/google/errorprone/error_prone_annotations/2.20.0/error_prone_annotations-2.20.0.jar:/external_jars/https/repo1.maven.org/maven2/io/perfmark/perfmark-api/0.26.0/perfmark-api-0.26.0.jar:/external_jars/https/repo1.maven.org/maven2/io/netty/netty-transport-native-unix-common/4.1.97.Final/netty-transport-native-unix-common-4.1.97.Final.jar:/external_jars/https/repo1.maven.org/maven2/io/opentelemetry/opentelemetry-api/1.32.0/opentelemetry-api-1.32.0.jar:/external_jars/https/repo1.maven.org/maven2/io/opentelemetry/opentelemetry-sdk-common/1.32.0/opentelemetry-sdk-common-1.32.0.jar:/external_jars/https/repo1.maven.org/maven2/io/opentelemetry/opentelemetry-extension-incubator/1.32.0-alpha/opentelemetry-extension-incubator-1.32.0-alpha.jar:/external_jars/https/repo1.maven.org/maven2/io/opentelemetry/opentelemetry-api-events/1.32.0-alpha/opentelemetry-api-events-1.32.0-alpha.jar:/external_jars/https/repo1.maven.org/maven2/com/squareup/okhttp3/okhttp/4.12.0/okhttp-4.12.0.jar:/external_jars/https/repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.15.3/jackson-core-2.15.3.jar:/external_jars/https/repo1.maven.org/maven2/io/grpc/grpc-api/1.59.0/grpc-api-1.59.0.jar:/external_jars/https/repo1.maven.org/maven2/com/google/code/gson/gson/2.10.1/gson-2.10.1.jar:/external_jars/https/repo1.maven.org/maven2/com/google/android/annotations/4.1.1.4/annotations-4.1.1.4.jar:/external_jars/https/repo1.maven.org/maven2/org/codehaus/mojo/animal-sniffer-annotations/1.23/animal-sniffer-annotations-1.23.jar:/external_jars/https/repo1.maven.org/maven2/io/grpc/grpc-context/1.59.0/grpc-context-1.59.0.jar:/external_jars/https/repo1.maven.org/maven2/io/grpc/grpc-util/1.59.0/grpc-util-1.59.0.jar:/external_jars/https/repo1.maven.org/maven2/io/netty/netty-common/4.1.97.Final/netty-common-4.1.97.Final.jar:/external_jars/https/repo1.maven.org/maven2/io/netty/netty-buffer/4.1.97.Final/netty-buffer-4.1.97.Final.jar:/external_jars/https/repo1.maven.org/maven2/io/netty/netty-transport/4.1.97.Final/netty-transport-4.1.97.Final.jar:/external_jars/https/repo1.maven.org/maven2/io/netty/netty-codec/4.1.97.Final/netty-codec-4.1.97.Final.jar:/external_jars/https/repo1.maven.org/maven2/io/netty/netty-handler/4.1.97.Final/netty-handler-4.1.97.Final.jar:/external_jars/https/repo1.maven.org/maven2/io/netty/netty-codec-http/4.1.97.Final/netty-codec-http-4.1.97.Final.jar:/external_jars/https/repo1.maven.org/maven2/io/netty/netty-codec-socks/4.1.97.Final/netty-codec-socks-4.1.97.Final.jar:/external_jars/https/repo1.maven.org/maven2/com/google/guava/failureaccess/1.0.1/failureaccess-1.0.1.jar:/external_jars/https/repo1.maven.org/maven2/com/google/guava/listenablefuture/9999.0-empty-to-avoid-conflict-with-guava/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/external_jars/https/repo1.maven.org/maven2/com/google/code/findbugs/jsr305/3.0.2/jsr305-3.0.2.jar:/external_jars/https/repo1.maven.org/maven2/org/checkerframework/checker-qual/3.33.0/checker-qual-3.33.0.jar:/external_jars/https/repo1.maven.org/maven2/com/google/j2objc/j2objc-annotations/2.8/j2objc-annotations-2.8.jar:/external_jars/https/repo1.maven.org/maven2/io/opentelemetry/opentelemetry-context/1.32.0/opentelemetry-context-1.32.0.jar:/external_jars/https/repo1.maven.org/maven2/com/squareup/okio/okio/3.6.0/okio-3.6.0.jar:/external_jars/https/repo1.maven.org/maven2/org/jetbrains/kotlin/kotlin-stdlib-jdk8/1.9.10/kotlin-stdlib-jdk8-1.9.10.jar:/external_jars/https/repo1.maven.org/maven2/io/netty/netty-resolver/4.1.97.Final/netty-resolver-4.1.97.Final.jar:/external_jars/https/repo1.maven.org/maven2/com/squareup/okio/okio-jvm/3.6.0/okio-jvm-3.6.0.jar:/external_jars/https/repo1.maven.org/maven2/org/jetbrains/kotlin/kotlin-stdlib/1.9.10/kotlin-stdlib-1.9.10.jar:/external_jars/https/repo1.maven.org/maven2/org/jetbrains/kotlin/kotlin-stdlib-jdk7/1.9.10/kotlin-stdlib-jdk7-1.9.10.jar:/external_jars/https/repo1.maven.org/maven2/org/jetbrains/kotlin/kotlin-stdlib-common/1.9.10/kotlin-stdlib-common-1.9.10.jar:/external_jars/https/repo1.maven.org/maven2/org/jetbrains/annotations/13.0/annotations-13.0.jar
Nov 30, 2023 7:22:50 AM org.openqa.selenium.grid.Bootstrap createExtendedClassLoader
WARNING: Extension file or directory does not exist: 
07:22:51.215 INFO [LoggingOptions.configureLogEncoding] - Using the system default encoding
07:22:51.258 INFO [OpenTelemetryTracer.createTracer] - Using OpenTelemetry for tracing
java.lang.reflect.InvocationTargetException
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.openqa.selenium.grid.Bootstrap.runMain(Bootstrap.java:77)
    at org.openqa.selenium.grid.Bootstrap.main(Bootstrap.java:70)
Caused by: java.lang.NoSuchMethodError: 'void io.opentelemetry.sdk.internal.DaemonThreadFactory.<init>(java.lang.String, boolean)'
    at io.opentelemetry.exporter.sender.okhttp.internal.OkHttpUtil.newDispatcher(OkHttpUtil.java:31)
    at io.opentelemetry.exporter.sender.okhttp.internal.OkHttpGrpcSender.<init>(OkHttpGrpcSender.java:82)
    at io.opentelemetry.exporter.sender.okhttp.internal.OkHttpGrpcSenderProvider.createSender(OkHttpGrpcSenderProvider.java:43)
    at io.opentelemetry.exporter.internal.grpc.GrpcExporterBuilder.build(GrpcExporterBuilder.java:158)
    at io.opentelemetry.exporter.otlp.trace.OtlpGrpcSpanExporterBuilder.build(OtlpGrpcSpanExporterBuilder.java:203)
    at io.opentelemetry.exporter.otlp.internal.OtlpSpanExporterProvider.createExporter(OtlpSpanExporterProvider.java:72)
    at io.opentelemetry.sdk.autoconfigure.SpiUtil.lambda$loadConfigurable$0(SpiUtil.java:48)
    at io.opentelemetry.sdk.autoconfigure.NamedSpiManager.tryLoadImplementationForName(NamedSpiManager.java:46)
    at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1705)
    at io.opentelemetry.sdk.autoconfigure.NamedSpiManager.getByName(NamedSpiManager.java:36)
    at io.opentelemetry.sdk.autoconfigure.SpanExporterConfiguration.configureExporter(SpanExporterConfiguration.java:91)
    at io.opentelemetry.sdk.autoconfigure.SpanExporterConfiguration.configureSpanExporters(SpanExporterConfiguration.java:66)
    at io.opentelemetry.sdk.autoconfigure.TracerProviderConfiguration.configureTracerProvider(TracerProviderConfiguration.java:51)
    at io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdkBuilder.build(AutoConfiguredOpenTelemetrySdkBuilder.java:356)
    at org.openqa.selenium.remote.tracing.opentelemetry.OpenTelemetryTracer.createTracer(OpenTelemetryTracer.java:85)
    at org.openqa.selenium.remote.tracing.opentelemetry.OpenTelemetryTracer.getInstance(OpenTelemetryTracer.java:59)
    at org.openqa.selenium.grid.log.LoggingOptions.getTracer(LoggingOptions.java:93)
    at org.openqa.selenium.grid.router.httpd.RouterServer.createHandlers(RouterServer.java:116)
    at org.openqa.selenium.grid.TemplateGridServerCommand.asServer(TemplateGridServerCommand.java:47)
    at org.openqa.selenium.grid.router.httpd.RouterServer.execute(RouterServer.java:197)
    at org.openqa.selenium.grid.TemplateGridCommand.lambda$configure$4(TemplateGridCommand.java:122)
    at org.openqa.selenium.grid.Main.launch(Main.java:83)
    at org.openqa.selenium.grid.Main.go(Main.java:56)
    at org.openqa.selenium.grid.Main.main(Main.java:41)
    ... 6 more
2023-11-30 07:22:51,820 INFO exited: selenium-grid-router (exit status 1; not expected)

Operating System

k8s

Docker Selenium version (tag or chart version)

4.15.0-20231129

[github-actions]

@Doofus100500, thank you for creating this issue. We will troubleshoot it as soon as we can.

---

Info for maintainers

Triage this issue by using labels.

If information is missing, add a helpful comment and then I-issue-template label.

If the issue is a question, add the I-question label.

If the issue is valid but there is no time to troubleshoot it, consider adding the help wanted label.

If the issue requires changes or fixes from an external project (e.g., ChromeDriver, GeckoDriver, MSEdgeDriver, W3C), add the applicable G-* label, and it will provide the correct link and auto-close the issue.

After troubleshooting the issue, please add the R-awaiting answer label.

Thank you!

[VietND96]

I saw a long message from here

Tracing is enabled
Classpath will be enriched with these external jars :  --ext ...

Can you please share all extra values, YAML that you set when deploy chart?

[VietND96]

Probably there is extraEnvironmentVariables redundant after the recent updates in #2017
We have a couple of tests without tracing, YAML refer to https://github.com/SeleniumHQ/docker-selenium/tree/trunk/charts/selenium-grid/ci, it passed

[Doofus100500]

I think the problem is somewhere here:

extraEnvironmentVariables:
  - name: SE_SESSION_REQUEST_TIMEOUT
    value: "1800"
  - name: SE_ENABLE_TRACING
    value: "true"
  - name: SE_JAVA_OPTS
    value: "-Dotel.traces.exporter=otlp -Dotel.exporter.otlp.endpoint=https://opentm.host -Dotel.resource.attributes=service.name=selenium-grid-my-team -Dotel.exporter.otlp.traces.compression-jar=gzip"

[VietND96]

Ok, looks like after #2017, it is broken due to redundant : appended.

docker-selenium/Router/start-selenium-grid-router.sh

Lines 57 to 64 in 113bb0a

	EXTRA_LIBS=""
	if [ ! -z "$SE_ENABLE_TRACING" ]; then
	EXTERNAL_JARS=$(</external_jars/.classpath.txt)
	EXTRA_LIBS="--ext "
	EXTRA_LIBS=${EXTRA_LIBS}:${EXTERNAL_JARS}
	echo "Tracing is enabled"
	echo "Classpath will be enriched with these external jars : " ${EXTRA_LIBS}

I will fix this. In the meantime, workaround is temporary set SE_ENABLE_TRACING is false

[Doofus100500]

Thanks a lot!

[VietND96]

Looks like #2017 is not the root cause, it was due to #2015 upgrade io.opentelemetry from 1.31.0 to 1.32.0 (could relate to new change in 1.32.0 of class DaemonThreadFactory) that selenium-grid 4.15 still not handle.
Rollback io.opentelemetry 1.31.0 - it works now.
Also, added sanity test for tracing enabled to guarantee regression issue.