[BUG]TraceableTcpTransportChannel class cast exception,

[Gaganjuneja]

What is the bug?
When the experimental feature flag (opensearch.experimental.feature.telemetry.enabled) is set to true and the security plugin installed. There is a bootstrap error.

Caused by: org.opensearch.core.common.io.stream.NotSerializableExceptionWrapper: class_cast_exception: class org.opensearch.telemetry.tracing.channels.TraceableTcpTransportChannel cannot be cast to class org.opensearch.transport.TcpTransportChannel (org.opensearch.telemetry.tracing.channels.TraceableTcpTransportChannel and org.opensearch.transport.TcpTransportChannel are in unnamed module of loader 'app')
at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceived(SecuritySSLRequestHandler.java:116) ~[?:?]
at org.opensearch.security.OpenSearchSecurityPlugin$6$1.messageReceived(OpenSearchSecurityPlugin.java:790) ~[?:?]
at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:106) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at org.opensearch.transport.InboundHandler.handleRequest(InboundHandler.java:262) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at org.opensearch.transport.InboundHandler.messageReceived(InboundHandler.java:140) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:123) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:770) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at org.opensearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:175) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:150) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:115) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:95) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) ~[?:?]

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

What is the expected behavior?
A clear and concise description of what you expected to happen.

What is your host/environment?

• OS: [e.g. iOS]
• Version [e.g. 22]
• Plugins

Do you have any screenshots?
If applicable, add screenshots to help explain your problem.

Do you have any additional context?
Add any other context about the problem.

[peternied]

@Gaganjuneja Thanks for filing - and its great to see the fix in PR.

However, this is not an RC blocker to me. The telemetry feature is experimental, and there are no test cases in the security plugin to cover this scenario - so this seems like it was manually discovered. This gives me little confidence that there won't be additional issues discovered over time. Lets add some automation to validate these scenarios and build our confidence - let me know how I can help.

[stephen-crawford]

[Triage] This issue has already been reviewed, going to add the triaged label.

[reta]

Closing this one, it should be fixed by #3917