[BUG] java.io.OptionalDataException error in OpenSearch 2.0 rc

[HyunSeaHoon]

Describe the bug
I am collecting data with OpenSearch with Logstash.
OpenSearch throws java.io.OptionalDataException error during service operation.

This did not happen in the OpenSearch 1.3 operating environment.

The occurrence of errors is irregular, and errors sometimes disappear after a certain period of time has passed.

Data collection is collected in different indexes depending on the data type, and some index collections work when an error occurs.
The data that is failing is not just a specific index, but a variety of things.

To Reproduce

Expected behavior

Plugins

Screenshots

Host/Environment (please complete the following information):

• OS: CentOS Linux release 7.9.2009
• Logstash: 7.16.2.
• Version: OpenSearch works in docker, and images use opensearchproject/opensearch:1.3.0 / opensearchproject/opensearch:2.0.0-rc1

Additional context
[OpenSearch Log]

2022-05-18T23:44:42.965700981Z [2022-05-18T23:44:42,965][INFO ][o.o.j.s.JobSweeper       ] [91616f4bd951] Running full sweep
2022-05-18T23:49:42.966011507Z [2022-05-18T23:49:42,965][INFO ][o.o.j.s.JobSweeper       ] [91616f4bd951] Running full sweep
2022-05-18T23:49:43.072676353Z [2022-05-18T23:49:43,072][INFO ][o.o.a.t.CronTransportAction] [91616f4bd951] Start running AD hourly cron.
2022-05-18T23:49:43.072820561Z [2022-05-18T23:49:43,072][INFO ][o.o.a.t.ADTaskManager    ] [91616f4bd951] Start to maintain running historical tasks
2022-05-18T23:49:43.073109887Z [2022-05-18T23:49:43,073][INFO ][o.o.a.c.HourlyCron       ] [91616f4bd951] Hourly maintenance succeeds
2022-05-18T23:50:30.021645243Z [2022-05-18T23:50:30,012][WARN ][r.suppressed             ] [91616f4bd951] path: /indexname-*/_search, params: {index=indexname-*}
2022-05-18T23:50:30.021674167Z org.opensearch.action.search.SearchPhaseExecutionException: all shards failed
2022-05-18T23:50:30.021678359Z 	at org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:642) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021681538Z 	at org.opensearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:360) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021684355Z 	at org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:677) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021687051Z 	at org.opensearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:457) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021689946Z 	at org.opensearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:291) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021692560Z 	at org.opensearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:72) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021695600Z 	at org.opensearch.transport.TransportService$6.handleException(TransportService.java:735) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021707976Z 	at org.opensearch.security.transport.SecurityInterceptor$RestoringTransportResponseHandler.handleException(SecurityInterceptor.java:318) [opensearch-security-2.0.0.0-rc1.jar:2.0.0.0-rc1]
2022-05-18T23:50:30.021711627Z 	at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1350) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021714318Z 	at org.opensearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1459) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021717064Z 	at org.opensearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1433) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021719769Z 	at org.opensearch.transport.TransportService.sendLocalRequest(TransportService.java:967) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021723529Z 	at org.opensearch.transport.TransportService$3.sendRequest(TransportService.java:147) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021726036Z 	at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:869) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021728354Z 	at org.opensearch.security.transport.SecurityInterceptor.sendRequestDecorate(SecurityInterceptor.java:212) [opensearch-security-2.0.0.0-rc1.jar:2.0.0.0-rc1]
2022-05-18T23:50:30.021731148Z 	at org.opensearch.security.OpenSearchSecurityPlugin$7$2.sendRequest(OpenSearchSecurityPlugin.java:665) [opensearch-security-2.0.0.0-rc1.jar:2.0.0.0-rc1]
2022-05-18T23:50:30.021733902Z 	at org.opensearch.transport.TransportService.sendRequest(TransportService.java:756) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021736648Z 	at org.opensearch.transport.TransportService.sendChildRequest(TransportService.java:831) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021739309Z 	at org.opensearch.action.search.SearchTransportService.sendCanMatch(SearchTransportService.java:149) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021742052Z 	at org.opensearch.action.search.CanMatchPreFilterSearchPhase.executePhaseOnShard(CanMatchPreFilterSearchPhase.java:128) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021744771Z 	at org.opensearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$3(AbstractSearchAsyncAction.java:278) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021747524Z 	at org.opensearch.action.search.AbstractSearchAsyncAction.performPhaseOnShard(AbstractSearchAsyncAction.java:312) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021750314Z 	at org.opensearch.action.search.AbstractSearchAsyncAction.run(AbstractSearchAsyncAction.java:249) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021753108Z 	at org.opensearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:415) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021755926Z 	at org.opensearch.action.search.AbstractSearchAsyncAction.start(AbstractSearchAsyncAction.java:215) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021758723Z 	at org.opensearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:994) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021764118Z 	at org.opensearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:757) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021767219Z 	at org.opensearch.action.search.TransportSearchAction.lambda$executeRequest$3(TransportSearchAction.java:398) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021770070Z 	at org.opensearch.action.ActionListener$1.onResponse(ActionListener.java:78) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021772833Z 	at org.opensearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:136) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021775554Z 	at org.opensearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:101) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021778319Z 	at org.opensearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:487) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021781263Z 	at org.opensearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:277) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021784064Z 	at org.opensearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:120) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021786868Z 	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:194) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021789576Z 	at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:118) [opensearch-index-management-2.0.0.0-rc1.jar:2.0.0.0-rc1]
2022-05-18T23:50:30.021792191Z 	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:192) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021794631Z 	at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:78) [opensearch-performance-analyzer-2.0.0.0-rc1.jar:2.0.0.0-rc1]
2022-05-18T23:50:30.021797236Z 	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:192) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021800054Z 	at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:325) [opensearch-security-2.0.0.0-rc1.jar:2.0.0.0-rc1]
2022-05-18T23:50:30.021802755Z 	at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:157) [opensearch-security-2.0.0.0-rc1.jar:2.0.0.0-rc1]
2022-05-18T23:50:30.021806267Z 	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:192) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021809178Z 	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:169) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021811919Z 	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:97) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021814537Z 	at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:108) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021819342Z 	at org.opensearch.rest.action.RestCancellableNodeClient.doExecute(RestCancellableNodeClient.java:104) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021822194Z 	at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:425) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021824698Z 	at org.opensearch.rest.action.search.RestSearchAction.lambda$prepareRequest$2(RestSearchAction.java:130) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021827375Z 	at org.opensearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:123) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021830341Z 	at org.opensearch.security.filter.SecurityRestFilter$1.handleRequest(SecurityRestFilter.java:128) [opensearch-security-2.0.0.0-rc1.jar:2.0.0.0-rc1]
2022-05-18T23:50:30.021832662Z 	at org.opensearch.rest.RestController.dispatchRequest(RestController.java:306) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021835200Z 	at org.opensearch.rest.RestController.tryAllHandlers(RestController.java:392) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021837529Z 	at org.opensearch.rest.RestController.dispatchRequest(RestController.java:235) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021840258Z 	at org.opensearch.security.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:63) [opensearch-security-2.0.0.0-rc1.jar:2.0.0.0-rc1]
2022-05-18T23:50:30.021843001Z 	at org.opensearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:361) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021845584Z 	at org.opensearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:440) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021848392Z 	at org.opensearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:351) [opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021851677Z 	at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:55) [transport-netty4-client-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021854379Z 	at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:41) [transport-netty4-client-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021857188Z 	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021859832Z 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021862547Z 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021865145Z 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021868012Z 	at org.opensearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:71) [transport-netty4-client-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.021873155Z 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021876009Z 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021878647Z 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021881288Z 	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021884253Z 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021887006Z 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021890227Z 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021893150Z 	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021895772Z 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021898247Z 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021900791Z 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021903611Z 	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021906357Z 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021909121Z 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021911671Z 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021914332Z 	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327) [netty-codec-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021917065Z 	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299) [netty-codec-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021919797Z 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021924273Z 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021927154Z 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021930364Z 	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021933214Z 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021935825Z 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021938494Z 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021941185Z 	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021943772Z 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021946507Z 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021949096Z 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021951894Z 	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1371) [netty-handler-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021954600Z 	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1234) [netty-handler-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021957283Z 	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1283) [netty-handler-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021959941Z 	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510) [netty-codec-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021962822Z 	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449) [netty-codec-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021965687Z 	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) [netty-codec-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021968924Z 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021971771Z 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021974631Z 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021979112Z 	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021982181Z 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021984750Z 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021987493Z 	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021990170Z 	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021992886Z 	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021995570Z 	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.021998068Z 	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.022005132Z 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.022007714Z 	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.022010101Z 	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.73.Final.jar:4.1.73.Final]
2022-05-18T23:50:30.022013231Z 	at java.lang.Thread.run(Thread.java:833) [?:?]
2022-05-18T23:50:30.022015580Z Caused by: org.opensearch.OpenSearchException: java.io.OptionalDataException
2022-05-18T23:50:30.022020352Z 	at org.opensearch.security.support.Base64Helper.deserializeObject(Base64Helper.java:185) ~[?:?]
2022-05-18T23:50:30.022023058Z 	at org.opensearch.security.transport.SecurityRequestHandler.messageReceivedDecorate(SecurityRequestHandler.java:155) ~[?:?]
2022-05-18T23:50:30.022025907Z 	at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceived(SecuritySSLRequestHandler.java:97) ~[?:?]
2022-05-18T23:50:30.022028556Z 	at org.opensearch.security.OpenSearchSecurityPlugin$7$1.messageReceived(OpenSearchSecurityPlugin.java:651) ~[?:?]
2022-05-18T23:50:30.022031217Z 	at org.opensearch.indexmanagement.rollup.interceptor.RollupInterceptor$interceptHandler$1.messageReceived(RollupInterceptor.kt:118) ~[?:?]
2022-05-18T23:50:30.022033939Z 	at org.opensearch.performanceanalyzer.transport.PerformanceAnalyzerTransportRequestHandler.messageReceived(PerformanceAnalyzerTransportRequestHandler.java:43) ~[?:?]
2022-05-18T23:50:30.022036620Z 	at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:98) ~[opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.022042345Z 	at org.opensearch.transport.TransportService.sendLocalRequest(TransportService.java:931) ~[opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.022045459Z 	... 101 more
2022-05-18T23:50:30.022047895Z Caused by: java.io.OptionalDataException
2022-05-18T23:50:30.022050506Z 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1767) ~[?:?]
2022-05-18T23:50:30.022053172Z 	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:514) ~[?:?]
2022-05-18T23:50:30.022055439Z 	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:472) ~[?:?]
2022-05-18T23:50:30.022057905Z 	at java.util.HashSet.readObject(HashSet.java:345) ~[?:?]
2022-05-18T23:50:30.022060408Z 	at jdk.internal.reflect.GeneratedMethodAccessor36.invoke(Unknown Source) ~[?:?]
2022-05-18T23:50:30.022062944Z 	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
2022-05-18T23:50:30.022065516Z 	at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
2022-05-18T23:50:30.022068083Z 	at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1231) ~[?:?]
2022-05-18T23:50:30.022070808Z 	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2434) ~[?:?]
2022-05-18T23:50:30.022073271Z 	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2268) ~[?:?]
2022-05-18T23:50:30.022075787Z 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1744) ~[?:?]
2022-05-18T23:50:30.022078237Z 	at java.io.ObjectInputStream$FieldValues.<init>(ObjectInputStream.java:2617) ~[?:?]
2022-05-18T23:50:30.022081688Z 	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2468) ~[?:?]
2022-05-18T23:50:30.022087904Z 	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2268) ~[?:?]
2022-05-18T23:50:30.022090864Z 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1744) ~[?:?]
2022-05-18T23:50:30.022093895Z 	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:514) ~[?:?]
2022-05-18T23:50:30.022096278Z 	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:472) ~[?:?]
2022-05-18T23:50:30.022098911Z 	at org.opensearch.security.support.Base64Helper.deserializeObject(Base64Helper.java:183) ~[?:?]
2022-05-18T23:50:30.022101744Z 	at org.opensearch.security.transport.SecurityRequestHandler.messageReceivedDecorate(SecurityRequestHandler.java:155) ~[?:?]
2022-05-18T23:50:30.022104165Z 	at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceived(SecuritySSLRequestHandler.java:97) ~[?:?]
2022-05-18T23:50:30.022106691Z 	at org.opensearch.security.OpenSearchSecurityPlugin$7$1.messageReceived(OpenSearchSecurityPlugin.java:651) ~[?:?]
2022-05-18T23:50:30.022109481Z 	at org.opensearch.indexmanagement.rollup.interceptor.RollupInterceptor$interceptHandler$1.messageReceived(RollupInterceptor.kt:118) ~[?:?]
2022-05-18T23:50:30.022112185Z 	at org.opensearch.performanceanalyzer.transport.PerformanceAnalyzerTransportRequestHandler.messageReceived(PerformanceAnalyzerTransportRequestHandler.java:43) ~[?:?]
2022-05-18T23:50:30.022117725Z 	at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:98) ~[opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.022119699Z 	at org.opensearch.transport.TransportService.sendLocalRequest(TransportService.java:931) ~[opensearch-2.0.0-rc1.jar:2.0.0-rc1]
2022-05-18T23:50:30.022121470Z 	... 101 more

[adnapibar]

@HyunSeaHoon Thanks for reporting the issue. Can you please provide steps to reproduce the issue?

[dblock]

@adnapibar ^ says The occurrence of errors is irregular, and errors sometimes disappear after a certain period of time has passed. I am afraid we have to work backwards from the error to understand how it could happen.

[adnapibar]

@dblock Yes agree! But I think we need more details that can help us to reproduce this error. Also, does this error happen with other API calls during the time? I see one discussion on es 7.9 reporting similar issue (not with logstash) https://discuss.elastic.co/t/temp-downtimes-optionaldataexception-500-on-all-apis-aws/257571 which may be completely unrelated.

[HyunSeaHoon]

Thank you for your interest.

Our service collects logs with logstash, stores them with ElasticSearch, and retrieves logs with a NodeJS-based web server.

We are changing from using ElasticSearch (with or without the Opendistro plugin) service to OpenSearch service.

logstash uses OSS version 7.16.2 and the output plugin uses logstash-output-opensearch (1.2.0).

The web server uses the javascript client elastic/elasticsearch.

As far as I know, the OptionalDataException error has been confirmed so far in the OpenSearch V2.0Rc environment.

Based on the symptoms occurring in the test environment, it seems to occur as follows:

1. OptionalDataException seems to be thrown first in Logstash during log collection.

2. It also happens through the OpenSearch search API. (Occurs when a request is made from the web server to the OpenSearch server)

3. Logs are collected in several types, and the log types in which OptionalDataException occurs are various. However, it seems that the object type is included in the data mapping among the collection histories in each log. For example, in the logindex generated by month, there was Object type mapping in logindex-April, and there was no Objecty type mapping in logindex-May, and it also occurred in the process of inserting data without object type. (The important thing is that OptionalDataException is thrown randomly.)

4. In the existing OpenSearch or ElasticSearch, the vm.max_map_count value was not set separately. (It seems that the default value of 65,530 was applied)
   After the occurrence of OptionalDataException was confirmed in the OpenSearch 2.0 Rc1 environment, when this value was changed to the recommended value (262,144), it seems that OptionalDataException occurs less.

5. After the vm.max_map_count value was changed, I did not experience the symptom that an OptionalDataException error occurred when searching in the web server.

6. When an OptionalDataException occurred before the vm.max_map_count value was changed, there were cases where the login function could not be used even in OpenSearchDashboard.
   (500 status code, Internal Server Error)

7. When an OptionalDataException occurred before the vm.max_map_count value was changed, the symptom seemed to continue until the OpenSearch service was restarted.

8. After changing the value of vm.max_map_count, I monitored the occurrence of OptionalDataException (only in Logstash, not when looking up the web server). It occurs about 5 times per minute for a certain period of time, then does not occur for a certain period of time and then occurs irregularly.

9. In general, when _cat/indices is searched, the response is in the following form.
   health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
   green open indexname JermuzKkTAa0dL7AxZ6l3A 1 0 449683 0 418.6mb 418.6mb
   When an OptionalDataException occurs in Logstash, when _cat/indices is searched with OpenSearch through curl, document and size information may not appear for all indexes as follows.
   health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
   green open indexname JermuzKkTAa0dL7AxZ6l3A

I wonder how many cases of this symptom are there, and I wonder if it is an opensearch bug, or a problem with the version of the library we use or a problem with data structure.

So far, it seems difficult to predict the cause of the occurrence.

Obviously, I have not experienced it with any version other than OpenSearch 2.0Rc1.

[louzadod]

It happened with 2.0.0 official release too. I migrated my cluster on 8h30 and the problem started 9h50 and took 1h do stop. It recovered from the error alone (no intervention) and I don't know if it is going to come back.
It is critical for us and now it is difficult to revert my upgrade.

[HyunSeaHoon]

The same symptoms appear in the full version of Opensearch 2.0.

The environment file used for testing is attached.

ostest.tar.gz

After checking the following, an error also occurred in that case.

• 5. After the vm.max_map_count value was changed, I did not experience the symptom that an OptionalDataException error occurred when searching in the web server.

After all the services are up (docker-compose up -d), an OptionalDataException is thrown at some point after a certain amount of time has passed. (Occurs within 1 hour, occurs after 10 hours, etc.)

The peculiar thing is that when monitoring with only the service running, OptionalDataException occurs in Logstash, but the log is not visible in OpenSearch.

At this time, when accessing the OpenSearchDashboard UI, an error 500 occurs, and an OptionalDataException occurs in the OpenSearchDashboard log and OpenSearch log.

[tlfeng]

Hi @HyunSeaHoon Thank you so much for providing such detailed steps to reproduce, appreciate your time.
And thanks @louzadod for reporting the issue as well.
I will discuss the issue with the other maintainers, and take it as a high priority.

[louzadod]

Hello @tlfeng . The architecture we adopted is as follows:
[Filebeat] --> [Logstash] --> [OS Client Nodes] --> [OS Data Nodes]

When the problem happens, the logs from Logstash to OpenSearch Client Nodes stops flowing. I assumed the problem was on Logstash and tried to restart it. After LS restart, logs were still not flowing.

Only when I restarted the OS Client Nodes, the logs started to flow again. It looks like a problem in the OpenSearch side.

[amalgamm]

Hi! Any update on this one?
I've got the same problem after upgrading from 1.2.4 to 2.0.0. I've got 2 data nodes and 1 master node. Several times during a day this stuff happens on random node. Cant read or write to this node until restart. Also no related warning or errors in +-5 minutes period.
Writing data directly to data nodes with fluent-bit (opensearch input), reading with grafana and opensearch-dashboards 2.0.0. No problems occurred before upgrade

[JustinasKO]

Same happens here. Multiple times a day Logstash fails to sent logs to OpenSearch cluster. Started happening after upgrade to V2.
Is downgrading back to v1.3.2 an option? maybe some other workaround?