[BUG] The thread context is not properly cleared and messes up the traces

[reta]

Describe the bug
We have the issue with cleaning up the thread context upon certain transport action invocations, the thread context keeps holding the spans from previous invocations, messing up the traces.

To Reproduce
Consider this simple PUT request to create an index:

  curl -X PUT -H "Content-Type: application/json"     http://localhost:9200/test51     -d '{
          "mappings": {
              "properties": {
                  "field": { "type": "date", "format": "epoch_second" }
              }
          },
          "settings": {
              "number_of_shards": 2,
              "number_of_replicas": 2
          }
      }'

It generates the following trace:

Now wait just a bit and observe the same trace is growing:

And growing:

The reason for that is that thread context was not cleaned up and the background tasks still picking the last span as the parent, attaching more and more spans to it.

Expected behavior
The thread context must be properly cleaned up.

Plugins
OpenTelementry

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• Any

Additional context

opensearch.experimental.feature.telemetry.enabled: true
telemetry.tracer.sampler.probability: 1.0
telemetry.otel.tracer.span.exporter.class: io.opentelemetry.exporter.otlp.trace.OtlpGrpcSpanExporter
telemetry.tracer.enabled: true 
telemetry.feature.tracer.enabled: true

CC @Gaganjuneja this is serious one

[Gaganjuneja]

Let me take a look.

[reta]

Let me take a look.

@Gaganjuneja please let me know if you need help

[Gaganjuneja]

@reta This issue seems to be happening because of below ThreadContext::stashContext which keeps the state when it's triggered from the create index cluster operation. It's happening at couple of other places as well.

OpenSearch/server/src/main/java/org/opensearch/cluster/service/ClusterApplierService.java

Line 380 in a09047a

try (ThreadContext.StoredContext ignore = threadContext.stashContext()) {

OpenSearch/server/src/main/java/org/opensearch/cluster/service/MasterService.java

Line 982 in a09047a

try (ThreadContext.StoredContext ignore = threadContext.stashContext()) {

OpenSearch/server/src/main/java/org/opensearch/index/seqno/RetentionLeaseBackgroundSyncAction.java

Line 123 in a09047a

try (ThreadContext.StoredContext ignore = threadContext.stashContext()) {

[reta]

Thanks @Gaganjuneja , so it seems like the thread context and tracing state are in conflict. One of the options to explore - the stash should move the trace context from current context to new context (but that could cause other issues), will be looking into that this week.

[reta]

@Gaganjuneja it took me a lot of time but I think I clearly understand what is happening. The culprit is #10291: the local transport hands off the request using thread pool and at one of the places (still hunting the exact one) it captures the span from the current thread and never cleans it up. It causes this issue to manifest.

The fix I did for now:

• do not instrument any local transport actions (will be covered by [Tracing Instrumentation] Add instrumentation at Local Transport Layer #10291)
• always populate current context from the HTTP or TCP transport headers
• change the current context only within span scope

See please #10873