[DOC] Add new documentation for distributed tracing#4964

Merged

vagimeli merged 72 commits intomainfrom

distributed-tracing

Oct 12, 2023

Contributor

vagimeli commented Sep 5, 2023 •

edited by hdhalter

Loading

Description

Add new documentation for distributed tracing

Issues Resolved

opensearch-project/OpenSearch#6750
Fixes #3951

Checklist

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.


          Create PR

3b46742

Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

vagimeli added In progress v2.10.0 labels

vagimeli self-assigned this

vagimeli requested review from AMoo-Miki, Naarcha-AWS, ananzh, cwillum, hdhalter, kolchfa-aws, natebower and seanneumann as code owners

September 5, 2023 18:53

This was referenced Sep 5, 2023

[RFC] Distributed Tracing opensearch-project/OpenSearch#6750

Open

[DOC] Distributed Tracing #3951

Closed

vagimeli added 5 commits

September 5, 2023 15:19


          First draft for technical review

31bc59e

Signed-off-by: Melissa Vagi <vagimeli@amazon.com>


          Fix copy labels

888319b

Signed-off-by: Melissa Vagi <vagimeli@amazon.com>


          Fix copy labels

083d6fc

Signed-off-by: Melissa Vagi <vagimeli@amazon.com>


          Fix copy labels

d07bac9

Signed-off-by: Melissa Vagi <vagimeli@amazon.com>


          Fix copy labels

a6f8969

Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

Contributor Author

vagimeli commented Sep 5, 2023 •

edited

Loading

@Gaganjuneja @reta The draft documentation is ready for technical review. Please make your suggestions/updates by end of week. Thank you :)

vagimeli added 2 commits

September 5, 2023 16:03


          Merge branch 'main' into distributed-tracing

82b79b3


          Fix copy labels

0c688e8

Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

hdhalter added Tech review and removed In progress labels

reta reviewed

View reviewed changes

_observing-your-data/trace/distributed-tracing.md Outdated Show resolved Hide resolved

reta reviewed

View reviewed changes

_observing-your-data/trace/distributed-tracing.md Outdated Show resolved Hide resolved

reta reviewed

View reviewed changes

_observing-your-data/trace/distributed-tracing.md Outdated Show resolved Hide resolved

reta reviewed

View reviewed changes

_observing-your-data/trace/distributed-tracing.md Outdated Show resolved Hide resolved

reta reviewed

View reviewed changes

_observing-your-data/trace/distributed-tracing.md Outdated

+. **Exporters:** Exporters are responsible for persisting the data. OpenTelemetry provides several out-of-the-box exporters, and OpenSearch currently supports the following:
+                  - `LoggingSpanExporter`: Exports spans to a log file, generating a separate file in the logs directory `_otel_traces.log`. Default is `telemetry.otel.tracer.span.exporter.class=io.opentelemetry.exporter.logging.LoggingSpanExporter`.
+                  - `OtlpGrpcSpanExporter`: Exports spans by using [gRPC](https://grpc.io/). To use this exporter, you need to install the `otel-collector` on the node and specify the endpoint using the setting `telemetry.otel.tracer.exporter.endpoint`. By default, it writes to the endpoint `http://localhost:4317/`. If you want to configure it for HTTPS, follow the guidance in the [OpenTelemetry Configuration](https://opentelemetry.io/docs/collector/configuration/) documentation. The setting is as follows:
+                      - `telemetry.otel.tracer.span.exporter.class=org.opensearch.telemetry.tracing.exporter.OtlpGrpcSpanExporterProvider`

Contributor

reta Sep 8, 2023

Also not there yet, the pull request is not ready

Contributor

Gaganjuneja Sep 8, 2023

Yes, Only support for LoggingSpanExporter is there as of 2.10.

Contributor Author

vagimeli Sep 11, 2023

We can update these parameters when they're available.

reta reviewed

View reviewed changes

_observing-your-data/trace/distributed-tracing.md Outdated Show resolved Hide resolved

vagimeli and others added 6 commits

October 11, 2023 10:47


          Update _observing-your-data/trace/distributed-tracing.md

a310dd3

Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>
Signed-off-by: Melissa Vagi <vagimeli@amazon.com>


          Update _observing-your-data/trace/distributed-tracing.md

9c7312c

Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>
Signed-off-by: Melissa Vagi <vagimeli@amazon.com>


          Update _observing-your-data/trace/distributed-tracing.md

1b6de17

Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>
Signed-off-by: Melissa Vagi <vagimeli@amazon.com>


          Update _observing-your-data/trace/distributed-tracing.md

2af4a53

Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>
Signed-off-by: Melissa Vagi <vagimeli@amazon.com>


          Update _observing-your-data/trace/distributed-tracing.md

b9aceb9

Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>
Signed-off-by: Melissa Vagi <vagimeli@amazon.com>


          Address doc review feedback

e00a470

Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

vagimeli added 5 - Final Editorial Review and removed Doc review labels

natebower reviewed

View reviewed changes

Contributor

natebower left a comment

@vagimeli Please see my comments and changes and let me know if you have any questions. Thanks!

_observing-your-data/trace/distributed-tracing.md Outdated Show resolved Hide resolved

_observing-your-data/trace/distributed-tracing.md Show resolved Hide resolved

_observing-your-data/trace/distributed-tracing.md Outdated Show resolved Hide resolved

_observing-your-data/trace/distributed-tracing.md Outdated Show resolved Hide resolved

_observing-your-data/trace/distributed-tracing.md Outdated


		## Get started

		The distributed tracing feature is experimental as of OpenSearch 2.10. To begin using the distributed tracing feature, you need to first enable it using `opensearch.experimental.feature.telemetry.enabled` feature flag, and subsequently activate the tracer, using the dynamic setting `telemetry.tracer.enabled`. It's important to exercise caution when enabling this feature, as it can consume system resources. Detailed information on enabling and configuring distributed tracing, including on-demand troubleshooting and request sampling, is described in the following sections.

Contributor

natebower Oct 12, 2023

Second sentence: "...and subsequently activate the tracer by using..."?

_observing-your-data/trace/distributed-tracing.md Outdated Show resolved Hide resolved

_observing-your-data/trace/distributed-tracing.md Outdated Show resolved Hide resolved

_observing-your-data/trace/distributed-tracing.md Outdated


		### Sampling

		Distributed tracing can generate numerous spans, consuming system resources unnecessarily. To reduce the number of traces, also called samples, you can configure different sampling thresholds. By default, sampling is configured to include only 1% of all HTTP requests. Sampling has the following types:

Contributor

natebower Oct 12, 2023

"Sampling includes" instead of "Sampling has"?

_observing-your-data/trace/distributed-tracing.md Outdated

+. **Head sampling:** Sampling decisions are made before initiating the root span of a request. OpenSearch supports two head sampling methods:
+                  - **Probabilistic:** A blanket limit on incoming requests, dynamically adjustable with the `telemetry.tracer.sampler.probability` setting. This setting ranges between 0 and 1. Default is 0.01, which indicates that 1% of incoming requests are sampled.
+                  - **On-Demand:** For debugging specific requests, you can send the `trace=true` attribute as part of the HTTP headers, causing those requests to be sampled regardless of the probabilistic sampling setting.
+. **Tail sampling:** To configure tail sampling, follow the [OpenTelemetry tail sampling documentation](https://opentelemetry.io/docs/concepts/sampling/#tail-sampling). Configuration depends on the type of collector you choose. Updates on ongoing work for OpenSearch are in the [RFC](https://github.com/opensearch-project/OpenSearch/issues/8918) on GitHub.

Contributor

natebower Oct 12, 2023

"follow the instructions in"? Can we delete the last sentence as unnecessary?

Contributor Author

vagimeli Oct 12, 2023

Revised. Yes, deleted.

_observing-your-data/trace/distributed-tracing.md Outdated Show resolved Hide resolved

vagimeli and others added 14 commits

October 12, 2023 13:12


          Update _observing-your-data/trace/distributed-tracing.md

Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Melissa Vagi <vagimeli@amazon.com>


          Update _observing-your-data/trace/distributed-tracing.md

138a8bc

Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Melissa Vagi <vagimeli@amazon.com>


          Update distributed-tracing.md

30512e2

Signed-off-by: Melissa Vagi <vagimeli@amazon.com>


          Update _observing-your-data/trace/distributed-tracing.md

66fe27e

Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Melissa Vagi <vagimeli@amazon.com>


          Update _observing-your-data/trace/distributed-tracing.md

ef33ea1

Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Melissa Vagi <vagimeli@amazon.com>


          Update _observing-your-data/trace/distributed-tracing.md

Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Melissa Vagi <vagimeli@amazon.com>


          Update _observing-your-data/trace/distributed-tracing.md

b12a52f

Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Melissa Vagi <vagimeli@amazon.com>


          Update _observing-your-data/trace/distributed-tracing.md

8ff95df

Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Melissa Vagi <vagimeli@amazon.com>


          Update _observing-your-data/trace/distributed-tracing.md

e1f3bd3

Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Melissa Vagi <vagimeli@amazon.com>


          Update _observing-your-data/trace/distributed-tracing.md

ebebef1

Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Melissa Vagi <vagimeli@amazon.com>


          Update _observing-your-data/trace/distributed-tracing.md

517552b

Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Melissa Vagi <vagimeli@amazon.com>


          Update _observing-your-data/trace/distributed-tracing.md

a5ecf52

Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Melissa Vagi <vagimeli@amazon.com>


          address editorial comments

c91e137

Signed-off-by: Melissa Vagi <vagimeli@amazon.com>


          Merge branch 'main' into distributed-tracing

c5172f2

vagimeli merged commit c4ff0aa into main

vagimeli deleted the distributed-tracing branch

October 12, 2023 20:12

hdhalter added release-notes and removed 5 - Final Editorial Review labels

vagimeli added a commit that referenced this pull request


          [DOC] Add new documentation for distributed tracing (#4964)

ddd311e

* Write new documentation about distributed tracing

Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

---------

Signed-off-by: Melissa Vagi <vagimeli@amazon.com>
Co-authored-by: Andriy Redko <drreta@gmail.com>
Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>
Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

harshavamsi pushed a commit to harshavamsi/documentation-website that referenced this pull request


          [DOC] Add new documentation for distributed tracing (opensearch-proje…

829f987

…ct#4964)

* Write new documentation about distributed tracing

Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

---------

Signed-off-by: Melissa Vagi <vagimeli@amazon.com>
Co-authored-by: Andriy Redko <drreta@gmail.com>
Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>
Co-authored-by: Nathan Bower <nbower@amazon.com>

vagimeli added a commit that referenced this pull request


          [DOC] Add new documentation for distributed tracing (#4964)

c4a3f30

* Write new documentation about distributed tracing

Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

---------

Signed-off-by: Melissa Vagi <vagimeli@amazon.com>
Co-authored-by: Andriy Redko <drreta@gmail.com>
Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>
Co-authored-by: Nathan Bower <nbower@amazon.com>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

kolchfa-aws kolchfa-aws approved these changes

cwillum Awaiting requested review from cwillum

hdhalter Awaiting requested review from hdhalter

Naarcha-AWS Awaiting requested review from Naarcha-AWS

ananzh Awaiting requested review from ananzh

seanneumann Awaiting requested review from seanneumann

AMoo-Miki Awaiting requested review from AMoo-Miki

+3 more reviewers

natebower natebower left review comments

Gaganjuneja Gaganjuneja requested changes

reta reta approved these changes

Labels

release-notes v2.11.0