[clickhouse] Create materialized view to store attribute metadata #7798
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Mahad Zaryab <mahadzaryab1@gmail.com>
Signed-off-by: Mahad Zaryab <mahadzaryab1@gmail.com>
Signed-off-by: Mahad Zaryab <mahadzaryab1@gmail.com>
mahadzaryab1
added
the
changelog:experimental
Signed-off-by: Mahad Zaryab <mahadzaryab1@gmail.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #7798 +/- ##
==========================================
+ Coverage 95.35% 95.47% +0.11%
==========================================
Files 310 307 -3
Lines 16075 15892 -183
==========================================
- Hits 15329 15173 -156
+ Misses 578 564 -14
+ Partials 168 155 -13
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
yurishkuro
reviewed
Dec 29, 2025
| CREATE MATERIALIZED VIEW IF NOT EXISTS attribute_metadata_mv TO attribute_metadata AS | ||
| SELECT | ||
| attribute_key, | ||
| type |
Member
There was a problem hiding this comment.
How about adding level: (resource | scope | span)?
I think the high level plan for the UI is to ask user to be explicit about which tags they are searching for by specifying a prefix like resource/{key}, so this information will be available in the reader and it can pinpoint the metadata more accurately.
We could also go all the way to capturing service name and span name, since strictly speaking an attribute X in different spans in different services do not have to mean the same thing of have the same type, and our query typically requires at least the service name. The only hesitation I have for that is if it will introduce too much overhead in CH for maintaining the materialized view.
Collaborator
Author
There was a problem hiding this comment.
Done! I just added the level for now.
yurishkuro
reviewed
Dec 29, 2025
| IF NOT EXISTS attribute_metadata ( | ||
| attribute_key String, | ||
| type String -- 'bool', 'double', 'int', 'string', 'bytes', 'map', 'slice' | ||
| ) ENGINE = MergeTree |
Member
There was a problem hiding this comment.
MergeTree is not going to deduplicate, correct? Is that what we want? Gemini suggests
CREATE TABLE IF NOT EXISTS attribute_metadata (
attribute_key String,
type String
) ENGINE = ReplacingMergeTree() -- Background deduplication
ORDER BY (attribute_key, type);
Collaborator
Author
There was a problem hiding this comment.
Thanks for the callout - we use ReplacingMergeTree for services and operations too! I actually learnt today that even with that engine, we need to use the FINAL keyword to perform the merge when querying (see https://clickhouse.com/docs/sql-reference/statements/select/from#final-modifier). We need to fix this for services and operations as well.
yurishkuro
reviewed
Dec 29, 2025
| WHERE | ||
| length(int_attributes.key) > 0 | ||
| UNION ALL | ||
| SELECT |
Member
There was a problem hiding this comment.
another Gemini suggestion to reduce the number of UNION steps:
SELECT
arrayJoin(arrayConcat(str_attributes.key, resource_str_attributes.key)) as attribute_key,
'str' as type
FROM spans
do we not handle scope attributes?
Collaborator
Author
There was a problem hiding this comment.
Done! And added scope attributes as well.
Signed-off-by: Mahad Zaryab <mahadzaryab1@gmail.com>
Signed-off-by: Mahad Zaryab <mahadzaryab1@gmail.com>
Signed-off-by: Mahad Zaryab <mahadzaryab1@gmail.com>
Metrics Comparison SummaryTotal changes across all snapshots: 73 Detailed changes per snapshotsummary_metrics_snapshot_elasticsearch📊 Metrics Diff SummaryTotal Changes: 73
🆕 Added Metrics
View diff sample+jaeger_storage_latency_seconds{le="+Inf",name="some_storage",operation="find_traces",otel_scope_name="jaeger-v2",otel_scope_schema_url="",otel_scope_version="",result="err",role="tracestore"}
+jaeger_storage_latency_seconds{le="0",name="some_storage",operation="find_traces",otel_scope_name="jaeger-v2",otel_scope_schema_url="",otel_scope_version="",result="err",role="tracestore"}
+jaeger_storage_latency_seconds{le="10",name="some_storage",operation="find_traces",otel_scope_name="jaeger-v2",otel_scope_schema_url="",otel_scope_version="",result="err",role="tracestore"}
+jaeger_storage_latency_seconds{le="100",name="some_storage",operation="find_traces",otel_scope_name="jaeger-v2",otel_scope_schema_url="",otel_scope_version="",result="err",role="tracestore"}
+jaeger_storage_latency_seconds{le="1000",name="some_storage",operation="find_traces",otel_scope_name="jaeger-v2",otel_scope_schema_url="",otel_scope_version="",result="err",role="tracestore"}
+jaeger_storage_latency_seconds{le="10000",name="some_storage",operation="find_traces",otel_scope_name="jaeger-v2",otel_scope_schema_url="",otel_scope_version="",result="err",role="tracestore"}
+jaeger_storage_latency_seconds{le="25",name="some_storage",operation="find_traces",otel_scope_name="jaeger-v2",otel_scope_schema_url="",otel_scope_version="",result="err",role="tracestore"}
...View diff sample+jaeger_storage_requests{name="some_storage",operation="find_traces",otel_scope_name="jaeger-v2",otel_scope_schema_url="",otel_scope_version="",result="err",role="tracestore"}View diff sample+rpc_server_duration_milliseconds{le="+Inf",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.37.0",otel_scope_version="0.64.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
+rpc_server_duration_milliseconds{le="0",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.37.0",otel_scope_version="0.64.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
+rpc_server_duration_milliseconds{le="10",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.37.0",otel_scope_version="0.64.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
+rpc_server_duration_milliseconds{le="100",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.37.0",otel_scope_version="0.64.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
+rpc_server_duration_milliseconds{le="1000",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.37.0",otel_scope_version="0.64.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
+rpc_server_duration_milliseconds{le="10000",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.37.0",otel_scope_version="0.64.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
+rpc_server_duration_milliseconds{le="25",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.37.0",otel_scope_version="0.64.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
...View diff sample+rpc_server_requests_per_rpc{le="+Inf",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.37.0",otel_scope_version="0.64.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
+rpc_server_requests_per_rpc{le="0",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.37.0",otel_scope_version="0.64.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
+rpc_server_requests_per_rpc{le="10",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.37.0",otel_scope_version="0.64.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
+rpc_server_requests_per_rpc{le="100",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.37.0",otel_scope_version="0.64.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
+rpc_server_requests_per_rpc{le="1000",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.37.0",otel_scope_version="0.64.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
+rpc_server_requests_per_rpc{le="10000",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.37.0",otel_scope_version="0.64.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
+rpc_server_requests_per_rpc{le="25",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.37.0",otel_scope_version="0.64.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
...View diff sample+rpc_server_responses_per_rpc{le="+Inf",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.37.0",otel_scope_version="0.64.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
+rpc_server_responses_per_rpc{le="0",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.37.0",otel_scope_version="0.64.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
+rpc_server_responses_per_rpc{le="10",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.37.0",otel_scope_version="0.64.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
+rpc_server_responses_per_rpc{le="100",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.37.0",otel_scope_version="0.64.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
+rpc_server_responses_per_rpc{le="1000",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.37.0",otel_scope_version="0.64.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
+rpc_server_responses_per_rpc{le="10000",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.37.0",otel_scope_version="0.64.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
+rpc_server_responses_per_rpc{le="25",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.37.0",otel_scope_version="0.64.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
... |
yurishkuro
approved these changes
Dec 30, 2025
ThatDeparted2061
pushed a commit
to ThatDeparted2061/jaeger
that referenced
this pull request
Dec 31, 2025
…egertracing#7798) Signed-off-by: ThatDeparted2061 <harshraocodesup@gmail.com>
ThatDeparted2061
pushed a commit
to ThatDeparted2061/jaeger
that referenced
this pull request
Jan 5, 2026
…egertracing#7798) Signed-off-by: ThatDeparted2061 <harshraocodesup@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which problem is this PR solving?
Description of the changes
FindTracesrequest as strings. In order to have ClickHouse query the correct Nested column for the attribute, this PR adds theattribute_metadatatable which contains a mapping from the attribute name to the attribute type. This table is populated through a materialized view.How was this change tested?
Spun up a ClickHouse server on my local machine and ran the scripts to set up all the tables. Then, ran a script to populate the spans table. Querying the
attribute_metadatatable correctly returns a mapping of all the attributes.Checklist
jaeger:make lint testjaeger-ui:npm run lintandnpm run test