Skip to content

[DOCS] Document dimension mapping parameter #75414

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jrodewig merged 1 commit into from
Jul 19, 2021
Merged

[DOCS] Document dimension mapping parameter #75414

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions docs/reference/index-modules.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -329,6 +329,13 @@ Defaults to `*`, which matches all fields eligible for
the default pipeline (if it exists). The special pipeline name `_none`
indicates no ingest pipeline will run.

[[index-mapping-dimension-fields-limit]]
`index.mapping.dimension_fields.limit`::
For internal use by {kib} only. Maximum number of time series dimensions for the
Copy link
Copy Markdown
Contributor

@jpountz jpountz Jul 29, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand how it relates to Kibana?

Copy link
Copy Markdown
Contributor Author

@jrodewig jrodewig Jul 29, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We previously discussed marking the dimension parameter (and by extension this setting) for internal use by the Elastic Observability (specifically metrics) solution only. That solution is shipped as a part of Kibana.

If that's changed, let me know. I can update this in a separate PR. Thanks!

Copy link
Copy Markdown
Contributor

@jpountz jpountz Jul 30, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the context. My understanding is that this was indeed the initial vision, but that we moved from per-field metadata to dedicated mapping parameters with the vision that these parameters would be used by Elasticsearch in the long term as well. E.g. rollups could auto-configure based on what fields are marked as dimensions, and the timeseries_mode setting could be computed automatically based on dimensions. None of these have been merged yet but hopefully this dimension flag is going to be used by more than Kibana in the near future. @csoulios Please keep me honest if the above is incorrect.

Copy link
Copy Markdown
Contributor

@csoulios csoulios Jul 30, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jpountz your understanding is correct. dimension and metric mapping parameters will be used by ES (both for TSDB and rollups) for discovering dimensions and metrics. Eventually, they will be available to end-users too.

It's only that at this point we want to keep this feature for internal use only until we feel confident there will be no changes required. See comment: #74450 (review)

Copy link
Copy Markdown
Contributor Author

@jrodewig jrodewig Jul 30, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the discussion. That aligns with my understanding. The important point is that end users should not be using the dimension parameter at this time.

I'll push a change to broaden the copy to "For internal use by Elastic only."

index. Defaults to `16`.
+
You can mark a field as a dimension using the `dimension` mapping parameter.

[discrete]
=== Settings in other index modules

Expand Down
2 changes: 2 additions & 0 deletions docs/reference/mapping/types/ip.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,8 @@ The following parameters are accepted by `ip` fields:

[horizontal]

include::keyword.asciidoc[tag=dimension]

<<doc-values,`doc_values`>>::

Should the field be stored on disk in a column-stride fashion, so that it
Expand Down
16 changes: 16 additions & 0 deletions docs/reference/mapping/types/keyword.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,22 @@ The following parameters are accepted by `keyword` fields:

[horizontal]

// tag::dimension[]
`dimension`::
For internal use by {kib} only. Marks the field as a time series dimension.
Accepts `true` or `false` (default).
+
The <<index-mapping-dimension-fields-limit,`index.mapping.dimension_fields.limit`>>
index setting limits the number of dimensions in an index.
+
Dimension fields have the following constraints:
+
* The `doc_values` and `index` mapping parameters must be `true`.
* Field values cannot be an <<array,array or multi-value>>.
// end::dimension[]
* Field values cannot be larger than 1024 bytes.
* The field cannot use a <<normalizer,`normalizer`>>.

<<doc-values,`doc_values`>>::

Should the field be stored on disk in a column-stride fashion, so that it
Expand Down
5 changes: 5 additions & 0 deletions docs/reference/mapping/types/numeric.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,11 @@ The following parameters are accepted by numeric types:
Accepts `true` (default) and `false`. Not applicable for `unsigned_long`.
Note that this cannot be set if the `script` parameter is used.

include::keyword.asciidoc[tag=dimension]
+
Of the numeric field types, only `byte`, `short`, `integer`, and `long` fields
support this parameter.

<<doc-values,`doc_values`>>::

Should the field be stored on disk in a column-stride fashion, so that it
Expand Down