Skip to content

Create a coordinating node level reader for tsdb#79197

Merged
elasticsearchmachine merged 3 commits intoelastic:masterfrom
nik9000:tsdb_reader
Oct 15, 2021
Merged

Create a coordinating node level reader for tsdb#79197
elasticsearchmachine merged 3 commits intoelastic:masterfrom
nik9000:tsdb_reader

Conversation

@nik9000
Copy link
Copy Markdown
Member

@nik9000 nik9000 commented Oct 14, 2021

This creates an interface that reads data in a time series compatible
way on the coordinating node. We believe that it can one day smooth out
querying time series data at a high level.

Right now there is a single implementation of this interface that
targets standard indices very inefficiently. It delegates down to our
standard _search APIs, specifically composite, top_hits, and
search_after. It is our hope that when we have fancier TSDB support we
can use it to speed the API.

The API itself looks like:


void latestInRange(metric, from, to, callback);
void latestInRanges(metric, from, to, step, callback);
void valuesInRange(metric, from, to, callback);

@nik9000 nik9000 added v8.0.0 :StorageEngine/TSDB You know, for Metrics labels Oct 14, 2021
@nik9000
Create a coordinating node level reader for tsdb
26389ef 
This creates an interface that reads data in a time series compatible
way on the coordinating node. We believe that it can one day smooth out
querying time series data at a high level.

Right now there is a single implementation of this interface that
targets standard indices very inefficiently. It delegates down to our
standard `_search` APIs, specifically `composite`, `top_hits`, and
`search_after`. It is our hope that when we have fancier TSDB support we
can use it to speed the API.

The API itself looks like:
```
// The latest value for all time series in the range
void latestInRange(metric, from, to, callback);
// The latest value for all time series in ranges starting from
// `from`, st
void latestInRanges(metric, from, to, step, callback);
void valuesInRange(metric, from, to, callback);
```
@nik9000
Copy link
Copy Markdown
Member Author

nik9000 commented Oct 14, 2021

run elasticsearch-ci/part-1

@nik9000 nik9000 marked this pull request as ready for review October 15, 2021 13:30
@elasticmachine elasticmachine added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Oct 15, 2021
@elasticmachine
Copy link
Copy Markdown
Collaborator

elasticmachine commented Oct 15, 2021

Pinging @elastic/es-analytics-geo (Team:Analytics)

@nik9000 nik9000 requested a review from imotov October 15, 2021 13:30
imotov
imotov approved these changes Oct 15, 2021
View reviewed changes
Copy link
Copy Markdown
Contributor

@imotov imotov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I think this is a good start, but since this is quite experimental I would like to confine to a more easily identifiable location. How about org.elasticsearch.timeseries.support?

server/src/internalClusterTest/java/org/elasticsearch/search/tsdb/TimeSeriesMetricsIT.java Outdated
* Side Public License, v 1.
*/

package org.elasticsearch.search.tsdb;
Copy link
Copy Markdown
Contributor

@imotov imotov Oct 15, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe org.elasticsearch.timeseries.support?

Copy link
Copy Markdown
Member Author

@nik9000 nik9000 Oct 15, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure!

@nik9000 nik9000 added the auto-merge-without-approval Automatically merge pull request when CI checks pass (NB doesn't wait for reviews!) label Oct 15, 2021
@nik9000 nik9000 mentioned this pull request Oct 15, 2021
Closed
@elasticsearchmachine elasticsearchmachine merged commit bd01654 into elastic:master Oct 15, 2021
weizijun added a commit to weizijun/elasticsearch that referenced this pull request Oct 18, 2021
@weizijun
Merge branch 'upstream/master' into reuse-date-stream-timestamp
c4a4b07 
* upstream/master: (109 commits)
  Migrate custom role providers to licensed feature (elastic#79127)
  Remove stale AwaitsFix in InternalEngineTests (elastic#79323)
  Fix errors in RefreshListenersTests (elastic#79324)
  Reeable BwC Tests after elastic#79318 (elastic#79320)
  Mute BwC Tests for elastic#79318 (elastic#79319)
  Reenable BwC Tests after elastic#79308 (elastic#79313)
  Disable BwC Tests for elastic#79308 (elastic#79310)
  Adjust BWC for node-level field cap requests (elastic#79301)
  Allow total memory to be overridden (elastic#78750)
  Fix SnapshotBasedIndexRecoveryIT#testRecoveryIsCancelledAfterDeletingTheIndex (elastic#79269)
  Disable BWC tests
  Mute GeoIpDownloaderCliIT.testStartWithNoDatabases (elastic#79299)
  Add alias support to fleet search API (elastic#79285)
  Create a coordinating node level reader for tsdb (elastic#79197)
  Route documents to the correct shards in tsdb (elastic#77731)
  Inject migrate action regardless of allocate action (elastic#79090)
  Migrate to data tiers should always ensure a TIER_PREFERENCE is set (elastic#79100)
  Skip building of BWC distributions when building release artifacts (elastic#79180)
  Default ENFORCE_DEFAULT_TIER_PREFERENCE to true (elastic#79275)
  Deprecation of transient cluster settings (elastic#78794)
  ...

# Conflicts:
#	server/src/main/java/org/elasticsearch/index/IndexMode.java
#	server/src/test/java/org/elasticsearch/index/TimeSeriesModeTests.java
weizijun added a commit to weizijun/elasticsearch that referenced this pull request Oct 18, 2021
@weizijun
Merge branch 'upstream/master' into tsdb-deal-timestreamp
e3b7256 
* upstream/master: (521 commits)
  Migrate custom role providers to licensed feature (elastic#79127)
  Remove stale AwaitsFix in InternalEngineTests (elastic#79323)
  Fix errors in RefreshListenersTests (elastic#79324)
  Reeable BwC Tests after elastic#79318 (elastic#79320)
  Mute BwC Tests for elastic#79318 (elastic#79319)
  Reenable BwC Tests after elastic#79308 (elastic#79313)
  Disable BwC Tests for elastic#79308 (elastic#79310)
  Adjust BWC for node-level field cap requests (elastic#79301)
  Allow total memory to be overridden (elastic#78750)
  Fix SnapshotBasedIndexRecoveryIT#testRecoveryIsCancelledAfterDeletingTheIndex (elastic#79269)
  Disable BWC tests
  Mute GeoIpDownloaderCliIT.testStartWithNoDatabases (elastic#79299)
  Add alias support to fleet search API (elastic#79285)
  Create a coordinating node level reader for tsdb (elastic#79197)
  Route documents to the correct shards in tsdb (elastic#77731)
  Inject migrate action regardless of allocate action (elastic#79090)
  Migrate to data tiers should always ensure a TIER_PREFERENCE is set (elastic#79100)
  Skip building of BWC distributions when building release artifacts (elastic#79180)
  Default ENFORCE_DEFAULT_TIER_PREFERENCE to true (elastic#79275)
  Deprecation of transient cluster settings (elastic#78794)
  ...

# Conflicts:
#	rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/tsdb/10_settings.yml
#	server/src/main/java/org/elasticsearch/common/settings/IndexScopedSettings.java
#	server/src/main/java/org/elasticsearch/common/settings/Setting.java
#	server/src/main/java/org/elasticsearch/index/IndexMode.java
#	server/src/test/java/org/elasticsearch/index/TimeSeriesModeTests.java
@wchaparro wchaparro assigned nik9000 and unassigned nik9000 Dec 16, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@imotov imotov imotov approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

auto-merge-without-approval Automatically merge pull request when CI checks pass (NB doesn't wait for reviews!) :StorageEngine/TSDB You know, for Metrics Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) v8.0.0-beta1

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@nik9000 @elasticmachine @imotov @jakelandis @elasticsearchmachine