Add a frozen engine implementation#34357
Merged
s1monw merged 21 commits intoelastic:masterfrom Nov 7, 2018
Merged
Conversation
This change adds a `frozen` engine that allows lazily open a directory reader on a read-only shard. The engine wraps general purpose searchers in a LazyDirectoryReader that also allows to release and reset the underlying index readers after any and before secondary search phases. Relates to elastic#34352 F
s1monw
added
>enhancement
v7.0.0
:Distributed/Engine
Collaborator
|
Pinging @elastic/es-distributed |
s1monw
added a commit
to s1monw/elasticsearch
that referenced
this pull request
Oct 8, 2018
This change adds a high level freeze API that allows to open an index frozen and vice versa. Indices must be closed in order to become frozen and an open but frozen index must be closed to be defrosted. This change also adds a `index.frozen` setting to mark frozen indices and integrates the frozen engine with the `SearchOperationListener` that resets and releases the directory reader after and before search phases. Relates to elastic#34352 Depends on elastic#34357
s1monw
added a commit
to s1monw/elasticsearch
that referenced
this pull request
Oct 8, 2018
`Engine.Searcher` is non-final today which makes it error prone in the case of wrapping the underlying reader or lucene `IndexSearcher` like we do in `IndexSearcherWrapper`. Yet, there is no subclass of it yet that would be dramtic to just drop on the floor. With the start of development of frozen indices this changed since in elastic#34357 functionality was added to a subclass which would be dropped if a `IndexSearcherWrapper` is installed on an index. This change locks down the `Engine.Searcher` to prevent such a functionality trap.
s1monw
added a commit
that referenced
this pull request
Oct 16, 2018
`Engine.Searcher` is non-final today which makes it error prone in the case of wrapping the underlying reader or lucene `IndexSearcher` like we do in `IndexSearcherWrapper`. Yet, there is no subclass of it yet that would be dramatic to just drop on the floor. With the start of development of frozen indices this changed since in #34357 functionality was added to a subclass which would be dropped if a `IndexSearcherWrapper` is installed on an index. This change locks down the `Engine.Searcher` to prevent such a functionality trap.
s1monw
added a commit
that referenced
this pull request
Oct 16, 2018
`Engine.Searcher` is non-final today which makes it error prone in the case of wrapping the underlying reader or lucene `IndexSearcher` like we do in `IndexSearcherWrapper`. Yet, there is no subclass of it yet that would be dramatic to just drop on the floor. With the start of development of frozen indices this changed since in #34357 functionality was added to a subclass which would be dropped if a `IndexSearcherWrapper` is installed on an index. This change locks down the `Engine.Searcher` to prevent such a functionality trap.
romseygeek
reviewed
Oct 17, 2018
Contributor
romseygeek
left a comment
romseygeek
left a comment
There was a problem hiding this comment.
I left a few comments. My only real question is where reset() and release() are going to be called from?
|
|
||
| @Override | ||
| protected DirectoryReader open(Directory directory) throws IOException { | ||
| // we fake an empty directly reader for the ReadOnlyEngine. this reader is only used |
x-pack/plugin/core/src/main/java/org/elasticsearch/index/engine/FrozenEngine.java
Outdated
Show resolved
Hide resolved
x-pack/plugin/core/src/main/java/org/elasticsearch/index/engine/FrozenEngine.java
Outdated
Show resolved
Hide resolved
x-pack/plugin/core/src/main/java/org/elasticsearch/index/engine/FrozenEngine.java
Outdated
Show resolved
Hide resolved
x-pack/plugin/core/src/test/java/org/elasticsearch/index/engine/FrozenEngineTests.java
Outdated
Show resolved
Hide resolved
x-pack/plugin/core/src/main/java/org/elasticsearch/index/engine/FrozenEngine.java
Show resolved
Hide resolved
Contributor
Author
good one, This will be exposed in a followup. I do have a branch for this already but I didn't want to make this change too big. The relevant code is here |
Contributor
|
TIL about SearchOperationListener. Nice. |
spinscale
reviewed
Oct 17, 2018
libs/core/src/main/java/org/elasticsearch/core/internal/io/IOUtils.java
Outdated
Show resolved
Hide resolved
kcm
pushed a commit
that referenced
this pull request
Oct 30, 2018
`Engine.Searcher` is non-final today which makes it error prone in the case of wrapping the underlying reader or lucene `IndexSearcher` like we do in `IndexSearcherWrapper`. Yet, there is no subclass of it yet that would be dramatic to just drop on the floor. With the start of development of frozen indices this changed since in #34357 functionality was added to a subclass which would be dropped if a `IndexSearcherWrapper` is installed on an index. This change locks down the `Engine.Searcher` to prevent such a functionality trap.
bleskes
reviewed
Nov 6, 2018
x-pack/plugin/core/src/main/java/org/elasticsearch/index/engine/FrozenEngine.java
Outdated
Show resolved
Hide resolved
x-pack/plugin/core/src/main/java/org/elasticsearch/index/engine/FrozenEngine.java
Show resolved
Hide resolved
x-pack/plugin/core/src/main/java/org/elasticsearch/index/engine/FrozenEngine.java
Outdated
Show resolved
Hide resolved
x-pack/plugin/core/src/main/java/org/elasticsearch/index/engine/FrozenEngine.java
Show resolved
Hide resolved
x-pack/plugin/core/src/main/java/org/elasticsearch/index/engine/FrozenEngine.java
Outdated
Show resolved
Hide resolved
x-pack/plugin/core/src/main/java/org/elasticsearch/index/engine/FrozenEngine.java
Show resolved
Hide resolved
x-pack/plugin/core/src/main/java/org/elasticsearch/index/engine/FrozenEngine.java
Outdated
Show resolved
Hide resolved
x-pack/plugin/core/src/test/java/org/elasticsearch/index/engine/FrozenEngineTests.java
Show resolved
Hide resolved
s1monw
added a commit
that referenced
this pull request
Nov 10, 2018
This change adds a high level freeze API that allows to mark an index as frozen and vice versa. Indices must be closed in order to become frozen and an open but frozen index must be closed to be defrosted. This change also adds a index.frozen setting to mark frozen indices and integrates the frozen engine with the SearchOperationListener that resets and releases the directory reader after and before search phases. Relates to #34352 Depends on #34357
s1monw
added a commit
to s1monw/elasticsearch
that referenced
this pull request
Nov 10, 2018
This change adds a special caching reader that caches all relevant values for a range query to rewrite correctly in a can_match phase without actually opening the underlying directory reader. This allows frozen indices to be filtered with can_match and in-turn searched with wildcards in a efficient way since it allows us to exclude shards that won't match based on their date-ranges without opening their directory readers. Relates to elastic#34352 Depends on elastic#34357
pgomulka
pushed a commit
to pgomulka/elasticsearch
that referenced
this pull request
Nov 13, 2018
This change adds a `frozen` engine that allows lazily open a directory reader on a read-only shard. The engine wraps general purpose searchers in a LazyDirectoryReader that also allows to release and reset the underlying index readers after any and before secondary search phases. Relates to elastic#34352
hendrikmuhs
pushed a commit
that referenced
this pull request
Nov 13, 2018
This change adds a special caching reader that caches all relevant values for a range query to rewrite correctly in a can_match phase without actually opening the underlying directory reader. This allows frozen indices to be filtered with can_match and in-turn searched with wildcards in a efficient way since it allows us to exclude shards that won't match based on their date-ranges without opening their directory readers. Relates to #34352 Depends on #34357
s1monw
added a commit
that referenced
this pull request
Nov 13, 2018
This change adds a special caching reader that caches all relevant values for a range query to rewrite correctly in a can_match phase without actually opening the underlying directory reader. This allows frozen indices to be filtered with can_match and in-turn searched with wildcards in a efficient way since it allows us to exclude shards that won't match based on their date-ranges without opening their directory readers. Relates to #34352 Depends on #34357
tlrx
added a commit
to tlrx/elasticsearch
that referenced
this pull request
Mar 1, 2019
Changes were made in elastic#34357 and elastic#36467
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This change adds a
frozenengine that allows lazily open a directory readeron a read-only shard. The engine wraps general purpose searchers in a LazyDirectoryReader
that also allows to release and reset the underlying index readers after any and before
secondary search phases.
Relates to #34352