Fix bug when formatting epoch dates#73955
Merged
not-napoleon merged 11 commits intoelastic:masterfrom Jun 15, 2021
Merged
Conversation
not-napoleon
added
>bug
v8.0.0
Team:Analytics
Collaborator
|
Pinging @elastic/es-analytics-geo (Team:Analytics) |
not-napoleon
commented
Jun 9, 2021
server/src/main/java/org/elasticsearch/search/DocValueFormat.java
Outdated
Show resolved
Hide resolved
not-napoleon
changed the title
imotov
approved these changes
Jun 14, 2021
| this.timeZone = Objects.requireNonNull(timeZone); | ||
| this.parser = formatter.toDateMathParser(); | ||
| this.formatter = formatter.withZone(timeZone); | ||
| this.parser = this.formatter.toDateMathParser(); |
Member
Author
There was a problem hiding this comment.
Credit to Przemyslaw who spotted it. I was just going to force it to UTC.
pgomulka
approved these changes
Jun 15, 2021
server/src/main/java/org/elasticsearch/search/DocValueFormat.java
Outdated
Show resolved
Hide resolved
Contributor
|
@elasticmachine update branch |
Member
Author
|
@elasticmachine update branch |
This was referenced Jun 15, 2021
not-napoleon
added a commit
to not-napoleon/elasticsearch
that referenced
this pull request
Jun 15, 2021
Date based aggregations accept a timezone, which gets applied to both the bucketing logic and the formatter. This is usually what you want, but in the case of date formats where a timezone doesn't make any sense, it can create problems. In particular, our formatting logic and our parsing logic were doing different things for epoch_second and epoch_millis formats with time zones. This led to a problem on composite where we'd return an after key for the last bucket that would parse to a time before the last bucket, so instead of correctly returning an empty response to indicate the end of the aggregation, we'd keep returning the same last page of data.
not-napoleon
added a commit
that referenced
this pull request
Jun 15, 2021
Date based aggregations accept a timezone, which gets applied to both the bucketing logic and the formatter. This is usually what you want, but in the case of date formats where a timezone doesn't make any sense, it can create problems. In particular, our formatting logic and our parsing logic were doing different things for epoch_second and epoch_millis formats with time zones. This led to a problem on composite where we'd return an after key for the last bucket that would parse to a time before the last bucket, so instead of correctly returning an empty response to indicate the end of the aggregation, we'd keep returning the same last page of data.
not-napoleon
added a commit
that referenced
this pull request
Jun 15, 2021
* Fix bug when formatting epoch dates (#73955) Date based aggregations accept a timezone, which gets applied to both the bucketing logic and the formatter. This is usually what you want, but in the case of date formats where a timezone doesn't make any sense, it can create problems. In particular, our formatting logic and our parsing logic were doing different things for epoch_second and epoch_millis formats with time zones. This led to a problem on composite where we'd return an after key for the last bucket that would parse to a time before the last bucket, so instead of correctly returning an empty response to indicate the end of the aggregation, we'd keep returning the same last page of data. * fix merge mistake
This was referenced Aug 31, 2021
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves #68963
Date based aggregations accept a timezone, which gets applied to both the bucketing logic and the formatter. This is usually what you want, but in the case of date formats where a timezone doesn't make any sense, it can create problems. In particular, our formatting logic and our parsing logic were doing different things for
epoch_secondandepoch_millisformats with time zones. This led to a problem on composite where we'd return an after key for the last bucket that would parse to a time before the last bucket, so instead of correctly returning an empty response to indicate the end of the aggregation, we'd keep returning the same last page of data.The proposed fix here is to always coerce the timezone to UTC for formatters only when using an epoch-based format. This aligns with the general expectation that epoch time is always UTC based.Turns out we just need to make the formatter timezone aware.