Skip to content

[Backport 2.x] Fix broken backward compatibility from 2.7 for IndexSorted field indices#10075

Merged
reta merged 2 commits into2.xfrom
backport/backport-10045-to-2.x
Sep 18, 2023
Merged

[Backport 2.x] Fix broken backward compatibility from 2.7 for IndexSorted field indices#10075
reta merged 2 commits into2.xfrom
backport/backport-10045-to-2.x

Conversation

@opensearch-trigger-bot
Copy link
Copy Markdown
Contributor

@opensearch-trigger-bot opensearch-trigger-bot bot commented Sep 15, 2023

Backport d34b352 from #10045.

@github-actions @reta
Fix broken backward compatibility from 2.7 for IndexSorted field indi…
748be94 
…ces (#10045)

* Fix broken backward comparibility from 2.7 for IndexSorted field indices

Signed-off-by: gashutos <gashutos@amazon.com>

* Adding CHANGELOG

Signed-off-by: gashutos <gashutos@amazon.com>

* Update server/src/main/java/org/opensearch/index/IndexSettings.java

Co-authored-by: Andriy Redko <drreta@gmail.com>
Signed-off-by: Chaitanya Gohel <104654647+gashutos@users.noreply.github.com>

* Removing unwanted logs

Signed-off-by: gashutos <gashutos@amazon.com>

* Removing unwanted logs

Signed-off-by: gashutos <gashutos@amazon.com>

* Adding index sort as part of mixed cluster to test this scenario

Signed-off-by: gashutos <gashutos@amazon.com>

* Removing optimization disable logic

Signed-off-by: gashutos <gashutos@amazon.com>

* Correcting some comments & version check to before( V_2_7_0) instead onOrBefire(V_2_6_1) since

Signed-off-by: gashutos <gashutos@amazon.com>

* Resolving spotless check error

Signed-off-by: gashutos <gashutos@amazon.com>

* Fixing broken UT - last minute checkin without copile

Signed-off-by: gashutos <gashutos@amazon.com>

* Improving code coverage to make codcov happy

Signed-off-by: gashutos <gashutos@amazon.com>

* Correcting typos and adding more tests

Signed-off-by: gashutos <gashutos@amazon.com>

* Removing unwanted imports

Signed-off-by: gashutos <gashutos@amazon.com>

---------

Signed-off-by: gashutos <gashutos@amazon.com>
Signed-off-by: Chaitanya Gohel <104654647+gashutos@users.noreply.github.com>
Co-authored-by: Andriy Redko <drreta@gmail.com>
(cherry picked from commit d34b352)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@gashutos gashutos mentioned this pull request Sep 15, 2023
Merged
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Sep 15, 2023 •
edited
Loading

Compatibility status:

Checks if related components are compatible with change 53f4239

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/performance-analyzer-rca.git]

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Sep 15, 2023

Gradle Check (Jenkins) Run Completed with:

@gashutos
Copy link
Copy Markdown
Contributor

gashutos commented Sep 15, 2023

Unrelated failures, @reta @andrross can we re-trigger gradlew check ?

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Sep 15, 2023

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Sep 15, 2023

Gradle Check (Jenkins) Run Completed with:

@gashutos
Copy link
Copy Markdown
Contributor

gashutos commented Sep 15, 2023

This failure is related to my changes.
org.opensearch.backwards.IndexingIT.testUpdateSnapshotStatus
checking....

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Sep 15, 2023

Gradle Check (Jenkins) Run Completed with:

@reta
Copy link
Copy Markdown
Contributor

reta commented Sep 15, 2023

This failure is related to my changes. org.opensearch.backwards.IndexingIT.testUpdateSnapshotStatus checking....

@gashutos identified the issue with tests:

Caused by: org.opensearch.transport.RemoteTransportException: [v1.3.13-0][127.0.0.1:42995][internal:index/shard/recovery/prepare_translog]
»  Caused by: java.lang.IllegalArgumentException: cannot change previous indexSort=<sortednumeric: "sortfield"> missingValue=9223372036854775807 selector=MIN type=LONG (from segment=_0(8.10.1):c8:[indexSort=<sortednumeric: "sortfield"> missingValue=9223372036854775807 selector=MIN type=LONG]:[diagnostics={os=Linux, timestamp=1694806619341, java.version=11.0.20, java.vm.version=11.0.20+8, lucene.version=8.10.1, source=flush, os.arch=amd64, java.runtime.version=11.0.20+8, os.version=5.4.254-175.358.amzn2int.x86_64, java.vendor=Eclipse Adoptium}]:[attributes={Lucene87StoredFieldsFormat.mode=BEST_SPEED}] :id=23nyyfkvqif07m3kcpvp96m6m) to new indexSort=<sortednumeric: "sortfield"> missingValue=2147483647 selector=MIN type=INT

The fix is in progress

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Sep 15, 2023

Gradle Check (Jenkins) Run Completed with:

@gashutos gashutos mentioned this pull request Sep 15, 2023
Closed
@reta
Copy link
Copy Markdown
Contributor

reta commented Sep 16, 2023

The fix is in progress

Cleared out

@gashutos gashutos mentioned this pull request Sep 16, 2023
Merged
@gashutos @reta
Finxing broken UT (#10087)
20b306d 
Signed-off-by: Gohel <gashutos@88665a2fc937.ant.amazon.com>
Co-authored-by: Gohel <gashutos@88665a2fc937.ant.amazon.com>
(cherry picked from commit d7aa6dd)
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
@reta reta force-pushed the backport/backport-10045-to-2.x branch from 53f4239 to 20b306d Compare September 16, 2023 15:27
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Sep 16, 2023

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Sep 16, 2023

Gradle Check (Jenkins) Run Completed with:

@codecov
Copy link
Copy Markdown

codecov bot commented Sep 16, 2023

Codecov Report

Merging #10075 (20b306d) into 2.x (4c9042f) will decrease coverage by 0.23%.
Report is 6 commits behind head on 2.x.
The diff coverage is 68.42%.

@@             Coverage Diff              @@
##                2.x   #10075      +/-   ##
============================================
- Coverage     70.89%   70.67%   -0.23%     
+ Complexity    58291    58136     -155     
============================================
  Files          4810     4810              
  Lines        275140   275156      +16     
  Branches      40440    40442       +2     
============================================
- Hits         195074   194460     -614     
- Misses        63389    64069     +680     
+ Partials      16677    16627      -50
Files Changed Coverage Δ
.../java/org/opensearch/common/collect/Iterators.java 75.00% <ø> (ø)
...n/java/org/opensearch/compress/ZstdCompressor.java 52.94% <ø> (-17.65%) ⬇️
...opensearch/core/common/breaker/CircuitBreaker.java 75.00% <ø> (ø)
.../core/common/breaker/CircuitBreakingException.java 89.28% <ø> (ø)
...search/core/common/breaker/NoopCircuitBreaker.java 61.53% <0.00%> (ø)
...earch/core/common/logging/LoggerMessageFormat.java 20.14% <ø> (ø)
...search/core/common/transport/TransportAddress.java 79.48% <ø> (ø)
.../org/opensearch/core/common/unit/ByteSizeUnit.java 80.00% <ø> (ø)
...g/opensearch/core/common/util/CollectionUtils.java 77.06% <ø> (ø)
...src/main/java/org/opensearch/core/index/Index.java 100.00% <ø> (ø)
... and 11 more

... and 486 files with indirect coverage changes

@reta reta merged commit 0e0cf0c into 2.x Sep 18, 2023
@github-actions github-actions bot deleted the backport/backport-10045-to-2.x branch September 18, 2023 13:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@reta reta reta approved these changes

@andrross andrross andrross approved these changes

@anasalkouz anasalkouz Awaiting requested review from anasalkouz anasalkouz is a code owner

@Bukhtawar Bukhtawar Awaiting requested review from Bukhtawar Bukhtawar is a code owner

@CEHENKLE CEHENKLE Awaiting requested review from CEHENKLE CEHENKLE is a code owner

@dblock dblock Awaiting requested review from dblock

@gbbafna gbbafna Awaiting requested review from gbbafna gbbafna is a code owner

@setiah setiah Awaiting requested review from setiah

@kartg kartg Awaiting requested review from kartg

@kotwanikunal kotwanikunal Awaiting requested review from kotwanikunal kotwanikunal is a code owner

@mch2 mch2 Awaiting requested review from mch2 mch2 is a code owner

@nknize nknize Awaiting requested review from nknize

@owaiskazi19 owaiskazi19 Awaiting requested review from owaiskazi19 owaiskazi19 is a code owner

@peternied peternied Awaiting requested review from peternied

@Rishikesh1159 Rishikesh1159 Awaiting requested review from Rishikesh1159 Rishikesh1159 is a code owner

@ryanbogan ryanbogan Awaiting requested review from ryanbogan

@saratvemulapalli saratvemulapalli Awaiting requested review from saratvemulapalli saratvemulapalli is a code owner

@shwetathareja shwetathareja Awaiting requested review from shwetathareja shwetathareja is a code owner

@dreamer-89 dreamer-89 Awaiting requested review from dreamer-89

@tlfeng tlfeng Awaiting requested review from tlfeng

@VachaShah VachaShah Awaiting requested review from VachaShah

@dbwiddis dbwiddis Awaiting requested review from dbwiddis dbwiddis is a code owner

@sachinpkale sachinpkale Awaiting requested review from sachinpkale sachinpkale is a code owner

@sohami sohami Awaiting requested review from sohami sohami is a code owner

@msfroh msfroh Awaiting requested review from msfroh msfroh is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@gashutos @reta @andrross