[BUG] Force merge in 3.0 OS is slower than 2.19 OS

[bharath-techie]

Describe the bug

Through different runs, we have noticed that force merge operation takes more time and when we deep dived, stored fields especially takes more time in 3.0 OS compared to 2.19 OS.

Other than stored fields, postings format too shows regression ~20%.

Related component

No response

To Reproduce

Ingest data via big5 workload via OSB.
Force merge to 10 segments initially and force merge to 1 segment and enable the trace logs of "SM" and "MS" to enable Lucene infostream logs to get an idea on how much time each format takes.

curl --location --request PUT 'localhost:9200/_cluster/settings' \
--header 'Content-Type: application/json' \
--data-raw '{
    "transient": {
        "logger.org.opensearch.index.engine.Engine.SM": "TRACE",
        "logger.org.opensearch.index.engine.Engine.MS": "TRACE",
        "logger.org.opensearch.index.engine.Engine.IW": "TRACE"
    }
}'

Do the same in 2.19 OS and 3.0 OS.

Benchmark runs

Merging the following segments to 1 ( forceMerge to 1 )took same amount of time [ except postings ] as they're similar size and all are non compound files

index shard prirep ip        segment generation docs.count docs.deleted  size size.memory committed searchable version compound
big5  0     p      127.0.0.1 _3r            135   23285805            0 4.8gb           0 true      true       10.1.0  false
big5  0     p      127.0.0.1 _aj            379   23121472            0 4.8gb           0 true      true       10.1.0  false
big5  0     p      127.0.0.1 _cc            444   23137636            0 4.8gb           0 true      true       10.1.0  false
big5  0     p      127.0.0.1 _f3            543   23290908            0 4.8gb           0 true      true       10.1.0  false
big5  0     p      127.0.0.1 _f5            545   23164179            0 4.8gb           0 true      true       10.1.0  false

2.19

index shard prirep ip        segment generation docs.count docs.deleted  size size.memory committed searchable version compound
big5  0     p      127.0.0.1 _3r            135   23313500            0 4.8gb           0 true      true       9.12.1  false
big5  0     p      127.0.0.1 _aj            379   22650054            0 4.7gb           0 true      true       9.12.1  false
big5  0     p      127.0.0.1 _cc            444   23130046            0 4.8gb           0 true      true       9.12.1  false
big5  0     p      127.0.0.1 _f1            541   23544447            0 4.8gb           0 true      true       9.12.1  false
big5  0     p      127.0.0.1 _f2            542   23361953            0 4.8gb           0 true      true       9.12.1  false

Format / Version	2.19 (9.12)	3.0 (10.1)	Difference (3.0 - 2.19)	% Change
Stored Fields	297,278 ms	280,421 ms	-16,857 ms	-5.67%
Norms	6,497 ms	6,388 ms	-109 ms	-1.68%
Postings	447,418 ms	575,100 ms	+127,682 ms	28.54%
Doc Values	223,610 ms	219,824 ms	-3,786 ms	-1.69%
Points	72,534 ms	75,501 ms	+2,967 ms	4.09%
Field Infos	0 ms	0 ms	0 ms	0%

When segments are of same size but some segments are part of compound file, we notice a big drop in time taken for stored fields

3.0

index shard prirep ip        segment generation docs.count docs.deleted    size size.memory committed searchable version compound
big5  0     p      127.0.0.1 _b              11   21474616            0   4.4gb           0 true      true       10.1.0  false
big5  0     p      127.0.0.1 _h              17   21556006            0   4.4gb           0 true      true       10.1.0  false
big5  0     p      127.0.0.1 _p              25   21447620            0   4.4gb           0 true      true       10.1.0  false
big5  0     p      127.0.0.1 _x              33   21544817            0   4.4gb           0 true      true       10.1.0  false
big5  0     p      127.0.0.1 _12             38    3582255            0 754.9mb           0 true      true       10.1.0  true
big5  0     p      127.0.0.1 _13             39    3579532            0 754.1mb           0 true      true       10.1.0  true
big5  0     p      127.0.0.1 _14             40    1071872            0 228.9mb           0 true      true       10.1.0  true
big5  0     p      127.0.0.1 _15             41   21743282            0   4.5gb           0 true      true       10.1.0  false

2.19

index shard prirep ip        segment generation docs.count docs.deleted    size size.memory committed searchable version compound
big5  0     p      127.0.0.1 _b              11   21474616            0   4.4gb           0 true      true       9.12.1  false
big5  0     p      127.0.0.1 _h              17   21427535            0   4.4gb           0 true      true       9.12.1  false
big5  0     p      127.0.0.1 _p              25   21549736            0   4.4gb           0 true      true       9.12.1  false
big5  0     p      127.0.0.1 _y              34   21568569            0   4.4gb           0 true      true       9.12.1  false
big5  0     p      127.0.0.1 _15             41    3581274            0 754.6mb           0 true      true       9.12.1  true
big5  0     p      127.0.0.1 _16             42    3578536            0 753.9mb           0 true      true       9.12.1  true
big5  0     p      127.0.0.1 _17             43   21889332            0   4.5gb           0 true      true       9.12.1  false
big5  0     p      127.0.0.1 _18             44     930402            0 199.1mb

Format / Version	2.19 (9.12)	3.0 (10.1)	Difference (3.0 - 2.19)	% Change
Stored Fields	282,483 ms	402,009 ms	+119,526 ms	42.31%
Norms	7,993 ms	6,186 ms	-1,807 ms	-22.61%
Postings	474,146 ms	571,138 ms	+96,992 ms	20.46%
Doc Values	206,165 ms	203,756 ms	-2,409 ms	-1.17%
Points	87,173 ms	78,683 ms	-8,490 ms	-9.74%
Field Infos	0 ms	0 ms	0 ms	0%
Total Runtime	1,058.0s	1,261.8s	+203.8s	19.26%

postings format observation

We see consistent ~20% regression for postings format. And this is a hot path as we had lots of changes in Lucene 10.

stored fields observation

But surprisingly stored fields also show regression with not many direct changes to the format itself and looks like the compound files merges have slowed down the stored fields based on the following data. Note how the segments where compound = true, stored fields is taking 5x - 10x more time.

3.0


index shard prirep ip        segment generation docs.count docs.deleted    size size.memory committed searchable version compound StoredFieldsTime
big5  0     p      127.0.0.1 _3r            135   22939930            0   4.7gb           0 true      true       10.1.0  false    21984
big5  0     p      127.0.0.1 _54            184   14294416            0   2.9gb           0 true      true       10.1.0  false    13314
big5  0     p      127.0.0.1 _8k            308   23703067            0   4.9gb           0 true      true       10.1.0  false    22745
big5  0     p      127.0.0.1 _af            375   18639586            0   3.8gb           0 true      true       10.1.0  false    40955 <-outlier
big5  0     p      127.0.0.1 _dq            494    2432310            0 513.1mb           0 true      true       10.1.0  true     43307
big5  0     p      127.0.0.1 _dv            499   23959061            0   4.9gb           0 true      true       10.1.0  false    22997
big5  0     p      127.0.0.1 _e1            505    2301159            0 485.6mb           0 true      true       10.1.0  true     40955
big5  0     p      127.0.0.1 _ec            516    2714350            0 572.6mb           0 true      true       10.1.0  true     48479
big5  0     p      127.0.0.1 _en            527    2432908            0   514mb           0 true      true       10.1.0  true     43387
big5  0     p      127.0.0.1 _ey            538    2583213            0   545mb           0 true      true       10.1.0  true     46936


2.19

 index shard prirep ip        segment generation docs.count docs.deleted     size size.memory committed searchable version compound  StoredFieldsTime
big5  0     p      127.0.0.1 _3r            135   23357403            0    4.8gb           0 true      true       9.12.1  false      23406
big5  0     p      127.0.0.1 _aj            379   23018556            0    4.7gb           0 true      true       9.12.1  false      23035
big5  0     p      127.0.0.1 _ca            442   23341816            0    4.8gb           0 true      true       9.12.1  false      23382
big5  0     p      127.0.0.1 _cp            457    4417375            0  929.9mb           0 true      true       9.12.1  true       4419
big5  0     p      127.0.0.1 _d9            477    4839945            0 1017.1mb           0 true      true       9.12.1  true       4842 
big5  0     p      127.0.0.1 _dv            499    4268096            0  897.9mb           0 true      true       9.12.1  true       4625
big5  0     p      127.0.0.1 _ef            519    4686246            0  986.2mb           0 true      true       9.12.1  true       4692
big5  0     p      127.0.0.1 _eq            530    2672224            0  563.9mb           0 true      true       9.12.1  true       2671
big5  0     p      127.0.0.1 _f0            540    2143226            0  453.4mb           0 true      true       9.12.1  true       2143
big5  0     p      127.0.0.1 _f1            541   23255113            0    4.8gb           0 true      true       9.12.1  false      22303

Expected behavior

The force merge times of various file formats must be same for 3.0 OS and 2.19 OS.

[bharath-techie]

Another datapoint for different set of segments w.r.t time taken for stored fields [ note compound = true ]

3.0 

index shard prirep ip        segment generation docs.count docs.deleted    size size.memory committed searchable version compound StoredFieldsTime
big5  0     p      127.0.0.1 _b              11   21474616            0   4.4gb           0 true      true       10.1.0  false     21497
big5  0     p      127.0.0.1 _h              17   21556006            0   4.4gb           0 true      true       10.1.0  false     20579
big5  0     p      127.0.0.1 _p              25   21447620            0   4.4gb           0 true      true       10.1.0  false     20477
big5  0     p      127.0.0.1 _x              33   21544817            0   4.4gb           0 true      true       10.1.0  false     20578
big5  0     p      127.0.0.1 _12             38    3582255            0 754.9mb           0 true      true       10.1.0  true      64568
big5  0     p      127.0.0.1 _13             39    3579532            0 754.1mb           0 true      true       10.1.0  true      64248
big5  0     p      127.0.0.1 _14             40    1071872            0 228.9mb           0 true      true       10.1.0  true      18537
big5  0     p      127.0.0.1 _15             41   21743282            0   4.5gb           0 true      true       10.1.0  false     21774  

2.19

index shard prirep ip        segment generation docs.count docs.deleted    size size.memory committed searchable version compound StoredFieldsTime
big5  0     p      127.0.0.1 _b              11   21474616            0   4.4gb           0 true      true       9.12.1  false     21497
big5  0     p      127.0.0.1 _h              17   21427535            0   4.4gb           0 true      true       9.12.1  false     21453
big5  0     p      127.0.0.1 _p              25   21549736            0   4.4gb           0 true      true       9.12.1  false     21571
big5  0     p      127.0.0.1 _y              34   21568569            0   4.4gb           0 true      true       9.12.1  false     21596
big5  0     p      127.0.0.1 _15             41    3581274            0 754.6mb           0 true      true       9.12.1  true      2593
big5  0     p      127.0.0.1 _16             42    3578536            0 753.9mb           0 true      true       9.12.1  true      3579
big5  0     p      127.0.0.1 _17             43   21889332            0   4.5gb           0 true      true       9.12.1  false     21909
big5  0     p      127.0.0.1 _18             44     930402            0 199.1mb           0 true      true       9.12.1  true      929

[bharath-techie]

monitor -c 60 org.apache.lucene.store.DataInput readBytes
2.19

 timestamp                  class                                   method                                 total         success      fail         avg-rt(ms)    fail-rate   
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 2025-03-27 16:57:09.441    org.apache.lucene.store.MemorySegmentI  readBytes                              709793        709793       0            0.01          0.00%       
                            ndexInput                                                                                                                                        
 2025-03-27 16:56:40.393    org.apache.lucene.store.BufferedIndexI  readBytes                              1006231       1006231      0            0.02          0.00%       
                            nput                                                                                                                                             
 2025-03-27 16:57:09.441    org.apache.lucene.store.DataInput       readBytes                              1212909       1212909      0            0.04          0.00%       
 2025-03-27 16:57:09.441    org.apache.lucene.store.BufferedChecks  readBytes                              1212909       1212909      0            0.03          0.00%       
                            umIndexInput                                                                                                                                     

3.0

 timestamp                  class                                   method                                 total         success      fail         avg-rt(ms)    fail-rate   
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 2025-03-27 16:55:28.724    org.apache.lucene.store.MemorySegmentI  readBytes                              36446         36446        0            0.13          0.00%       
                            ndexInput                                                                                                                                        
 2025-03-27 16:55:28.724    org.apache.lucene.store.DataInput       readBytes                              898332        898332       0            0.06          0.00%       
 2025-03-27 16:55:22.832    org.apache.lucene.store.BufferedIndexI  readBytes                              1723772       1723772      0            0.02          0.00%       
                            nput                                                                                                                                             
 2025-03-27 16:55:28.724    org.apache.lucene.store.BufferedChecks  readBytes                              898332        898332       0            0.05          0.00%       
                            umIndexInput                                                                                                                                     

 timestamp                  class                                   method                                 total         success      fail         avg-rt(ms)    fail-rate   
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 2025-03-27 16:56:20.846    org.apache.lucene.store.MemorySegmentI  readBytes                              275586        275586       0            0.15          0.00%       
                            ndexInput                                                                                                                                        
 2025-03-27 16:56:28.725    org.apache.lucene.store.DataInput       readBytes                              401138        401138       0            0.14          0.00%       
 2025-03-27 16:56:28.725    org.apache.lucene.store.BufferedIndexI  readBytes                              251104        251104       0            0.02          0.00%       
                            nput                                                                                                                                             
 2025-03-27 16:56:28.725    org.apache.lucene.store.BufferedChecks  readBytes                              401138        401138       0            0.13          0.00%       
                            umIndexInput      

One thing I do see in arthas is the average time taken for MemorySegmentInput readBytes is a lot higher [0.15] in 3.0 compared to 2.19 [0.1].

Flow :

So the majority of the time goes to checkIntegrity() method of readers during stored fields merge.

As part of it CodecUtil.checksumEntireFile gets called which calls seek() which calls skipByReading which eventually calls the main.readBytes() in BufferedChecksumInput which could be any of the above indexInput types and memorySegmentIndexInput seems to have the difference.

CodecUtil.java
  /**
   * Clones the provided input, reads all bytes from the file, and calls {@link #checkFooter}
   *
   * <p>Note that this method may be slow, as it must process the entire file. If you just need to
   * extract the checksum value, call {@link #retrieveChecksum}.
   */
  public static long checksumEntireFile(IndexInput input) throws IOException {
    long st = System.currentTimeMillis();
    IndexInput clone = input.clone();
    System.out.println("Clone took : " + (System.currentTimeMillis() - st));
    clone.seek(0);
    st = System.currentTimeMillis();
    System.out.println("Clone seek took : " + (System.currentTimeMillis() - st));
    st = System.currentTimeMillis();
    ChecksumIndexInput in = new BufferedChecksumIndexInput(clone);
    System.out.println("ChecksumIndexInput init took : " + (System.currentTimeMillis() - st));
    assert in.getFilePointer() == 0;
    if (in.length() < footerLength()) {
      throw new CorruptIndexException(
          "misplaced codec footer (file truncated?): length="
              + in.length()
              + " but footerLength=="
              + footerLength(),
          input);
    }
    st = System.currentTimeMillis();
    System.out.println("Length : " + in.length() + " , footer length : " + footerLength()); 
    in.seek(in.length() - footerLength()); // THIS takes the Majority of time
    System.out.println("Seek took : " + (System.currentTimeMillis() - st));

    return checkFooter(in);
  }

BufferedChecksumIndexInput.java 
  @Override
  public void readBytes(byte[] b, int offset, int len) throws IOException {
    //long st = System.currentTimeMillis();
    main.readBytes(b, offset, len); // This calls the indexinput readBytes
    //System.out.println("Main read bytes time : " + (System.currentTimeMillis() - st));
    //st = System.currentTimeMillis();
    digest.update(b, offset, len);
    //System.out.println("Digest update time : " + (System.currentTimeMillis() - st));
  }

[bharath-techie]

Uploading flame graphs for different merges bench 2.zip

[bharath-techie]

Sample logs for 2.19 :

index shard prirep ip        segment generation docs.count docs.deleted    size size.memory committed searchable version compound StoredFieldsTime
big5  0     p      127.0.0.1 _b 11   21474616            0   4.4gb           0 true      true       9.12.1  false     21497
big5  0     p      127.0.0.1 _h 17   21427535            0   4.4gb           0 true      true       9.12.1  false     21453
big5  0     p      127.0.0.1 _p 25   21549736            0   4.4gb           0 true      true       9.12.1  false     21571
big5  0     p      127.0.0.1 _y 34   21568569            0   4.4gb           0 true      true       9.12.1  false     21596
big5  0     p      127.0.0.1 _1541    3581274            0 754.6mb           0 true      true       9.12.1  true      2593
big5  0     p      127.0.0.1 _1642    3578536            0 753.9mb           0 true      true       9.12.1  true      3579
big5  0     p      127.0.0.1 _1743   21889332            0   4.5gb           0 true      true       9.12.1  false     21909
big5  0     p      127.0.0.1 _1844     930402            0 199.1mb           0 true      true       9.12.1  true      929

Check integrity :

[2025-03-28T08:47:59,583][INFO ][stdout] Time taken to check integrity : 2593
[2025-03-28T08:48:21,180][INFO ][stdout] Time taken to check integrity : 21595
[2025-03-28T08:48:24,761][INFO ][stdout] Time taken to check integrity : 3579
[2025-03-28T08:48:46,334][INFO ][stdout] Time taken to check integrity : 21571
[2025-03-28T08:49:07,789][INFO ][stdout] Time taken to check integrity : 21453
[2025-03-28T08:49:29,700][INFO ][stdout] Time taken to check integrity : 21909
[2025-03-28T08:49:30,631][INFO ][stdout] Time taken to check integrity : 929
[2025-03-28T08:49:52,129][INFO ][stdout] Time taken to check integrity : 21497
[2025-03-28T08:49:52,130][INFO ][stdout] Time taken to add sub : 21498

Copy chunks : 

Format - NumDocs CheckIntegrityTime
CopyChunks time

[2025-03-28T08:49:52,499][INFO ][stdout] Num docs: 3581274 2593
[2025-03-28T08:49:52,499][INFO ][stdout] Time taken : 368
[2025-03-28T08:49:55,036][INFO ][stdout]  Num docs: 21568569 21596
[2025-03-28T08:49:55,036][INFO ][stdout] Time taken : 2536
[2025-03-28T08:49:55,383][INFO ][stdout]  Num docs:  : 3578536 3579
[2025-03-28T08:49:55,383][INFO ][stdout] Time taken : 347
[2025-03-28T08:50:10,969][INFO ][stdout]  Num docs : 21549736 21571
[2025-03-28T08:50:10,969][INFO ][stdout] Time taken : 15585
[2025-03-28T08:51:14,512][INFO ][stdout]  Num docs  : 21427535 21453
[2025-03-28T08:51:14,512][INFO ][stdout] Time taken : 63543
[2025-03-28T08:51:59,856][INFO ][stdout]  Num docs:  : 21889332 21909
[2025-03-28T08:51:59,857][INFO ][stdout] Time taken : 45345
[2025-03-28T08:51:59,949][INFO ][stdout]  Num docs:  : 930402 929
[2025-03-28T08:51:59,949][INFO ][stdout] Time taken : 92
[2025-03-28T08:52:44,082][INFO ][stdout]  Num docs : 21474616 21497
[2025-03-28T08:52:44,083][INFO ][stdout] Time taken : 44134

[bharath-techie]

When I make compound as false , can see that stored fields checkIntegrity performance is same for both 3.0 and 2.19. So we can conclude that issue is present only for cfs files and probably because they're memory mapped. [FDT files are nioFs ]
3.0

index shard prirep ip        segment generation docs.count docs.deleted    size size.memory committed searchable version compound StoredFieldsTime (CheckIntegrity)
big5  0     p      127.0.0.1 _3r            135   24041539            0   4.9gb           0 true      true       10.1.0  false     24071
big5  0     p      127.0.0.1 _55            185   13830976            0   2.8gb           0 true      true       10.1.0  false     13840
big5  0     p      127.0.0.1 _8i            306   24155048            0     5gb           0 true      true       10.1.0  false     24180
big5  0     p      127.0.0.1 _9v            355   13964122            0   2.8gb           0 true      true       10.1.0  false     13982
big5  0     p      127.0.0.1 _af            375    4350771            0 915.8mb           0 true      true       10.1.0  false     4355
big5  0     p      127.0.0.1 _dp            493    2314600            0   488mb           0 true      true       10.1.0  false     2315
big5  0     p      127.0.0.1 _dt            497   24207153            0     5gb           0 true      true       10.1.0  false     24244
big5  0     p      127.0.0.1 _e1            505    2715792            0 573.1mb           0 true      true       10.1.0  false     2718
big5  0     p      127.0.0.1 _ec            516    2316202            0 489.4mb           0 true      true       10.1.0  false     2321
big5  0     p      127.0.0.1 _ev            535    4103797            0 864.7mb           0 true      true       10.1.0  false     3114

SM: 271191 ms to merge stored fields [116000000 docs]

2.19

index shard prirep ip        segment generation docs.count docs.deleted    size size.memory committed searchable version compound StoredFieldsTime
big5  0     p      127.0.0.1 _3r            135   23271771            0   4.8gb           0 true      true       9.12.1  false    22968
big5  0     p      127.0.0.1 _aj            379   22951217            0   4.7gb           0 true      true       9.12.1  false    22965
big5  0     p      127.0.0.1 _ca            442   22470512            0   4.6gb           0 true      true       9.12.1  false    22496
big5  0     p      127.0.0.1 _cp            457    4398798            0 925.6mb           0 true      true       9.12.1  false    4416
big5  0     p      127.0.0.1 _d9            477    4671258            0 982.2mb           0 true      true       9.12.1  false    4674
big5  0     p      127.0.0.1 _dv            499    4387189            0 923.4mb           0 true      true       9.12.1  false    4389
big5  0     p      127.0.0.1 _eg            520    4682758            0 984.9mb           0 true      true       9.12.1  false    4686
big5  0     p      127.0.0.1 _f1            541    4418359            0 930.4mb           0 true      true       9.12.1  false    4427
big5  0     p      127.0.0.1 _f7            547   22902725            0   4.7gb           0 true      true       9.12.1  false    22938
big5  0     p      127.0.0.1 _f8            548    1845413            0   391mb           0 true      true       9.12.1  false    1185

SM: 294214 ms to merge stored fields [116000000 docs]

Settings

LogByteSizeMergePolicyProvider.java

     public static final Setting<Double> INDEX_LBS_NO_CFS_RATIO_SETTING = new Setting<>(
         "index.merge.log_byte_size_policy.no_cfs_ratio",
         Double.toString(0.0),
         TieredMergePolicyProvider::parseNoCFSRatio,
         Setting.Property.Dynamic,
         Setting.Property.IndexScope

EngineConfig.java
public static final Setting<Boolean> INDEX_USE_COMPOUND_FILE = Setting.boolSetting(
        "index.use_compound_file",
        false,
        Property.IndexScope
    );

[bharath-techie]

tagging @reta @Bukhtawar @msfroh for visibility and also to check if we've seen any associated issues / any pointers that could help.

[issue number 13325 in Lucene - I've seen this but could not find any relation to this ]

[bharath-techie]

Stats on MemorySegmentIndexInput.readBytes() method for one of the CFS / compound segments
2.19

[2025-04-01T06:40:02,074][INFO ][] Class: org.apache.lucene.store.MemorySegmentIndexInput$SingleSegmentImpl
[2025-04-01T06:40:02,074][INFO ][]   Invocations: 274904
[2025-04-01T06:40:02,074][INFO ][]   Total bytes: 281501526
[2025-04-01T06:40:02,074][INFO ][]   Average bytes per invocation: 1023
[2025-04-01T06:40:02,075][INFO ][]   Total time (ms): 2097.277617
[2025-04-01T06:40:02,075][INFO ][]   Average time per invocation (ms): 0.007629127320810175
[2025-04-01T06:40:02,075][INFO ][]   Throughput (MB/s): 128.00440590186597
[2025-04-01T06:40:02,075][INFO ][] Seek took : 2142
[2025-04-01T06:40:02,075][INFO ][] check footer took : 0
[2025-04-01T06:40:02,075][INFO ][] Time taken to check integrity : 2144

3.0

[2025-04-01T06:22:19,965][INFO ][] Class: org.apache.lucene.store.MemorySegmentIndexInput$SingleSegmentImpl
[2025-04-01T06:22:19,965][INFO ][]   Invocations: 312455
[2025-04-01T06:22:19,965][INFO ][]   Total bytes: 319953161
[2025-04-01T06:22:19,965][INFO ][]   Average bytes per invocation: 1023
[2025-04-01T06:22:19,966][INFO ][]   Total time (ms): 43439.569294
[2025-04-01T06:22:19,966][INFO ][]   Average time per invocation (ms): 0.1390266415771871
[2025-04-01T06:22:19,966][INFO ][]   Throughput (MB/s): 7.0242665168606875
[2025-04-01T06:22:19,966][INFO ][] Seek took : 43498
[2025-04-01T06:22:19,966][INFO ][] check footer took : 0
[2025-04-01T06:22:19,966][INFO ][] Time taken to check integrity : 43502

2.19 big5 data on 3.0

Class: org.apache.lucene.store.MemorySegmentIndexInput$SingleSegmentImpl
[2025-04-01T12:11:24,051][INFO ][]   Invocations: 601762
[2025-04-01T12:11:24,051][INFO ][]   Total bytes: 616203390
[2025-04-01T12:11:24,051][INFO ][]   Average bytes per invocation: 1023
[2025-04-01T12:11:24,051][INFO ][]   Total time (ms): 85482.77517
[2025-04-01T12:11:24,051][INFO ][]   Average time per invocation (ms): 0.14205412633233735
[2025-04-01T12:11:24,051][INFO ][]   Throughput (MB/s): 6.8745702927494685
[2025-04-01T12:11:24,052][INFO ][] Seek took : 85593
[2025-04-01T12:11:24,052][INFO ][] check footer took : 0
[2025-04-01T12:11:24,052][INFO ][] Time taken to check integrity : 85602

So time taken to check integrity regresses regardless of 9.12.1 Lucene data or 10.1 Lucene data on OpenSearch 3.0 / Lucene 10.1.

I even tried reverting the changes in MemorySegmentIndexInput and associated files, still the regression continues.