Make ForUtil#BLOCK_SIZE changeable [LUCENE-10319]

[asfimport]

In #11351, I tried to generate a ForUtil whose BLOCK_SIZE=512}, I thought it could be simple since it looks like i only need to change the BLOCK_SIZE, but it turns out that there are a lot of values related to the BLOCK_SIZE but hard coded.

So this approach is trying to make all hard code value related to BLOCK_SIZE to be generated from the BLOCK_SIZE in case we need a different BLOCK_SIZE ForUtil somewhere else or want to change BLOCK_SIZE in postings in feature.

I tried to make the BLOCK_SIZE = 64 / 256 and all tests passed.

---

Migrated from LUCENE-10319 by Feng Guo (@gf2121), updated Feb 08 2022
Pull requests: #545

[asfimport]

Feng Guo (@gf2121) (migrated from JIRA)

Out of curiosity, I tried to run the luceneutil wikimedium1m for block size = 256, but got an error there:

WARNING: cat=AndHighHigh: hit counts differ: 10274+ vs 10884+
WARNING: cat=HighTerm: hit counts differ: 5969+ vs 9423+
WARNING: cat=LowTerm: hit counts differ: 2394+ vs 3325+
WARNING: cat=MedTerm: hit counts differ: 4558+ vs 7118+
WARNING: cat=OrHighHigh: hit counts differ: 5986+ vs 5987+
WARNING: cat=OrHighMed: hit counts differ: 3044+ vs 3445+
Traceback (most recent call last):
  File "/Users/gf/Documents/projects/luceneutil/lucene_benchmark/src/python/localrun.py", line 60, in <module>
    comp.benchmark("baseline_vs_patch")
  File "/Users/gf/Documents/projects/luceneutil/lucene_benchmark/src/python/competition.py", line 494, in benchmark
    searchBench.run(id, base, challenger,
  File "/Users/gf/Documents/projects/luceneutil/lucene_benchmark/src/python/searchBench.py", line 196, in run
    raise RuntimeError('errors occurred: %s' % str(cmpDiffs))
RuntimeError: errors occurred: ([], ['query=+body:web +body:up filter=None sort=None groupField=None hitCount=10274+: wrong hitCount: 10274+ vs 10884+', 'query=body:he body:resulting filter=None sort=None groupField=None hitCount=3044+: wrong hitCount: 3044+ vs 3445+', 'query=body:official filter=None sort=None groupField=None hitCount=4558+: wrong hitCount: 4558+ vs 7118+', 'query=body:thumb filter=None sort=None groupField=None hitCount=5969+: wrong hitCount: 5969+ vs 9423+', 'query=body:years body:pages filter=None sort=None groupField=None hitCount=5986+: wrong hitCount: 5986+ vs 5987+', 'query=body:goods filter=None sort=None groupField=None hitCount=2394+: wrong hitCount: 2394+ vs 3325+'], 1.0)

I guess this error may be something about MaxScore optimization? So i changed the #TOTAL_HITS_THRESHOLD to a very large number for both baseline and candidate  and rerun the benchmark, everything looks good now and i got a rather good report.
But notice that this report does not really make sense since we changed the #TOTAL_HITS_THRESHOLD}, this is just to verify the results are right.

                            TaskQPS baseline      StdDevQPS my_modified_version      StdDev                Pct diff p-value
                          Fuzzy1      118.73     (11.5%)      114.82     (13.0%)   -3.3% ( -24% -   23%) 0.407
                         LowTerm     2369.88      (9.2%)     2323.31      (5.7%)   -2.0% ( -15% -   14%) 0.428
                        PKLookup      250.07      (5.0%)      245.42      (4.3%)   -1.9% ( -10% -    7%) 0.214
                         Prefix3      306.43      (6.9%)      301.82      (7.0%)   -1.5% ( -14% -   13%) 0.502
                        Wildcard      221.77      (5.2%)      218.64      (4.0%)   -1.4% ( -10% -    8%) 0.348
               HighTermMonthSort     1161.02     (12.7%)     1156.95     (11.1%)   -0.4% ( -21% -   26%) 0.928
       BrowseDayOfYearSSDVFacets      140.62      (1.3%)      140.48      (1.1%)   -0.1% (  -2% -    2%) 0.791
                          Fuzzy2       47.51      (8.9%)       47.57      (7.0%)    0.1% ( -14% -   17%) 0.961
                         Respell      200.51      (2.7%)      200.82      (1.4%)    0.2% (  -3% -    4%) 0.823
                       OrHighMed      197.90      (3.0%)      198.36      (3.6%)    0.2% (  -6% -    7%) 0.830
           BrowseMonthSSDVFacets      152.24      (2.8%)      152.74      (1.0%)    0.3% (  -3% -    4%) 0.630
                       OrHighLow      245.11      (3.5%)      245.97      (3.1%)    0.4% (  -6% -    7%) 0.744
                      AndHighLow     1598.05      (7.2%)     1604.55      (4.6%)    0.4% ( -10% -   13%) 0.836
       BrowseDayOfYearTaxoFacets       28.84      (3.0%)       28.99      (3.3%)    0.5% (  -5% -    7%) 0.603
                      OrHighHigh      109.37      (4.2%)      110.14      (4.0%)    0.7% (  -7% -    9%) 0.599
           BrowseMonthTaxoFacets       30.77      (3.5%)       31.00      (4.1%)    0.8% (  -6% -    8%) 0.541
            BrowseDateTaxoFacets       28.71      (3.2%)       28.93      (3.3%)    0.8% (  -5% -    7%) 0.461
           HighTermDayOfYearSort      593.30     (13.5%)      599.82     (13.2%)    1.1% ( -22% -   32%) 0.800
                     AndHighHigh      441.62      (5.0%)      452.99      (4.1%)    2.6% (  -6% -   12%) 0.083
                          IntNRQ      121.71      (6.2%)      124.89      (4.2%)    2.6% (  -7% -   13%) 0.127
                        HighTerm      599.78      (4.2%)      615.86      (2.6%)    2.7% (  -3% -    9%) 0.019
                 MedSloppyPhrase      397.69      (3.1%)      411.46      (3.3%)    3.5% (  -2% -   10%) 0.001
                     MedSpanNear       75.75      (2.8%)       78.59      (1.5%)    3.7% (   0% -    8%) 0.000
            HighIntervalsOrdered      108.30      (2.8%)      112.66      (2.3%)    4.0% (   0% -    9%) 0.000
                    HighSpanNear       23.10      (3.2%)       24.25      (1.5%)    5.0% (   0% -    9%) 0.000
                         MedTerm     1001.40      (4.2%)     1055.70      (2.4%)    5.4% (  -1% -   12%) 0.000
                       LowPhrase      258.65      (2.3%)      278.10      (2.2%)    7.5% (   2% -   12%) 0.000
                      HighPhrase       67.81      (3.0%)       72.94      (3.7%)    7.6% (   0% -   14%) 0.000
                HighSloppyPhrase       20.13      (6.0%)       21.69      (5.9%)    7.7% (  -3% -   20%) 0.000
                       MedPhrase      258.96      (2.6%)      279.48      (3.0%)    7.9% (   2% -   13%) 0.000
             LowIntervalsOrdered      476.40      (3.2%)      516.31      (2.8%)    8.4% (   2% -   14%) 0.000
             MedIntervalsOrdered      112.10      (2.4%)      121.85      (2.9%)    8.7% (   3% -   14%) 0.000
                      AndHighMed      784.68      (5.2%)      856.24      (5.1%)    9.1% (  -1% -   20%) 0.000
                     LowSpanNear       92.93      (1.8%)      101.80      (2.5%)    9.5% (   5% -   14%) 0.000
                 LowSloppyPhrase      250.51      (3.0%)      279.69      (3.6%)   11.6% (   4% -   18%) 0.000

Then, i rollbacked #TOTAL_HITS_THRESHOLD to 1000 and deleted the check of TotalHits In LuceneUtil and rerun the benchmark. As expected, we can see that QPS of tasks with a totalHits diff decreased and others increased. I post the report here in case some one would be interested in. (Not really related to this ISSUE though)

                            TaskQPS baseline      StdDevQPS my_modified_version      StdDev                Pct diff p-value
                     AndHighHigh      214.93      (3.8%)      183.83      (2.6%)  -14.5% ( -20% -   -8%) 0.000
                         MedTerm     2589.52      (4.5%)     2303.67      (5.5%)  -11.0% ( -20% -   -1%) 0.000
                        HighTerm     1750.90      (4.0%)     1560.54      (4.3%)  -10.9% ( -18% -   -2%) 0.000
                      HighPhrase      238.61      (2.8%)      218.08      (4.3%)   -8.6% ( -15% -   -1%) 0.000
                      OrHighHigh      117.03      (1.9%)      107.52      (4.8%)   -8.1% ( -14% -   -1%) 0.000
               HighTermMonthSort      905.11     (10.5%)      864.34      (9.3%)   -4.5% ( -21% -   17%) 0.150
           HighTermDayOfYearSort     1095.73     (10.4%)     1056.20     (11.0%)   -3.6% ( -22% -   19%) 0.288
                        PKLookup      249.62      (3.8%)      241.15      (4.6%)   -3.4% ( -11% -    5%) 0.011
                         LowTerm     2761.54      (4.6%)     2681.22      (6.8%)   -2.9% ( -13% -    8%) 0.111
                         Respell      163.65      (3.4%)      159.17      (3.8%)   -2.7% (  -9% -    4%) 0.016
                        Wildcard      587.89      (2.9%)      573.02      (4.8%)   -2.5% (  -9% -    5%) 0.044
                          IntNRQ      654.86      (4.4%)      644.88      (5.4%)   -1.5% ( -10% -    8%) 0.328
                       LowPhrase      596.01      (4.3%)      587.28      (5.5%)   -1.5% ( -10% -    8%) 0.349
            HighIntervalsOrdered       16.48      (8.9%)       16.26      (6.4%)   -1.3% ( -15% -   15%) 0.586
                      AndHighLow     1665.94      (6.4%)     1649.07      (6.1%)   -1.0% ( -12% -   12%) 0.610
       BrowseDayOfYearSSDVFacets      142.76      (2.5%)      141.87      (3.3%)   -0.6% (  -6% -    5%) 0.507
            BrowseDateTaxoFacets       29.49      (4.2%)       29.40      (3.8%)   -0.3% (  -8% -    8%) 0.796
                       MedPhrase      653.42      (4.6%)      652.05      (5.6%)   -0.2% (  -9% -   10%) 0.897
                          Fuzzy1      116.77      (6.3%)      116.59     (10.4%)   -0.2% ( -15% -   17%) 0.956
       BrowseDayOfYearTaxoFacets       29.58      (4.3%)       29.55      (4.1%)   -0.1% (  -8% -    8%) 0.929
                          Fuzzy2       73.12     (10.4%)       73.04     (10.7%)   -0.1% ( -19% -   23%) 0.974
           BrowseMonthTaxoFacets       31.65      (5.0%)       31.64      (4.9%)   -0.0% (  -9% -   10%) 0.985
           BrowseMonthSSDVFacets      155.25      (3.5%)      155.27      (3.8%)    0.0% (  -7% -    7%) 0.991
                       OrHighMed      267.80      (5.9%)      268.44      (6.2%)    0.2% ( -11% -   13%) 0.900
                       OrHighLow      820.94      (8.5%)      832.70      (7.8%)    1.4% ( -13% -   19%) 0.579
                         Prefix3      483.34      (5.8%)      490.76      (7.1%)    1.5% ( -10% -   15%) 0.453
                 LowSloppyPhrase      268.01      (2.2%)      279.16      (3.9%)    4.2% (  -1% -   10%) 0.000
                     LowSpanNear      518.44      (3.8%)      542.08      (5.2%)    4.6% (  -4% -   14%) 0.002
                 MedSloppyPhrase      252.28      (2.4%)      264.31      (2.2%)    4.8% (   0% -    9%) 0.000
                HighSloppyPhrase      157.88      (2.6%)      165.44      (3.1%)    4.8% (   0% -   10%) 0.000
                    HighSpanNear      232.57      (2.5%)      243.72      (3.5%)    4.8% (  -1% -   11%) 0.000
             LowIntervalsOrdered      697.59      (3.8%)      734.23      (4.8%)    5.3% (  -3% -   14%) 0.000
                     MedSpanNear      171.60      (3.1%)      181.41      (4.4%)    5.7% (  -1% -   13%) 0.000
             MedIntervalsOrdered      356.52      (3.1%)      383.69      (4.1%)    7.6% (   0% -   15%) 0.000
                      AndHighMed      555.66      (4.4%)      617.40      (5.7%)   11.1% (   0% -   22%) 0.000