Skip to content

Unblock geonames for public serverless#465

Merged
gbanasiak merged 2 commits intoelastic:masterfrom
gbanasiak:serverless-geonames
Oct 17, 2023
Merged

Unblock geonames for public serverless#465
gbanasiak merged 2 commits intoelastic:masterfrom
gbanasiak:serverless-geonames

Conversation

@gbanasiak
Copy link
Copy Markdown
Contributor

@gbanasiak gbanasiak commented Oct 6, 2023

Prepares geonames track for public serverless use by:

  • skipping index settings which are not available to serverless users,
  • introducing an optional post-ingest sleep in place of a force merge followed by index stats loop.

Remarks:

  • An alternative to "index.requests.cache.enable": false is to enable request_cache=true parameter in the queries which is proposed in Add serverless-aware runners rally#1789.
  • The build_flavor != "serverless" or serverless_operator == true expression is safe in older Rally versions where build_flavor and serverless_operator variables are undefined (unlike build_flavor != "serverless" or serverless_operator) . In such case the expression is evaluated as true.
  • The tags in {# ... #} comments are meant to allow us an automated edit/removal in the future.

Test:

Details
% esrally race --track-path=../rally-tracks/geonames --target-hosts=${ES_HOST}:443 --pipeline=benchmark-only --client-options="use_ssl:true,api_key:${ES_API_KEY}" --on-error=abort --test-mode --track-params="post_ingest_sleep:true,post_ingest_sleep_duration:5"

    ____        ____
   / __ \____ _/ / /_  __
  / /_/ / __ `/ / / / / /
 / _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, /
                /____/

[INFO] Race id is [48b2a64e-537f-453f-bf57-2be99b9e3b5b]
[INFO] Detected Elasticsearch Serverless mode with operator=[False].
[INFO] Excluding [check-cluster-health], [force-merge], [wait-until-merges-finish], [index-stats], [node-stats] as challenge [append-no-conflicts] is run on serverless.
[INFO] Racing on track [geonames], challenge [append-no-conflicts] and car ['external'] with version [oss].

Running delete-index                                                           [100% done]
Running create-index                                                           [100% done]
Running index-append                                                           [100% done]
Running refresh-after-index                                                    [100% done]
Running refresh-after-force-merge                                              [100% done]
Running post-ingest-sleep                                                      [100% done]
Running default                                                                [100% done]
Running term                                                                   [100% done]
Running phrase                                                                 [100% done]
Running country_agg_uncached                                                   [100% done]
Running country_agg_cached                                                     [100% done]
Running scroll                                                                 [100% done]
Running expression                                                             [100% done]
Running painless_static                                                        [100% done]
Running painless_dynamic                                                       [100% done]
Running decay_geo_gauss_function_score                                         [100% done]
Running decay_geo_gauss_script_score                                           [100% done]
Running field_value_function_score                                             [100% done]
Running field_value_script_score                                               [100% done]
Running large_terms                                                            [100% done]
Running large_filtered_terms                                                   [100% done]
Running large_prohibited_terms                                                 [100% done]
Running desc_sort_population                                                   [100% done]
Running desc_sort_population_can_match_shortcut                                [100% done]
Running desc_sort_population_no_can_match_shortcut                             [100% done]
Running asc_sort_population                                                    [100% done]
Running asc_sort_with_after_population                                         [100% done]
Running desc_sort_geonameid                                                    [100% done]
Running desc_sort_with_after_geonameid                                         [100% done]
Running asc_sort_geonameid                                                     [100% done]
Running asc_sort_with_after_geonameid                                          [100% done]
Running sort_country_code_can_match_shortcut                                   [100% done]
Running sort_country_code_no_can_match_shortcut                                [100% done]

------------------------------------------------------
    _______             __   _____
   / ____(_)___  ____ _/ /  / ___/_________  ________
  / /_  / / __ \/ __ `/ /   \__ \/ ___/ __ \/ ___/ _ \
 / __/ / / / / / /_/ / /   ___/ / /__/ /_/ / /  /  __/
/_/   /_/_/ /_/\__,_/_/   /____/\___/\____/_/   \___/
------------------------------------------------------
            
|                        Metric |                                       Task |     Value |    Unit |
|------------------------------:|-------------------------------------------:|----------:|--------:|
|                Min Throughput |                               index-append | 1770.73   |  docs/s |
|               Mean Throughput |                               index-append | 1770.73   |  docs/s |
|             Median Throughput |                               index-append | 1770.73   |  docs/s |
|                Max Throughput |                               index-append | 1770.73   |  docs/s |
|       50th percentile latency |                               index-append |  366.799  |      ms |
|      100th percentile latency |                               index-append |  553.757  |      ms |
|  50th percentile service time |                               index-append |  366.799  |      ms |
| 100th percentile service time |                               index-append |  553.757  |      ms |
|                    error rate |                               index-append |    0      |       % |
|                Min Throughput |                                    default |    6.6    |   ops/s |
|               Mean Throughput |                                    default |    6.6    |   ops/s |
|             Median Throughput |                                    default |    6.6    |   ops/s |
|                Max Throughput |                                    default |    6.6    |   ops/s |
|      100th percentile latency |                                    default |  202.443  |      ms |
| 100th percentile service time |                                    default |   49.5111 |      ms |
|                    error rate |                                    default |    0      |       % |
|                Min Throughput |                                       term |    6.61   |   ops/s |
|               Mean Throughput |                                       term |    6.61   |   ops/s |
|             Median Throughput |                                       term |    6.61   |   ops/s |
|                Max Throughput |                                       term |    6.61   |   ops/s |
|      100th percentile latency |                                       term |  200.857  |      ms |
| 100th percentile service time |                                       term |   48.8655 |      ms |
|                    error rate |                                       term |    0      |       % |
|                Min Throughput |                                     phrase |    6.75   |   ops/s |
|               Mean Throughput |                                     phrase |    6.75   |   ops/s |
|             Median Throughput |                                     phrase |    6.75   |   ops/s |
|                Max Throughput |                                     phrase |    6.75   |   ops/s |
|      100th percentile latency |                                     phrase |  197.453  |      ms |
| 100th percentile service time |                                     phrase |   48.637  |      ms |
|                    error rate |                                     phrase |    0      |       % |
|                Min Throughput |                       country_agg_uncached |    6.4    |   ops/s |
|               Mean Throughput |                       country_agg_uncached |    6.4    |   ops/s |
|             Median Throughput |                       country_agg_uncached |    6.4    |   ops/s |
|                Max Throughput |                       country_agg_uncached |    6.4    |   ops/s |
|      100th percentile latency |                       country_agg_uncached |  207.075  |      ms |
| 100th percentile service time |                       country_agg_uncached |   50.155  |      ms |
|                    error rate |                       country_agg_uncached |    0      |       % |
|                Min Throughput |                         country_agg_cached |    6.18   |   ops/s |
|               Mean Throughput |                         country_agg_cached |    6.18   |   ops/s |
|             Median Throughput |                         country_agg_cached |    6.18   |   ops/s |
|                Max Throughput |                         country_agg_cached |    6.18   |   ops/s |
|      100th percentile latency |                         country_agg_cached |  210.88   |      ms |
| 100th percentile service time |                         country_agg_cached |   48.5347 |      ms |
|                    error rate |                         country_agg_cached |    0      |       % |
|                Min Throughput |                                     scroll |    4.45   | pages/s |
|               Mean Throughput |                                     scroll |    4.45   | pages/s |
|             Median Throughput |                                     scroll |    4.45   | pages/s |
|                Max Throughput |                                     scroll |    4.45   | pages/s |
|      100th percentile latency |                                     scroll |  619.022  |      ms |
| 100th percentile service time |                                     scroll |  168.274  |      ms |
|                    error rate |                                     scroll |    0      |       % |
|                Min Throughput |                                 expression |    5.97   |   ops/s |
|               Mean Throughput |                                 expression |    5.97   |   ops/s |
|             Median Throughput |                                 expression |    5.97   |   ops/s |
|                Max Throughput |                                 expression |    5.97   |   ops/s |
|      100th percentile latency |                                 expression |  218.801  |      ms |
| 100th percentile service time |                                 expression |   50.5396 |      ms |
|                    error rate |                                 expression |    0      |       % |
|                Min Throughput |                            painless_static |    6.46   |   ops/s |
|               Mean Throughput |                            painless_static |    6.46   |   ops/s |
|             Median Throughput |                            painless_static |    6.46   |   ops/s |
|                Max Throughput |                            painless_static |    6.46   |   ops/s |
|      100th percentile latency |                            painless_static |  204.884  |      ms |
| 100th percentile service time |                            painless_static |   48.9381 |      ms |
|                    error rate |                            painless_static |    0      |       % |
|                Min Throughput |                           painless_dynamic |    6.87   |   ops/s |
|               Mean Throughput |                           painless_dynamic |    6.87   |   ops/s |
|             Median Throughput |                           painless_dynamic |    6.87   |   ops/s |
|                Max Throughput |                           painless_dynamic |    6.87   |   ops/s |
|      100th percentile latency |                           painless_dynamic |  194.251  |      ms |
| 100th percentile service time |                           painless_dynamic |   47.8687 |      ms |
|                    error rate |                           painless_dynamic |    0      |       % |
|                Min Throughput |             decay_geo_gauss_function_score |    6.57   |   ops/s |
|               Mean Throughput |             decay_geo_gauss_function_score |    6.57   |   ops/s |
|             Median Throughput |             decay_geo_gauss_function_score |    6.57   |   ops/s |
|                Max Throughput |             decay_geo_gauss_function_score |    6.57   |   ops/s |
|      100th percentile latency |             decay_geo_gauss_function_score |  204.853  |      ms |
| 100th percentile service time |             decay_geo_gauss_function_score |   51.6823 |      ms |
|                    error rate |             decay_geo_gauss_function_score |    0      |       % |
|                Min Throughput |               decay_geo_gauss_script_score |    6.32   |   ops/s |
|               Mean Throughput |               decay_geo_gauss_script_score |    6.32   |   ops/s |
|             Median Throughput |               decay_geo_gauss_script_score |    6.32   |   ops/s |
|                Max Throughput |               decay_geo_gauss_script_score |    6.32   |   ops/s |
|      100th percentile latency |               decay_geo_gauss_script_score |  212.024  |      ms |
| 100th percentile service time |               decay_geo_gauss_script_score |   52.4458 |      ms |
|                    error rate |               decay_geo_gauss_script_score |    0      |       % |
|                Min Throughput |                 field_value_function_score |    6.69   |   ops/s |
|               Mean Throughput |                 field_value_function_score |    6.69   |   ops/s |
|             Median Throughput |                 field_value_function_score |    6.69   |   ops/s |
|                Max Throughput |                 field_value_function_score |    6.69   |   ops/s |
|      100th percentile latency |                 field_value_function_score |  197.333  |      ms |
| 100th percentile service time |                 field_value_function_score |   47.2593 |      ms |
|                    error rate |                 field_value_function_score |    0      |       % |
|                Min Throughput |                   field_value_script_score |    6.37   |   ops/s |
|               Mean Throughput |                   field_value_script_score |    6.37   |   ops/s |
|             Median Throughput |                   field_value_script_score |    6.37   |   ops/s |
|                Max Throughput |                   field_value_script_score |    6.37   |   ops/s |
|      100th percentile latency |                   field_value_script_score |  208.193  |      ms |
| 100th percentile service time |                   field_value_script_score |   50.3659 |      ms |
|                    error rate |                   field_value_script_score |    0      |       % |
|                Min Throughput |                                large_terms |    1.54   |   ops/s |
|               Mean Throughput |                                large_terms |    1.54   |   ops/s |
|             Median Throughput |                                large_terms |    1.54   |   ops/s |
|                Max Throughput |                                large_terms |    1.54   |   ops/s |
|      100th percentile latency |                                large_terms |  995.18   |      ms |
| 100th percentile service time |                                large_terms |  334.262  |      ms |
|                    error rate |                                large_terms |    0      |       % |
|                Min Throughput |                       large_filtered_terms |    1.59   |   ops/s |
|               Mean Throughput |                       large_filtered_terms |    1.59   |   ops/s |
|             Median Throughput |                       large_filtered_terms |    1.59   |   ops/s |
|                Max Throughput |                       large_filtered_terms |    1.59   |   ops/s |
|      100th percentile latency |                       large_filtered_terms |  902.484  |      ms |
| 100th percentile service time |                       large_filtered_terms |  262.027  |      ms |
|                    error rate |                       large_filtered_terms |    0      |       % |
|                Min Throughput |                     large_prohibited_terms |    1.21   |   ops/s |
|               Mean Throughput |                     large_prohibited_terms |    1.21   |   ops/s |
|             Median Throughput |                     large_prohibited_terms |    1.21   |   ops/s |
|                Max Throughput |                     large_prohibited_terms |    1.21   |   ops/s |
|      100th percentile latency |                     large_prohibited_terms | 1162.51   |      ms |
| 100th percentile service time |                     large_prohibited_terms |  324.493  |      ms |
|                    error rate |                     large_prohibited_terms |    0      |       % |
|                Min Throughput |                       desc_sort_population |    6.57   |   ops/s |
|               Mean Throughput |                       desc_sort_population |    6.57   |   ops/s |
|             Median Throughput |                       desc_sort_population |    6.57   |   ops/s |
|                Max Throughput |                       desc_sort_population |    6.57   |   ops/s |
|      100th percentile latency |                       desc_sort_population |  204.742  |      ms |
| 100th percentile service time |                       desc_sort_population |   51.9004 |      ms |
|                    error rate |                       desc_sort_population |    0      |       % |
|                Min Throughput |    desc_sort_population_can_match_shortcut |    6.53   |   ops/s |
|               Mean Throughput |    desc_sort_population_can_match_shortcut |    6.53   |   ops/s |
|             Median Throughput |    desc_sort_population_can_match_shortcut |    6.53   |   ops/s |
|                Max Throughput |    desc_sort_population_can_match_shortcut |    6.53   |   ops/s |
|      100th percentile latency |    desc_sort_population_can_match_shortcut |  204.692  |      ms |
| 100th percentile service time |    desc_sort_population_can_match_shortcut |   50.8307 |      ms |
|                    error rate |    desc_sort_population_can_match_shortcut |    0      |       % |
|                Min Throughput | desc_sort_population_no_can_match_shortcut |    6.48   |   ops/s |
|               Mean Throughput | desc_sort_population_no_can_match_shortcut |    6.48   |   ops/s |
|             Median Throughput | desc_sort_population_no_can_match_shortcut |    6.48   |   ops/s |
|                Max Throughput | desc_sort_population_no_can_match_shortcut |    6.48   |   ops/s |
|      100th percentile latency | desc_sort_population_no_can_match_shortcut |  205.692  |      ms |
| 100th percentile service time | desc_sort_population_no_can_match_shortcut |   50.7306 |      ms |
|                    error rate | desc_sort_population_no_can_match_shortcut |    0      |       % |
|                Min Throughput |                        asc_sort_population |    6.56   |   ops/s |
|               Mean Throughput |                        asc_sort_population |    6.56   |   ops/s |
|             Median Throughput |                        asc_sort_population |    6.56   |   ops/s |
|                Max Throughput |                        asc_sort_population |    6.56   |   ops/s |
|      100th percentile latency |                        asc_sort_population |  204.55   |      ms |
| 100th percentile service time |                        asc_sort_population |   51.4523 |      ms |
|                    error rate |                        asc_sort_population |    0      |       % |
|                Min Throughput |             asc_sort_with_after_population |    6.53   |   ops/s |
|               Mean Throughput |             asc_sort_with_after_population |    6.53   |   ops/s |
|             Median Throughput |             asc_sort_with_after_population |    6.53   |   ops/s |
|                Max Throughput |             asc_sort_with_after_population |    6.53   |   ops/s |
|      100th percentile latency |             asc_sort_with_after_population |  202.312  |      ms |
| 100th percentile service time |             asc_sort_with_after_population |   48.5369 |      ms |
|                    error rate |             asc_sort_with_after_population |    0      |       % |
|                Min Throughput |                        desc_sort_geonameid |    6.54   |   ops/s |
|               Mean Throughput |                        desc_sort_geonameid |    6.54   |   ops/s |
|             Median Throughput |                        desc_sort_geonameid |    6.54   |   ops/s |
|                Max Throughput |                        desc_sort_geonameid |    6.54   |   ops/s |
|      100th percentile latency |                        desc_sort_geonameid |  207.345  |      ms |
| 100th percentile service time |                        desc_sort_geonameid |   53.7286 |      ms |
|                    error rate |                        desc_sort_geonameid |    0      |       % |
|                Min Throughput |             desc_sort_with_after_geonameid |    6.39   |   ops/s |
|               Mean Throughput |             desc_sort_with_after_geonameid |    6.39   |   ops/s |
|             Median Throughput |             desc_sort_with_after_geonameid |    6.39   |   ops/s |
|                Max Throughput |             desc_sort_with_after_geonameid |    6.39   |   ops/s |
|      100th percentile latency |             desc_sort_with_after_geonameid |  206.959  |      ms |
| 100th percentile service time |             desc_sort_with_after_geonameid |   49.8219 |      ms |
|                    error rate |             desc_sort_with_after_geonameid |    0      |       % |
|                Min Throughput |                         asc_sort_geonameid |    6.5    |   ops/s |
|               Mean Throughput |                         asc_sort_geonameid |    6.5    |   ops/s |
|             Median Throughput |                         asc_sort_geonameid |    6.5    |   ops/s |
|                Max Throughput |                         asc_sort_geonameid |    6.5    |   ops/s |
|      100th percentile latency |                         asc_sort_geonameid |  203.484  |      ms |
| 100th percentile service time |                         asc_sort_geonameid |   48.3056 |      ms |
|                    error rate |                         asc_sort_geonameid |    0      |       % |
|                Min Throughput |              asc_sort_with_after_geonameid |    5.86   |   ops/s |
|               Mean Throughput |              asc_sort_with_after_geonameid |    5.86   |   ops/s |
|             Median Throughput |              asc_sort_with_after_geonameid |    5.86   |   ops/s |
|                Max Throughput |              asc_sort_with_after_geonameid |    5.86   |   ops/s |
|      100th percentile latency |              asc_sort_with_after_geonameid |  219.319  |      ms |
| 100th percentile service time |              asc_sort_with_after_geonameid |   48.2602 |      ms |
|                    error rate |              asc_sort_with_after_geonameid |    0      |       % |
|                Min Throughput |       sort_country_code_can_match_shortcut |    6.21   |   ops/s |
|               Mean Throughput |       sort_country_code_can_match_shortcut |    6.21   |   ops/s |
|             Median Throughput |       sort_country_code_can_match_shortcut |    6.21   |   ops/s |
|                Max Throughput |       sort_country_code_can_match_shortcut |    6.21   |   ops/s |
|      100th percentile latency |       sort_country_code_can_match_shortcut |  209.477  |      ms |
| 100th percentile service time |       sort_country_code_can_match_shortcut |   47.6568 |      ms |
|                    error rate |       sort_country_code_can_match_shortcut |    0      |       % |
|                Min Throughput |    sort_country_code_no_can_match_shortcut |    6.73   |   ops/s |
|               Mean Throughput |    sort_country_code_no_can_match_shortcut |    6.73   |   ops/s |
|             Median Throughput |    sort_country_code_no_can_match_shortcut |    6.73   |   ops/s |
|                Max Throughput |    sort_country_code_no_can_match_shortcut |    6.73   |   ops/s |
|      100th percentile latency |    sort_country_code_no_can_match_shortcut |  195.251  |      ms |
| 100th percentile service time |    sort_country_code_no_can_match_shortcut |   46.133  |      ms |
|                    error rate |    sort_country_code_no_can_match_shortcut |    0      |       % |


--------------------------------
[INFO] SUCCESS (took 34 seconds)
--------------------------------

@gbanasiak gbanasiak changed the title Prepare geonames for serverless public Prepare geonames for public serverless Oct 6, 2023
@gbanasiak gbanasiak requested a review from a team October 6, 2023 12:18
@gbanasiak gbanasiak marked this pull request as ready for review October 9, 2023 06:09
@gbanasiak gbanasiak requested a review from inqueue October 9, 2023 06:10
@gbanasiak gbanasiak changed the title Prepare geonames for public serverless Unblock geonames for public serverless Oct 17, 2023
Copy link
Copy Markdown
Member

@gareth-ellis gareth-ellis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@gbanasiak gbanasiak merged commit e2c8cb9 into elastic:master Oct 17, 2023
@gbanasiak gbanasiak deleted the serverless-geonames branch October 17, 2023 15:36
inqueue pushed a commit to inqueue/rally-tracks that referenced this pull request Dec 6, 2023
Unblocks geonames track for public serverless use by:
* skipping index settings which are not available to serverless users,
* introducing an optional post-ingest sleep in place of a force merge
  followed by index stats loop.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants