Upgrade Elasticsearch Python client to 9.x and urllib3 2.x#2024
Closed
fressi-elastic wants to merge 64 commits into
Closed
Upgrade Elasticsearch Python client to 9.x and urllib3 2.x#2024fressi-elastic wants to merge 64 commits into
fressi-elastic wants to merge 64 commits into
Conversation
fressi-elastic
force-pushed
the
esclient-upgrade-2
branch
3 times, most recently
from
ee26164 to
1e8aeeb
Compare
fressi-elastic
force-pushed
the
esclient-upgrade-2
branch
from
1e8aeeb to
1421ad0
Compare
fressi-elastic
force-pushed
the
esclient-upgrade-2
branch
from
553574d to
de1c485
Compare
…pgrade of elastisearch client libraries.
fressi-elastic
force-pushed
the
esclient-upgrade-2
branch
from
b9c230c to
46c6f19
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
This PR upgrades the Elasticsearch client libraries and urllib3 to newer major versions, updating from Elasticsearch 8.x to 9.x and urllib3 1.x to 2.x. The changes include API parameter updates, refactoring of header handling logic, and deprecation of certain track dependencies.
Changes:
- Updated
elasticsearch[async]from 8.6.1 to 9.2.1,elastic-transportfrom 8.4.1 to 9.2.1, andurllib3from 1.26.19 to 2.6.3 - Renamed
maxsizeparameter toconnections_per_nodein client initialization to match the new API - Refactored header handling logic into reusable functions and improved MIME type compatibility handling
Reviewed changes
Copilot reviewed 11 out of 12 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| pyproject.toml | Updated dependency versions for elasticsearch, elastic-transport, urllib3, and pip |
| esrally/client/factory.py | Changed maxsize to connections_per_node parameter |
| tests/client/factory_test.py | Updated test to use connections_per_node parameter |
| esrally/client/common.py | Refactored header combining and MIME type compatibility logic into dedicated functions |
| esrally/client/synchronous.py | Updated to use new header handling functions and added endpoint_id/path_parts parameters |
| esrally/client/asynchronous.py | Updated RallyAsyncElasticsearch signature and header handling to use new functions |
| esrally/track/loader.py | Added deprecation handling for elasticsearch and elastic-transport dependencies in tracks |
| esrally/utils/versions.py | Added type hints to is_version_identifier function |
| it/init.py | Updated test distributions to include 9.2.4 and removed 6.8.0/ARM logic |
| tests/client/common_test.py | Removed test file (likely obsolete after refactoring) |
| tests/driver/runner_test.py | Added typing import and @typing.no_type_check decorator to test |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
fressi-elastic
changed the title
urlib3 libraries.
…ocker-compose` (second part).
fressi-elastic
force-pushed
the
esclient-upgrade-2
branch
from
b1a7027 to
24f6479
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 20 out of 21 changed files in this pull request and generated 2 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
fressi-elastic
changed the title
urlib3 libraries.urlib3 libraries.
Contributor
Author
|
This change should fix #2018 |
fressi-elastic
changed the title
urlib3 libraries.
fressi-elastic
added
enhancement
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 36 out of 37 changed files in this pull request and generated 5 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Remove obsolete types-urllib3 stubs and require types-requests>=2.31.0.7 so stubs align with urllib3 2.x inline types. Regenerate uv.lock. Made-with: Cursor
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Copilot: want was unused when want_error is set; drop want for clarity. Made-with: Cursor
Replace docker compose --help with docker compose version so integration test setup still verifies Compose is available without flooding logs with the full help text. Made-with: Cursor
Contributor
Author
|
This PR has been split in two independent PRs: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Upgrades Rally’s Elasticsearch dependencies to the 9.x Python client stack and urllib3 2.x, updates the sync/async Rally client subclasses to match current transport APIs and stricter Elasticsearch 9 request expectations, and tightens documented Elasticsearch version support for Rally 2.14.0+.
Dependency changes
Client factory uses connections_per_node (replacing the old pool maxsize-style parameter) — see
esrally/client/factory.pyBehavior and compatibility
Minimum benchmark target is Elasticsearch 8.0.0
esrally/min-es-version.txt; support for 6.8 / 7.x as benchmark clusters is dropped with this release line.Rally extensions around the official client remain focused on compatibility mode (talking to ES 8 with a 9.x client where appropriate), header handling (including bulk application/x-ndjson where ES 9 is strict), and serverless behavior (e.g. not sending compatible-with where it does not apply).
Legacy _ProductChecker / product-header workaround code paths are removed; client code is aligned with upstream patterns where possible.
Documentation
docs/versions.rst: EOL / support policy for 2.14.0+ (current + previous major) and a Rally ↔ Elasticsearch compatibility matrix.Other doc updates align examples and migration/add-track guidance with 8.x / 9.x (no OSS-distribution wording that no longer applies).
Testing
Integration tests exercise version.minimum_es_version() (8.0.0) and a 9.x distribution — see
it/__init__.py.Unit tests cover the refactored client helpers and perform_request behavior.
Related work
Track dependency fix for elastic/security (pinned elasticsearch / elastic-transport in tracks) lives in rally-tracks (see linked PR there); Rally does not need a long-term loader workaround for that case.
Closes #2018 (if still accurate when merging).
Release / rollout notes
Intended to land in a 2.14.x Rally line after a 2.13.x release so users still on old clusters have a final supported Rally version (per maintainer discussion on the PR).
Performance A/B vs the old client was not explicitly benchmarked in isolation; rely on CI, integration tests, and optional nightly/staging soak as needed.