Skip to content

fix: Update s3 chunk size to 10 MB#220

Merged
floriankrb merged 1 commit intomainfrom
fix/update-s3-buffer-size
Sep 24, 2025
Merged

fix: Update s3 chunk size to 10 MB#220
floriankrb merged 1 commit intomainfrom
fix/update-s3-buffer-size

Conversation

@gareth-j
Copy link
Member

Description

Increases the s3 chunk size from 1 MB to 10 MB.

What problem does this change solve?

Issues when writing objects to store.

What issue or task does this change relate to?

Additional notes

As a contributor to the Anemoi framework, please ensure that your changes include unit tests, updates to any affected dependencies and documentation, and have been tested in a parallel setting (i.e., with multiple GPUs). As a reviewer, you are also responsible for verifying these aspects and requesting changes if they are not adequately addressed. For guidelines about those please refer to https://anemoi.readthedocs.io/en/latest/

By opening this pull request, I affirm that all authors agree to the Contributor License Agreement.

@gareth-j gareth-j requested a review from b8raoult September 24, 2025 08:17
@gareth-j gareth-j self-assigned this Sep 24, 2025
@github-project-automation github-project-automation bot moved this to To be triaged in Anemoi-dev Sep 24, 2025
@gareth-j gareth-j changed the title Update s3 chunk size to 10 MB fix: Update s3 chunk size to 10 MB Sep 24, 2025
@floriankrb floriankrb merged commit aa20fa8 into main Sep 24, 2025
73 of 76 checks passed
@github-project-automation github-project-automation bot moved this from To be triaged to Done in Anemoi-dev Sep 24, 2025
@floriankrb floriankrb deleted the fix/update-s3-buffer-size branch September 24, 2025 12:00
anaprietonem pushed a commit that referenced this pull request Sep 30, 2025
🤖 Automated Release PR

This PR was created by `release-please` to prepare the next release.
Once merged:

1. A new version tag will be created
2. A GitHub release will be published
3. The changelog will be updated

Changes to be included in the next release:
---


##
[0.4.37](0.4.36...0.4.37)
(2025-09-30)


### Features

* **mlflow auth:** Support for multiple servers
([#217](#217))
([8ccfb1a](8ccfb1a))


### Bug Fixes

* Update s3 chunk size to 10 MB
([#220](#220))
([aa20fa8](aa20fa8))
* Use `yaml` and `json` flag in metadata get command
([#222](#222))
([6af46c4](6af46c4))

---
> [!IMPORTANT]
> Please do not change the PR title, manifest file, or any other
automatically generated content in this PR unless you understand the
implications. Changes here can break the release process.
> ⚠️ Merging this PR will:
> - Create a new release
> - Trigger deployment pipelines
> - Update package versions

 **Before merging:**
 - Ensure all tests pass
 - Review the changelog carefully
 - Get required approvals

[Release-please
documentation](https://github.com/googleapis/release-please)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants