Add LOG_MAX_SIZE environment variables to log groomer by pcorliss · Pull Request #61559 · apache/airflow

pcorliss · 2026-02-06T19:50:50Z

Adds two optional environment variables to the clean-logs.sh script to to delete logs once a AIRFLOW__LOG_MAX_SIZE_PERCENT threshold has been breached or a AIRFLOW__LOG_MAX_SIZE_BYTES has been breached. If either environment variable is present and the threshold has been breached the retention days is reduced until the threshold is met and then resets. This makes it easier to avoid out of disk space issues.

airflow-core/tests/unit/charts/log_groomer.py was updated in parallel with helm-tests/tests/chart_utils/log_groomer.py but both files appear to be duplicates of one another. Removing the airflow-core/tests/unit/charts/log_groomer.py files felt out of scope for this PR but I'm happy to delete it if it's no longer in use.

Was generative AI tooling used to co-author this PR?

Yes (please specify the tool below)

Generated-by: [Cursor + Claude Opus 4.5] following the guidelines

The code has been heavily modified after generation and has been reviewed by a human (me). Code has been tested in our local environment.

boring-cyborg · 2026-02-06T19:50:57Z

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
Be sure to read the Airflow Coding style.
Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack

pcorliss · 2026-02-07T14:29:38Z

I suspect the test failures are due to the following recently fixed issue. I've rebased off latest main branch.

Limit FastAPI version to <0.128.4 #61579

potiuk · 2026-02-15T14:45:14Z

Nice !

boring-cyborg · 2026-02-15T14:45:19Z

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

…1559) * Add LOG_MAX_SIZE_BYTES option to log cleaner * Updated clean logs * Charts updated and tests passing * Linting fixes (cherry picked from commit 1e5f789) Co-authored-by: Philip Corliss <pcorliss@gmail.com>

github-actions · 2026-02-15T14:46:26Z

Backport successfully created: v3-1-test

Status	Branch	Result
✅	v3-1-test

…ache#61559) * Add LOG_MAX_SIZE_BYTES option to log cleaner * Updated clean logs * Charts updated and tests passing * Linting fixes (cherry picked from commit 1e5f789) Co-authored-by: Philip Corliss <pcorliss@gmail.com>

…1559) (#61950) * Add LOG_MAX_SIZE_BYTES option to log cleaner * Updated clean logs * Charts updated and tests passing * Linting fixes (cherry picked from commit 1e5f789) Co-authored-by: Philip Corliss <pcorliss@gmail.com>

n-badtke-cg · 2026-02-16T09:43:42Z

scripts/docker/clean-logs.sh

+    fi
+  fi
+
  find "${DIRECTORY}"/logs -type d -empty -delete || true


L65 should be executed before L54

Hello @n-badtke-cg , I don't quite see the logic bug you're referring to, could you elaborate?

The way I think it works is the following but perhaps I've missed something.

Delete log files based on retention days

If Max size is set and size exceed max size reduce retention days by 1 and go to step 1, otherwise go to step 3.

Delete empty log directories

Sleep

Hello @n-badtke-cg , I don't quite see the logic bug you're referring to, could you elaborate?

The way I think it works is the following but perhaps I've missed something.

Delete log files based on retention days

If Max size is set and size exceed max size reduce retention days by 1 and go to step 1, otherwise go to step 3.

Delete empty log directories

Sleep

I guess you are referring to an old message by me "found a logical bug" that was a false message by me, sorry for that!

And also with this message, I just misinterpret the code, sorry. It's actually kinda nice. It's reducing the retention time temporarily to free up more space in specific situations when needed. After the retention period has been temporarily shortened and storage space freed up, the retention period is reset to the env value.

Fair. Again sorry for the unnecessary messages.

* Add LOG_MAX_SIZE_BYTES option to log cleaner * Updated clean logs * Charts updated and tests passing * Linting fixes

…1559) (#61950) * Add LOG_MAX_SIZE_BYTES option to log cleaner * Updated clean logs * Charts updated and tests passing * Linting fixes (cherry picked from commit 1e5f789) Co-authored-by: Philip Corliss <pcorliss@gmail.com>

* Add LOG_MAX_SIZE_BYTES option to log cleaner * Updated clean logs * Charts updated and tests passing * Linting fixes

pcorliss requested review from amoghrajesh, ashb, bugraoz93, dstandish, gopidesupavan, hussein-awala, jason810496, jedcunningham, jscheffl, kaxil and potiuk as code owners February 6, 2026 19:50

boring-cyborg bot added area:dev-tools area:helm-chart Airflow Helm Chart area:production-image Production image improvements and fixes backport-to-v3-1-test Mark PR with this label to backport to v3-1-test branch labels Feb 6, 2026

pcorliss added 4 commits February 7, 2026 08:28

Add LOG_MAX_SIZE_BYTES option to log cleaner

939e82b

Updated clean logs

5994852

Charts updated and tests passing

f4a106d

Linting fixes

da82a1e

pcorliss force-pushed the log-cleaner-max-bytes branch from 807d65e to da82a1e Compare February 7, 2026 14:28

jscheffl added this to the Airflow Helm Chart 1.20.0 milestone Feb 7, 2026

potiuk approved these changes Feb 15, 2026

View reviewed changes

potiuk merged commit 1e5f789 into apache:main Feb 15, 2026
129 checks passed

This comment was marked as outdated.

Sign in to view

n-badtke-cg reviewed Feb 16, 2026

View reviewed changes

choo121600 pushed a commit to choo121600/airflow that referenced this pull request Feb 22, 2026

Add LOG_MAX_SIZE environment variables to log groomer (apache#61559)

943e498

* Add LOG_MAX_SIZE_BYTES option to log cleaner * Updated clean logs * Charts updated and tests passing * Linting fixes

Subham-KRLX pushed a commit to Subham-KRLX/airflow that referenced this pull request Mar 4, 2026

Add LOG_MAX_SIZE environment variables to log groomer (apache#61559)

81624af

* Add LOG_MAX_SIZE_BYTES option to log cleaner * Updated clean logs * Charts updated and tests passing * Linting fixes

dominikhei pushed a commit to dominikhei/airflow that referenced this pull request Mar 11, 2026

Add LOG_MAX_SIZE environment variables to log groomer (apache#61559)

08f2f71

* Add LOG_MAX_SIZE_BYTES option to log cleaner * Updated clean logs * Charts updated and tests passing * Linting fixes

Ankurdeewan pushed a commit to Ankurdeewan/airflow that referenced this pull request Mar 15, 2026

Add LOG_MAX_SIZE environment variables to log groomer (apache#61559)

ffcc220

* Add LOG_MAX_SIZE_BYTES option to log cleaner * Updated clean logs * Charts updated and tests passing * Linting fixes

jscheffl mentioned this pull request Mar 17, 2026

Status of testing of Apache Airflow Helm Chart 1.20.0rc1 #63859

Open

33 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add LOG_MAX_SIZE environment variables to log groomer#61559

Add LOG_MAX_SIZE environment variables to log groomer#61559
potiuk merged 4 commits intoapache:mainfrom
pcorliss:log-cleaner-max-bytes

pcorliss commented Feb 6, 2026

Uh oh!

boring-cyborg bot commented Feb 6, 2026

Uh oh!

pcorliss commented Feb 7, 2026

Uh oh!

potiuk commented Feb 15, 2026

Uh oh!

Uh oh!

boring-cyborg bot commented Feb 15, 2026

Uh oh!

github-actions bot commented Feb 15, 2026

Uh oh!

This comment was marked as outdated.

Uh oh!

n-badtke-cg Feb 16, 2026

Uh oh!

pcorliss Feb 16, 2026

Uh oh!

n-badtke-cg Feb 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

pcorliss commented Feb 6, 2026

Was generative AI tooling used to co-author this PR?

Uh oh!

boring-cyborg bot commented Feb 6, 2026

Uh oh!

pcorliss commented Feb 7, 2026

Uh oh!

potiuk commented Feb 15, 2026

Uh oh!

Uh oh!

boring-cyborg bot commented Feb 15, 2026

Uh oh!

github-actions bot commented Feb 15, 2026

Backport successfully created: v3-1-test

Uh oh!

This comment was marked as outdated.

Uh oh!

n-badtke-cg Feb 16, 2026

Choose a reason for hiding this comment

Uh oh!

pcorliss Feb 16, 2026

Choose a reason for hiding this comment

Uh oh!

n-badtke-cg Feb 16, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants