feat(docker): add minute-level log retention to clean-logs script by pugarte7 · Pull Request #61855 · apache/airflow

pugarte7 · 2026-02-13T11:36:07Z

What does this PR support?

I’m adding the possibility to define an environment variable to clean logs on a minute basis instead of just days, which is how it works right now.

New variable: AIRFLOW__LOG_RETENTION_MINUTES

Why?

Many users, like @n-badtke-cg (the issue opener), needed a more atomic way of getting rid of logs. In big projects, waiting a full day to clean logs isn't optimal, as the files pile up so fast that it makes debugging difficult and storage expensive.

How?

I’ve updated scripts/docker/clean-logs.sh with some conditional logic:

If AIRFLOW__LOG_RETENTION_MINUTES is set and bigger than 0, the script uses find -mmin.

Otherwise, it falls back to the existing -mtime (day-based) logic.
The default is 0, so nothing changes for users unless they opt-in. This also makes it easy to "rollback" to daily cleanup just by setting the minutes back to 0.

Backward Compatibility

There are no breaking changes here. I’ve maintained the existing logic, and the new minute-level option follows the exact same structure that’s already there. If you don't touch the new variable, Airflow behaves exactly as it did before.

Testing

I verified this locally by creating test log files.
Confirmed that it deletes files correctly based on minute-level thresholds.
Confirmed it still falls back to day-based retention when the minute variable is 0 or unset.

Co-authored-by: shreeharshshinde shreeharshshinde@users.noreply.github.com

Closes #61814

boring-cyborg · 2026-02-13T11:36:10Z

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
Be sure to read the Airflow Coding style.
Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack

n-badtke-cg · 2026-02-13T12:15:10Z

Duplication with #61853

chart/values.schema.json

Dockerfile

chart/templates/workers/worker-deployment.yaml

potiuk · 2026-03-10T23:15:02Z

@Wastelander777 This PR has been converted to draft because it does not yet meet our Pull Request quality criteria.

Issues found:

❌ mypy (type checking): Failing: CI image checks / MyPy checks (mypy-providers). Run prek --stage manual mypy-providers --all-files locally to reproduce. You need breeze ci-image build --python 3.10 for Docker-based mypy. See mypy (type checking) docs.
❌ Build docs: Failing: CI image checks / Build documentation (--docs-only). Run breeze build-docs locally to reproduce. See Build docs docs.
❌ Other failing CI checks: Failing: CI image checks / Build documentation (--spellcheck-only). Run prek run --from-ref main locally to reproduce. See static checks docs.

Note: Your branch is 177 commits behind main. Some check failures may be caused by changes in the base branch rather than by your PR. Please rebase your branch and push again to get up-to-date CI results.

What to do next:

The comment informs you what you need to do.
Fix each issue, then mark the PR as "Ready for review" in the GitHub UI - but only after making sure that all the issues are fixed.
Maintainers will then proceed with a normal review.

Converting a PR to draft is not a rejection — it is an invitation to bring the PR up to the project's standards so that maintainer review time is spent productively. If you have questions, feel free to ask on the Airflow Slack.

pugarte7 · 2026-03-11T13:03:09Z

Hi @n-badtke-cg @potiuk !! I have addressed the comments, and the code is passing the CI issues, it should be ready to merge. Feel free to request anything else. Thank you!

n-badtke-cg · 2026-03-11T13:14:26Z

Duplication with #61853

This PR is currently the most progressed one.

boring-cyborg · 2026-03-11T19:38:43Z

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

…s script (#61855) * feat(docker): add minute-level log retention support to clean-logs.sh * feat(docker): import coauthored changes * update inlined scripts in Dockerfile * comments fixed (cherry picked from commit de01a6b) Co-authored-by: Pablo Ugarte <99680272+Wastelander777@users.noreply.github.com>

github-actions · 2026-03-11T19:44:19Z

Backport successfully created: v3-1-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
✅	v3-1-test

potiuk · 2026-03-11T19:45:36Z

scripts/docker/clean-logs.sh

@@ -45,10 +46,12 @@ fi
 retention_days="${RETENTION}"


There is a bug here = RETENTION is sttill used here @Wastelander777

Ups, my bad 😣. What should be the approach? How can I fix that? Do I re-open the issue?

sorry, went over my head too :/

Do I re-open the issue?

no, the PR is already merged, there needs to be a new PR unfortunately

potiuk · 2026-03-11T19:46:19Z

@jscheffl -> I was just about to post this comment before you merged :)

jscheffl · 2026-03-11T20:24:23Z

@jscheffl -> I was just about to post this comment before you merged :)

Sorry... are you on a correction or shall I?

…ache#61855) * feat(docker): add minute-level log retention support to clean-logs.sh * feat(docker): import coauthored changes * update inlined scripts in Dockerfile * comments fixed

n-badtke-cg

Answer

n-badtke-cg · 2026-03-12T09:18:03Z

scripts/docker/clean-logs.sh

@@ -45,10 +46,12 @@ fi
 retention_days="${RETENTION}"


sorry, went over my head too :/

Do I re-open the issue?

no, the PR is already merged, there needs to be a new PR unfortunately

n-badtke-cg · 2026-03-12T09:50:16Z

I created PR #63421

n-badtke-cg · 2026-03-12T12:24:43Z

Issues fixed with merge of #63421

…ache#61855) * feat(docker): add minute-level log retention support to clean-logs.sh * feat(docker): import coauthored changes * update inlined scripts in Dockerfile * comments fixed

pugarte7 requested review from amoghrajesh, ashb, bugraoz93, gopidesupavan, jason810496, jscheffl, kaxil and potiuk as code owners February 13, 2026 11:36

boring-cyborg bot added area:dev-tools backport-to-v3-1-test Mark PR with this label to backport to v3-1-test branch labels Feb 13, 2026

pugarte7 requested review from hussein-awala and jedcunningham as code owners March 6, 2026 10:24

pugarte7 force-pushed the feat/docker-minute-log-retention branch from 5483b2f to 772b333 Compare March 6, 2026 10:43

n-badtke-cg suggested changes Mar 6, 2026

View reviewed changes

chart/values.schema.json Show resolved Hide resolved

Dockerfile Show resolved Hide resolved

chart/templates/workers/worker-deployment.yaml Outdated Show resolved Hide resolved

n-badtke-cg mentioned this pull request Mar 6, 2026

Container Log Groomer should be configurable with a log retention time <1d - OTEL Collector FileLogReceiver #61814

Closed

2 tasks

potiuk marked this pull request as draft March 10, 2026 23:15

pugarte7 added 4 commits March 11, 2026 13:38

feat(docker): add minute-level log retention support to clean-logs.sh

0418328

feat(docker): import coauthored changes

e8206b6

update inlined scripts in Dockerfile

c93b924

comments fixed

61d4562

pugarte7 force-pushed the feat/docker-minute-log-retention branch from e296f06 to 61d4562 Compare March 11, 2026 12:39

Merge branch 'main' into feat/docker-minute-log-retention

d264e95

pugarte7 marked this pull request as ready for review March 11, 2026 13:01

Merge branch 'main' into feat/docker-minute-log-retention

7d2945c

pugarte7 requested a review from n-badtke-cg March 11, 2026 13:04

n-badtke-cg approved these changes Mar 11, 2026

View reviewed changes

n-badtke-cg mentioned this pull request Mar 11, 2026

feat(docker): add minute-level log retention support to clean-logs.sh #61853

Closed

jscheffl approved these changes Mar 11, 2026

View reviewed changes

jscheffl added this to the Airflow Helm Chart 1.20.0 milestone Mar 11, 2026

jscheffl merged commit de01a6b into apache:main Mar 11, 2026
235 of 236 checks passed

potiuk reviewed Mar 11, 2026

View reviewed changes

n-badtke-cg reviewed Mar 12, 2026

View reviewed changes

This was referenced Mar 12, 2026

Add minute-level log retention for log groomer | issue #61814 #61859

Closed

fix(docker): log groomer clean-logs.sh script #63421

Merged

jscheffl mentioned this pull request Mar 17, 2026

Status of testing of Apache Airflow Helm Chart 1.20.0rc1 #63859

Open

33 tasks

Conversation

pugarte7 commented Feb 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR support?

Why?

How?

Backward Compatibility

Testing

Uh oh!

boring-cyborg bot commented Feb 13, 2026

Uh oh!

n-badtke-cg commented Feb 13, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

potiuk commented Mar 10, 2026

Uh oh!

pugarte7 commented Mar 11, 2026

Uh oh!

n-badtke-cg commented Mar 11, 2026

Uh oh!

Uh oh!

boring-cyborg bot commented Mar 11, 2026

Uh oh!

github-actions bot commented Mar 11, 2026

Backport successfully created: v3-1-test

Uh oh!

potiuk Mar 11, 2026

Choose a reason for hiding this comment

Uh oh!

pugarte7 Mar 11, 2026

Choose a reason for hiding this comment

Uh oh!

n-badtke-cg Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

potiuk commented Mar 11, 2026

Uh oh!

jscheffl commented Mar 11, 2026

Uh oh!

n-badtke-cg left a comment

Choose a reason for hiding this comment

Uh oh!

n-badtke-cg Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

n-badtke-cg commented Mar 12, 2026

Uh oh!

n-badtke-cg commented Mar 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

pugarte7 commented Feb 13, 2026 •

edited

Loading

n-badtke-cg commented Mar 12, 2026 •

edited

Loading