Manage Image Overflow in our Quay.io Repository

[dlaw4608]

Closes #769

This pull request introduces a proposed solution for managing the accumulation of image tags in the Quay.io repository.

Core Process:

It automates the cleanup process by filtering out older tags that are no longer relevant, while preserving tags deemed important (e.g., those containing a specific substring). The main components of this solution include fetching tags, filtering based on age and relevance, and deleting obsolete tags.

Key Components:

1. Tag Fetching (fetchTags):

Retrieves image tags from the Quay.io API using the appropriate authentication method.
Handles various HTTP response scenarios and errors.

2. Tag Filtering (filterTags):

Filters tags based on their last modified date and whether they contain a predefined substring.
Classifies tags into those to be deleted and those to be retained.

3. Tag Deletion (deleteTag):

Deletes specified tags from the repository using HTTP DELETE requests.
Handles different response scenarios to determine the success of the deletion operation.
package main

Testing:

To ensure the correctness of the implementation, mock tests have been added to simulate different HTTP responses and scenarios:

fetchTags Tests:

• Error Making Request: Simulates a network error to verify error handling.

• Non-200 Status Codes: Simulates API responses with error status codes (e.g., 400) to ensure proper error reporting.

• JSON Parsing Errors: Tests the handling of malformed JSON responses.

• Successful Response: Validates correct parsing of a successful response with sample tag data.

deleteTag Tests:

• Successful Deletion: Verifies the behavior when a tag is successfully deleted.
• Error Response: Simulates server errors during deletion to check error handling.
• Request Error: Tests the scenario where a network error occurs during deletion.

filterTags Tests:

• Correct Filtering: Tests filtering logic with a mix of old and recent tags, including those with preserved substrings.
• No Deletions: Verifies the behavior when all tags are recent and should not be deleted.

Proposed future solution:

Instead of relying on a custom script that requires manual execution, leveraging the built-in expiration policy available in Quay.io presents a more clean and efficient solution. By using Quay.io's expiration policy, we can automatically manage and clean up old image tags without the need for additional maintenance.

1. Expiration Policy Setup:

Quay.io allows you to define expiration rules directly within the repository settings. These rules can automatically delete image tags after a specified period, such as 6 months, ensuring that outdated and unused images are purged from the repository.

2. Tagging Strategy:

Modify the image tagging process in your CI/CD pipeline to include a timestamp on non-critical tags. For example, tags could be formatted as v1.0.0-20250229 (for an image that expires on February 29, 2025).
Tags that should be preserved indefinitely, like latest, would not include a timestamp, thereby excluding them from the expiration policy.

3. Implementation in CI/CD Workflow:

Update your CI/CD pipeline to automatically append a 6-month expiration date to image tags that are not critical. This approach ensures that images are only retained as long as they are relevant, reducing manual cleanup efforts.

[KevFan]

Nice work @dlaw4608! I think this a great contribution that we'll need to run at least once across multiple quay repositories as we have so many stale/obselete tags that is comsuming unnessary quota in our quay repositories.

I'll continue on your work and address anything outstanding 👍

[guicassolato]

So we want to delete all old tags that are not latest!? What about vX.Y.Z release tags? We may want to expand this to a list of patterns to preserve, I suppose.

[KevFan]

I think it's only latest as it was a PoC, but yes, I'm also thinking of a list of patterns to perserve, such as released tags and release branches 🤔

[KevFan]

Thinking on this, we should probably prevent the image overflow from continuing first (#851) before we look to run something like this to clean up the old/obselete tags

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 82.04%. Comparing base (b90235e) to head (e5ea790).
Report is 584 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #843      +/-   ##
==========================================
- Coverage   82.08%   82.04%   -0.05%     
==========================================
  Files          76       76              
  Lines        6090     6092       +2     
==========================================
- Hits         4999     4998       -1     
- Misses        746      747       +1     
- Partials      345      347       +2     

Flag	Coverage Δ
bare-k8s-integration	4.50% <ø> (-0.01%)	⬇️
controllers-integration	71.85% <ø> (-0.08%)	⬇️
gatewayapi-integration	11.06% <ø> (+0.09%)	⬆️
istio-integration	56.18% <ø> (-0.17%)	⬇️
unit	31.57% <ø> (-0.02%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
api/v1beta1 (u)	71.42% <ø> (ø)
api/v1beta2 (u)	85.35% <ø> (ø)
pkg/common (u)	88.13% <ø> (ø)
pkg/istio (u)	72.50% <ø> (ø)
pkg/log (u)	94.73% <ø> (ø)
pkg/reconcilers (u)	∅ <ø> (∅)
pkg/rlptools (u)	83.64% <ø> (ø)
controllers (i)	82.47% <100.00%> (-0.03%)	⬇️

see 7 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[KevFan]

Given there's no movement on this PR, I'm going to close and reopen again if this comes up again as something we still want to do