Skip to content

[WIP] Fix compress middleware's shouldSkip method to avoid memory growth#4284

Draft
Claude wants to merge 3 commits into
mainfrom
claude/fix-compress-middleware-shouldskip
Draft

[WIP] Fix compress middleware's shouldSkip method to avoid memory growth#4284
Claude wants to merge 3 commits into
mainfrom
claude/fix-compress-middleware-shouldskip

Conversation

@Claude
Copy link
Copy Markdown
Contributor

@Claude Claude AI commented May 14, 2026

Thanks for asking me to work on this. I will get started on it and keep this PR's description up to date as I form a plan and make progress.


This section details on the original issue you should resolve

<issue_title>🐛 [Bug]: Compress middleware's shouldSkip method reads the entire payload into memory while streaming data</issue_title>
<issue_description>### Bug Description

The middleware/compress middleware in Fiber v3 reads the entire response body into memory before deciding whether to compress, even for streaming responses set via c.SendStream() / Response.SetBodyStream(). This defeats streaming and produces unbounded memory growth proportional to the response size.

The trigger is the shouldSkip helper introduced in #3745, specifically the len(c.Response().Body()) == 0 check:

// middleware/compress/compress.go
func shouldSkip(c fiber.Ctx) bool {
    if c.Method() == fiber.MethodHead {
        return true
    }
    status := c.Response().StatusCode()
    if status < 200 ||
        status == fiber.StatusNoContent ||
        status == fiber.StatusResetContent ||
        status == fiber.StatusNotModified ||
        status == fiber.StatusPartialContent ||
        len(c.Response().Body()) == 0 ||                       // <-- this line
        c.Get(fiber.HeaderRange) != "" ||
        hasToken(c.Get(fiber.HeaderCacheControl), "no-transform") ||
        hasToken(c.GetRespHeader(fiber.HeaderCacheControl), "no-transform") {
        return true
    }
    return false
}

In fasthttp, Response.Body() materializes any active bodyStream into the in-memory bodyBuffer via copyZeroAlloc:

// https://github.com/valyala/fasthttp/blob/v1.58.0/http.go#L411-L422
func (resp *Response) Body() []byte {
    if resp.bodyStream != nil {
        bodyBuf := resp.bodyBuffer()
        bodyBuf.Reset()
        _, err := copyZeroAlloc(bodyBuf, resp.bodyStream)
        resp.closeBodyStream(err)
        ...
    }
    return resp.bodyBytes()
}

So a handler that does c.SendStream(s3Body, contentLength) for a large object will have its entire payload pulled into RAM the moment shouldSkip runs.

How to Reproduce

  1. Build the minimal program below and run with a soft memory cap so the impact is visible:

    GOMEMLIMIT=256MiB go run main.go
    
  2. From another terminal, request a 512 MiB streamed payload — no Accept-Encoding header is needed:

    curl -sS --output /dev/null http://localhost:8080/file/512
    
  3. Observe the server's memory ticker: HeapAlloc climbs to ~1 GiB (≈ 2× the payload size due to bytebufferpool reallocations during growth) and Sys grows to ~2 GiB.

  4. As a control, change app.Use(compress.New()) to a no-op and re-run: HeapAlloc stays flat at ~1 MiB for the same 512 MiB streamed response.

In a memory-capped container (docker run -m 512m …), the kernel OOMKills the process on step 2.

Expected Behavior

Streaming responses set via c.SendStream / Response.SetBodyStream should pass through the compress middleware without being materialized into memory.

Fiber Version

v3.2.0

Checklist:

  • I agree to follow Fiber's Code of Conduct.
  • I have checked for existing issues that describe my problem prior to opening this one.
  • I understand that improperly formatted bug reports may be closed without explanation.</issue_description>

Comments on the Issue (you are @claude[agent] in this section)

…esponses

The shouldSkip method in the compress middleware was calling c.Response().Body()
which materializes any active bodyStream into memory. This defeats the purpose
of streaming and causes unbounded memory growth for large streamed responses.

The fix checks if the response is using a body stream via IsBodyStream() before
calling Body() to check if the body is empty. This allows streaming responses
set via c.SendStream/Response.SetBodyStream to pass through without being
materialized into memory.

Added tests to verify:
- Streaming responses without compression work correctly
- Streaming responses with compression are properly compressed by fasthttp

Fixes gofiber/fiber#4294

Agent-Logs-Url: https://github.com/gofiber/fiber/sessions/47dff18e-6997-459d-a63d-4d83aca1b48d

Co-authored-by: gaby <835733+gaby@users.noreply.github.com>
Claude finished work on behalf of gaby May 14, 2026 13:12
@Claude Claude AI requested a review from gaby May 14, 2026 13:12
@ReneWerner87 ReneWerner87 added this to v3 May 14, 2026
@ReneWerner87 ReneWerner87 added this to the v3 milestone May 14, 2026
@codecov
Copy link
Copy Markdown

codecov Bot commented May 14, 2026

Codecov Report

❌ Patch coverage is 0% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.24%. Comparing base (33b7fd4) to head (29ebe66).

Files with missing lines Patch % Lines
middleware/compress/compress.go 0.00% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4284      +/-   ##
==========================================
+ Coverage   91.21%   91.24%   +0.03%     
==========================================
  Files         130      130              
  Lines       12763    12764       +1     
==========================================
+ Hits        11642    11647       +5     
+ Misses        709      705       -4     
  Partials      412      412              
Flag Coverage Δ
unittests 91.24% <0.00%> (+0.03%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR addresses a streaming-related memory growth bug in Fiber’s compress middleware by avoiding response body materialization during the shouldSkip decision logic when the response is a stream.

Changes:

  • Update shouldSkip to check Response.IsBodyStream() before calling Response.Body() to prevent buffering streamed responses in memory.
  • Add streaming-focused middleware tests for requests with and without Accept-Encoding.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
middleware/compress/compress.go Avoids calling Response.Body() for streamed responses during skip checks, preventing stream materialization.
middleware/compress/compress_test.go Adds tests covering streaming responses with/without Accept-Encoding (though current assertions don’t directly detect the regression).

Comment thread middleware/compress/compress_test.go Outdated
Comment thread middleware/compress/compress_test.go Outdated
@gaby
Copy link
Copy Markdown
Member

gaby commented May 14, 2026

@copilot apply changes based on the comments in this thread

Copy link
Copy Markdown
Contributor

Copilot AI commented May 14, 2026

@copilot apply changes based on the comments in this thread

Applied the review feedback in 29ebe66. The streaming regression coverage is now a unit-level shouldSkip() test that verifies the body stream is neither read nor closed, and the remaining streaming test wording was updated to match its actual purpose.

Copilot finished work on behalf of gaby May 14, 2026 13:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

🐛 [Bug]: Compress middleware's shouldSkip method reads the entire payload into memory while streaming data

5 participants