Skip to content

fix(reqresp): clear composed response timeout signals#9029

Merged
nflaig merged 2 commits intoChainSafe:unstablefrom
lodekeeper:fix/8969-reqresp-pr
Mar 13, 2026
Merged

fix(reqresp): clear composed response timeout signals#9029
nflaig merged 2 commits intoChainSafe:unstablefrom
lodekeeper:fix/8969-reqresp-pr

Conversation

@lodekeeper
Copy link
Copy Markdown
Contributor

Summary

Open the Lodestar side of the memory-leak mitigation by clearing composed response-timeout signals in req/resp request handling.

This is the Lodestar-local follow-up for the network-thread leak investigation in #8969. It addresses the req/resp timeout-signal retention path by replacing the AbortSignal.any(...) composition with a clearable composed signal and explicitly clearing listeners once the response stream is done.

This PR is intentionally scoped to the req/resp fix only.
The separate residual gossipsub topic-retention issue is being tracked upstream in ChainSafe/js-libp2p-gossipsub#543, and we can pick that up once a new libp2p release lands.

Change

  • add createRespSignal() helper in packages/reqresp/src/request/index.ts
  • compose request signal + timeout signal via an AbortController
  • remove abort listeners explicitly via respSignal.clear() after the response closes

Validation

  • pnpm build
  • pnpm lint (warning-only: pre-existing unused biome suppression in packages/light-client/test/unit/webEsmBundle.browser.test.ts)

AI disclosure

This PR was authored with AI assistance (Lodekeeper 🌟).

@lodekeeper lodekeeper requested a review from a team as a code owner March 13, 2026 09:42
@gemini-code-assist
Copy link
Copy Markdown
Contributor

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses a memory leak within the req/resp request handling by refining how timeout signals are managed. It introduces a custom createRespSignal utility that allows for explicit cleanup of AbortSignal listeners, ensuring that resources are released once a response stream is completed. This change is a targeted mitigation for a specific memory retention issue, improving the overall stability and resource efficiency of the system.

Highlights

  • Memory Leak Mitigation: Addressed a memory leak by clearing composed response-timeout signals in req/resp request handling.
  • Custom AbortSignal Composition: Replaced the use of AbortSignal.any() with a new createRespSignal() helper function that provides a clearable composed signal.
  • Explicit Listener Cleanup: Implemented explicit clearing of abort listeners via a new clear() method on the composed signal once the response stream is done.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog
  • packages/reqresp/src/request/index.ts
    • Added a new createRespSignal() helper function to compose and manage AbortSignal instances with a clear() method.
    • Replaced the direct use of AbortSignal.any() with the new createRespSignal() within the sendRequest() function for response timeout handling.
    • Included a call to respSignal.clear() in the sendRequest() function's finally block to explicitly remove abort listeners when the response stream closes.
Activity
  • No human activity has been recorded on this pull request yet.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request effectively addresses a memory leak by introducing a createRespSignal helper to manually manage AbortSignal listeners, replacing AbortSignal.any. The implementation is sound, and the cleanup logic is correctly placed within a finally block to ensure it's always executed. I have one suggestion to further improve the robustness of the new helper function.

Copy link
Copy Markdown
Member

@nflaig nflaig left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems a bit overly complex but given these changes have been tested for 2 days already and fix all memory leaks we have observed it seems fine


type ClearableSignal = AbortSignal & {clear: () => void};

function createRespSignal(signal: AbortSignal | undefined, timeoutMs: number): ClearableSignal {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lodekeeper can you add a comment to the upstream issue: nodejs/node#54614

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure! I'll add a comment with our real-world experience from Lodestar's req/resp networking layer. The composed signals from AbortSignal.any() accumulated ~85MB+ of old-space growth over 24h on a mainnet beacon node — exactly the pattern described in that issue. Our workaround in this PR (manual signal composition with explicit cleanup) eliminated the accumulation.

Drafting the comment now — will post shortly.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Posted on the upstream Node issue here:

I summarized that we saw the same retention pattern in Lodestar's real req/resp networking workload (not just a tight synthetic loop), and linked this PR as the application-side mitigation.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well my intention with my comment here was that we would document the upstream issue in our own code/project. But I guess that works 🤷

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lodekeeper just re-reading @wemeetagain I think he meant you should add a reference in the code to the upstream issue, can you open a follow up PR for that

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call — I misread the intent. Opened #9033 to add a JSDoc comment on createRespSignal() referencing the upstream Node.js issue and the investigation context.

Copy link
Copy Markdown
Member

@matthewkeil matthewkeil left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🤖 🚀

@nflaig nflaig merged commit aeab9f9 into ChainSafe:unstable Mar 13, 2026
18 checks passed
@codecov
Copy link
Copy Markdown

codecov bot commented Mar 13, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 52.32%. Comparing base (8cdde18) to head (9e2cd9b).
⚠️ Report is 2 commits behind head on unstable.

Additional details and impacted files
@@            Coverage Diff            @@
##           unstable    #9029   +/-   ##
=========================================
  Coverage     52.32%   52.32%           
=========================================
  Files           848      848           
  Lines         62507    62507           
  Branches       4603     4603           
=========================================
  Hits          32705    32705           
  Misses        29736    29736           
  Partials         66       66           
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

lodekeeper added a commit to lodekeeper/lodestar that referenced this pull request Mar 13, 2026
Add JSDoc comment to createRespSignal() documenting why we use manual
AbortController composition instead of AbortSignal.any() — it works
around a memory leak in Node.js (nodejs/node#54614) where .any() retains
references to source signals for the lifetime of the longest-lived
signal.

Refs: ChainSafe#8969, ChainSafe#9029
nflaig pushed a commit that referenced this pull request Mar 13, 2026
## Summary

Add a JSDoc comment to `createRespSignal()` in
`packages/reqresp/src/request/index.ts` documenting the upstream Node.js
issue that motivates the manual `AbortController` composition pattern.

Follow-up to #9029 per [review
feedback](#9029 (comment))
— the code should document _why_ we avoid `AbortSignal.any()` and link
to the upstream issue for future maintainers.

## References

- Upstream: nodejs/node#54614
(`AbortSignal.any()` memory leak)
- Investigation: #8969
- Fix PR: #9029

## AI disclosure

This PR was authored with AI assistance (Lodekeeper 🌟).

Co-authored-by: lodekeeper <lodekeeper@users.noreply.github.com>
@wemeetagain
Copy link
Copy Markdown
Member

🎉 This PR is included in v1.41.0 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants