Memory leak on network thread

[twoeths]

Describe the bug

There is memory leak on network thread, here is on semi node of unstable

looks like this is due to libp2p upgrade #8890 merged 5 days ago

Expected behavior

no memory leak

Steps to reproduce

No response

Additional context

No response

Operating system

Linux

Lodestar version or commit hash

unstable

[lodekeeper]

Investigation: Network Thread Memory Leak

Confirmed Regression

Comparing unstable-semi (has libp2p v3) vs stable-semi (pre-v3) over 5 days:

Metric	unstable-semi	stable-semi
Network worker heap used	174 → 401 MB (+227 MB)	211 → 183 MB (flat)
Old space heap	117 → 332 MB (+215 MB)	118 → 126 MB (+8 MB)
Socket handles	202 → 202 (flat)	200 → 202 (flat)
External memory	59 → 66 MB (+7 MB)	similar

Growth rate: ~43 MB/day, all in V8 old generation (long-lived JS objects). Not a socket/FD leak.

Gossipsub Cache Comparison (5 days)

Cache	unstable-semi	stable-semi
topics	1441 → 1981 (+540)	2168 → 2201 (+33)
backoff	593 → 432 (-161)	471 → 223 (-248)
score	930 → 947 (+17)	1046 → 951 (-95)
seenCache	114k → 141k (+27k)	175k → 247k (+73k)
mcache	575 → 325 (flat)	1502 → 2876 (fluctuating)

The topics map in gossipsub is growing 16x faster on unstable vs stable. This map stores Map<topicStr, Set<peerId>> — remote peers' subscriptions.

Suspects (PR #8890 — libp2p v3 migration)

1. @libp2p/prometheus-metrics@5 stream tracking
The _track() method adds a message event listener + monkey-patches send on every protocol stream via trackProtocolStream. In libp2p v3, streams use EventTarget (not EventEmitter) — there is no removeAllListeners(). These listeners are never explicitly removed when streams close. If anything holds references to closed streams, they cannot be GC'd.

2. @libp2p/gossipsub@15 topics map
The removePeer() method removes peers from all topic Sets but does NOT remove empty topics from the topics Map. Over time with fork transitions creating new topics, empty/stale topics accumulate. This alone accounts for only a small fraction of the 215 MB, but may indicate broader lifecycle issues.

3. Internal stream/connection tracking
libp2p v3 connection manager tracks protocol streams per connection. If closed streams are retained in any internal data structure, combined with the prometheus-metrics listeners, this prevents GC.

Recommended Next Steps

1. Heap snapshot on unstable-semi network worker — take two snapshots ~1h apart and compare retained sizes
2. Quick test: disable prometheus metrics on network worker (metrics: undefined in libp2p config) to isolate whether _track() is the cause
3. Quick test: revert gossipsub from @libp2p/gossipsub@15 to @chainsafe/libp2p-gossipsub@14 to isolate gossipsub vs libp2p core

/cc @nflaig

[lodekeeper]

Further Findings — Root Cause Identified

After extended local profiling (5 heap snapshots over ~50 min, mainnet checkpoint sync + engineMock), I traced the memory leak to missing signal cleanup in @libp2p/utils AdaptiveTimeout and libp2p ConnectionMonitor.

Root Cause

ConnectionMonitor (connection-monitor.js) runs a ping heartbeat every 10 seconds for every active connection. With ~150 peers, that is ~15 calls/sec to AdaptiveTimeout.getTimeoutSignal(), which creates:

1. AbortSignal.timeout(ms) — V8 allocates internal sourceSignalRef/composedSignalRef WeakRef pairs via FinalizationRegistry
2. anySignal([parentSignal, timeoutSignal]) — attaches event listeners that hold references

The returned signal has a .clear() method (from any-signal@4.1.1) that removes these listeners and releases refs. But neither ConnectionMonitor nor AdaptiveTimeout.cleanUp() ever calls .clear().

For comparison, dial-queue.js, upgrader.js, and random-walk.js all correctly call signal.clear() after use.

Heap Evidence

Snapshot diff (30 min, after cache warm-up plateau):

Object	Count Δ	Rate
WeakRef	+16,641 (12,604 → 29,245)	~550/min
Timeout	+1,766	~59/min
AbortSignal	+1,264	~42/min
{sourceSignalRef, composedSignalRef}	+12,403	~413/min

After forced GC (heapdump triggers GC), WeakRef growth slowed dramatically — confirming V8 can collect them but falls behind under sustained load. Over 5 days, this accumulates to the observed ~215MB old-space growth.

Proposed Fix

In @libp2p/utils AdaptiveTimeout.cleanUp():

cleanUp(signal) {
    // Release anySignal event listeners and AbortSignal.timeout() references
    if (typeof signal.clear === 'function') {
        signal.clear();
    }
    // ... existing timing stats code
}

In libp2p ConnectionMonitor: Add finally block to call signal.clear() + this.timeout.cleanUp(signal) after each ping attempt. Currently cleanUp() is never called at all.

What about the other suspects?

• ❌ prometheus-metrics _track() — listeners GC'd with streams, no leak
• ❌ gossipsub topics map — caps at 291, stabilizes
• ❌ seenCache/fastMsgIdCache — plateaus at ~226K entries, same config as pre-v3
• ❌ MplexStream retention — tracks active connections within expected range
• ❌ retainScore — bounded by TTL, same config as pre-v3

Next Steps

Running A/B comparison right now — two local nodes (patched vs unpatched) side-by-side with identical traffic. Will report heap growth rate comparison. Plan to file upstream PR against libp2p/js-libp2p and libp2p/js-libp2p-utils.

[lodekeeper]

Update from fresh feat1-super network-thread heap snapshots (2026-03-10):

New data

Captured on host via write_heapdump?thread=network:

• network_thread_2026-03-10T14:03:29.032Z.heapsnapshot
• network_thread_2026-03-10T14:05:19.648Z.heapsnapshot
• network_thread_2026-03-10T14:52:45.922Z.heapsnapshot

Retainer signal still showed growth in WeakRef / WeakCell / Listener with abort-composition internals (sourceSignalRef / composedSignalRef).

Root cause identified

Missing cleanup in @libp2p/utils repeating-task timeout path:

• src/repeating-task.ts
• dist/src/repeating-task.js

Per-loop anySignal([...]) was created but not .clear()ed in steady-state execution, so abort listeners accumulate while the task keeps running.

Fix implemented

On hotfix branch, patched repeating-task to clear composed signal in .finally() every loop iteration.

Commit: bf0349f339 (fix(libp2p-utils): clear repeatingTask timeout signals)

Also includes existing adaptive-timeout cleanup fix in the same patch file.

Quick validation

Synthetic harness instrumenting AbortSignal.add/removeEventListener:

• before fix mid-run: add=220, remove=0 (accumulating)
• after fix mid-run: add=216, remove=216 (no accumulation)

Post-stop both eventually balance, but the key is steady-state behavior while running.

[lodekeeper]

Uploaded snapshot artifacts (compressed) here:

• Release: https://github.com/lodekeeper/lodestar/releases/tag/heap-8969-feat1-super-20260310
• Snapshot A (14:03):
  https://github.com/lodekeeper/lodestar/releases/download/heap-8969-feat1-super-20260310/network_thread_2026-03-10T14.03.29.032Z.heapsnapshot.zst
• Snapshot B (14:05):
  https://github.com/lodekeeper/lodestar/releases/download/heap-8969-feat1-super-20260310/network_thread_2026-03-10T14.05.19.648Z.heapsnapshot.zst
• Snapshot C (14:52):
  https://github.com/lodekeeper/lodestar/releases/download/heap-8969-feat1-super-20260310/network_thread_2026-03-10T14.52.45.922Z.heapsnapshot.zst
• Checksums:
  https://github.com/lodekeeper/lodestar/releases/download/heap-8969-feat1-super-20260310/SHA256SUMS.txt

Decompress with:
zstd -d <file>.heapsnapshot.zst

[lodekeeper]

Interim post-deploy validation update (branch nflaig/libp2p-memory-leak@18fec22 rolled out on feat1 fleet).

Warm-up-aware window check (~16:45 → 17:12 UTC)

Using network_worker_nodejs_heap_space_size_used_bytes{space="old"} with peer-convergence context:

• feat1-super peers ~200: old-space first3→last3 avg 116.4 → 114.3 MB (Δ -2.0 MB)
• feat1-solo peers ~200: old-space first3→last3 avg 76.8 → 75.1 MB (Δ -1.7 MB)
• feat1-mainnet-super peers ~207: old-space first3→last3 avg 87.0 → 82.7 MB (Δ -4.3 MB)

This is consistent with no runaway old-space growth on converged nodes in the initial post-warm-up period.

Still warming / not fully converged

• feat1-semi peers ~171 (partial convergence)
• feat1-sas peers ~36 (still low-peer special behavior)

Holding final verdict until the extended 1–2h window completes for these nodes.

Current recommendation: patch looks effective on converged nodes; continue monitoring to finalize fleet-wide verdict.

[lodekeeper]

2h post-deploy follow-up (after rollout of nflaig/libp2p-memory-leak@18fec22):

Window used

• warm-up-aware view from ~16:45 UTC to ~18:05 UTC
• metric: network_worker_nodejs_heap_space_size_used_bytes{space="old"}

Current state

• peers now: feat1-super~204, feat1-solo~201, feat1-mainnet-super~206, feat1-semi~196, feat1-sas~64
• first3→last3 avg old-space deltas in that window:
  • feat1-super: +2.9 MB
  • feat1-solo: +3.7 MB
  • feat1-mainnet-super: +5.5 MB
  • feat1-semi: +8.9 MB (still converging during part of window)
  • feat1-sas: +18.3 MB (low-peer/non-converged behavior)

Interpretation / recommendation

• No immediate runaway pattern on the converged core nodes (super/solo/mainnet-super) in the first 2h post-warm-up window.
• semi is near convergence and sas remains low-peer/noisy; treating those as inconclusive for final closure.
• Recommend continuing passive monitoring for a longer horizon (overnight / ~12-24h) before calling this fully resolved fleet-wide.

Patch direction still looks correct and materially improved vs pre-rollout behavior.

[lodekeeper]

Overnight follow-up check (~18:05 → 21:05 UTC) after nflaig/libp2p-memory-leak@18fec22 rollout:

Snapshot

• peers now: solo~200, semi~204, super~201, mainnet-super~212, sas~67
• socket handles remained stable over the last hour on converged nodes (no upward runaway in active sockets).

Old-space trend (18:05→21:05)

• feat1-super: +22.7 MB over ~3h; last-hour delta -11.0 MB
• feat1-solo: +23.3 MB over ~3h; last-hour delta -4.8 MB
• feat1-mainnet-super: +12.5 MB over ~3h; last-hour delta -5.5 MB
• feat1-semi: +11.1 MB over ~3h; last-hour delta +21.9 MB (watch)
• feat1-sas: low-peer/noisy regime (still not representative for closure)

Recommendation / verdict

• No renewed fleet-wide runaway signal on the converged core nodes (super/solo/mainnet-super) in this window.
• Treat the fix as mitigating in production for core nodes, but keep semi and especially sas under continued passive monitoring before full closure.

If semi continues rising with a sustained positive slope over the next window, I recommend taking fresh post-fix network-thread snapshots there and repeating the retainer diff.

[lodekeeper]

Overnight monitoring update (~18:05 → 22:05 UTC):

Comparison context

I compared feat1 post-fix old-space behavior against control behavior over the same wall-clock window.

• feat1 deltas (median first 30m → median last 30m):
  • feat1-super: +10.3 MB
  • feat1-solo: +21.8 MB
  • feat1-mainnet-super: +18.9 MB
  • feat1-semi: +9.4 MB
  • feat1-sas: -45.0 MB (low-peer/noisy regime)
• control reference in same window:
  • unstable-super: +34.0 MB
  • stable-super: -56.6 MB

Socket handles on converged feat1 nodes remained broadly stable (no runaway handle signature).

Final recommendation (for now)

• Treat this as mitigated in production on converged feat1 nodes; no distinct renewed runaway signature attributable to the original repeating-task leak path.
• Keep passive watch on feat1-semi and low-peer feat1-sas for confidence, but no immediate emergency action indicated.
• Proceed with upstream fix PR path for the repeating-task anySignal cleanup (signal.clear() in steady-state loop).

[lodekeeper]

Upstream work item is now open: libp2p/js-libp2p#3401 (repeating-task anySignal cleanup via per-run signal.clear()).

[lodekeeper]

Post-fix heap snapshot verification update (feat1-super):

I captured fresh post-fix network-thread snapshots using the same endpoint/method as initial RCA:

• T0: network_thread_2026-03-10T22:29:09.189Z.heapsnapshot
• T1: network_thread_2026-03-10T22:48:14.323Z.heapsnapshot

Initial T0↔T1 (~19m) diff shows positive growth in:

• WeakRef
• WeakCell
• sourceSignalRef / composedSignalRef

So at this point, despite patched deployment verification, full closure is not yet proven by post-fix heap evidence.

I have scheduled another later snapshot (T2) to extend the post-fix window and will compare T1↔T2 to determine whether this is warm/churn noise vs sustained residual leak.

[lodekeeper]

Extended post-fix heap verification update (feat1-super, same network-thread snapshot method):

• T0: 2026-03-10T22:29:09.189Z
• T1: 2026-03-10T22:48:14.323Z
• T2: 2026-03-10T23:24:50.615Z

T1 → T2 (~36.6m) constructor/signature deltas:

• object::WeakRef: 237,900 -> 258,068 (+20,168, ~550.9/min)
• hidden::system / WeakCell: 119,379 -> 129,475 (+10,096, ~275.8/min)
• sourceSignalRef / composedSignalRef: 118,831 -> 128,914 (+10,083, ~275.4/min)

Interpretation:

• The signal.clear() patch is present in deployed @libp2p/utils and likely removed one retention source.
• However, this extended post-fix window still shows positive signal on the same family (WeakRef/WeakCell + signal refs), so full closure is not proven yet.

I’ll continue with upstream #3401 follow-up and next RCA slice for the remaining retention path.

[lodekeeper]

Concrete retention-root update from the new RCA slice (T2 snapshot):

I traced one level deeper than sourceSignalRef/composedSignalRef holders to answer "who keeps those holders alive".

Retention chain (observed)

1. object::Object holder nodes (the ones with sourceSignalRef + composedSignalRef) are retained by:
   • hidden::system / WeakCell --[[2|3]]--> holder
2. Those WeakCells are anchored by:
   • object::FinalizationRegistry --[[4|5]]--> WeakCell
3. The FinalizationRegistry roots include system-context registries:
   • sourceSignalsCleanupRegistry
   • dependantSignalsCleanupRegistry

So the dominant post-fix retained population is now in the WeakCell/FinalizationRegistry cleanup-registry path associated with composed-signal cleanup, not the earlier missing signal.clear() path alone.

Why this matters

• signal.clear() removed one retention source and is deployed.
• But snapshot evidence still shows growth in the same family (WeakRef/WeakCell + signal-ref holders), now rooted through cleanup registries.

Mitigation plan (next step)

1. Upstream: keep Benchmark cost of a throw #3401 moving (current CI failure appears unrelated/transient; rerun requested).
2. Prepare a follow-up upstream patch that removes anySignal from repeatingTask timeout composition path and uses explicit listener/timer lifecycle cleanup (no FinalizationRegistry-dependent cleanup path in the hot loop).
3. Re-validate with the same T0/T1/T2 heap workflow on feat1-super.

If this direction looks good, I’ll draft the follow-up patch immediately.