TestPlan can itself lead to excessive memory usage

[ben-manes]

I am migrating Caffeine's 12M test suite from TestNG to Jupiter. The main benefit is that JUnit streams test results to reporting and allows the test arguments to be aggressively garbage collected. TestNG collects these for the reporting at the end, so I was forced to modify internal details to assist the garbage collector. For example, I would simplify the display name if successful to a shared constant, falling back to the detailed name if a failure occurred.

Everything works very well when test sharding across multiple Github Action jobs. When running locally to verify behavior, though, it is unable to succeed when running a single test class due to framework memory bloat. Ideally all of the test invocation information should be streamed, garbage collectable, and have steady-state memory usage. It seems that the test identity information is kept centrally, is heavy, and allowed to accumulate throughout. Is it necessary to retain after being streamed to the reporting infrastructure?

./gradlew caffeine:test --tests "com.github.benmanes.caffeine.cache.AsMapTest" -PjavaVersion=25 --rerun

GC Histogram

$ jcmd | rg "Gradle Test"
76408 worker.org.gradle.process.internal.worker.GradleWorkerMain 'Gradle Test Executor 34'
$ jcmd 76408 GC.class_histogram

 num     #instances         #bytes  class name (module)
-------------------------------------------------------
   1:      11815353     2011298144  [B (java.base@25)
   2:      11813465      283523160  java.lang.String (java.base@25)
   3:       4732245      113573880  java.util.concurrent.ConcurrentHashMap$Node (java.base@25)
   4:       2354782       94191280  org.junit.platform.launcher.TestIdentifier
   5:       2461470       78767040  java.util.LinkedHashMap$Entry (java.base@25)
   6:       2451485       77146448  [Ljava.lang.Object; (java.base@25)
   7:       2355539       75377248  java.lang.ref.SoftReference (java.base@25)
   8:       2354780       75352960  org.junit.platform.engine.support.descriptor.MethodSource
   9:       2354484       75343488  org.gradle.api.internal.tasks.testing.DefaultTestDescriptor
  10:       2354782       56514768  org.junit.platform.engine.UniqueId
  11:       2357670       37722720  java.lang.Long (java.base@25)
  12:       2355195       37683120  java.util.ImmutableCollections$ListN (java.base@25)
  13:       2354782       37676512  org.junit.platform.engine.UniqueId$Segment
  14:       2354647       37674352  org.gradle.internal.id.CompositeIdGenerator$CompositeId
  15:           153       33737968  [Ljava.util.concurrent.ConcurrentHashMap$Node; (java.base@25)
  16:          5585       16684352  [Ljava.util.HashMap$Node; (java.base@25)
  17:         84641        4062768  org.mockito.internal.invocation.InterceptedInvocation
  18:         84641        3385640  java.lang.StackFrameInfo (java.base@25)
  19:         86240        2069760  java.util.LinkedList$Node (java.base@25)
  20:         84777        2034648  java.lang.ref.WeakReference (java.base@25)
  21:        113888        1822208  com.github.benmanes.caffeine.testing.Int

Heap

jcmd 76408 GC.heap_dump /Users/ben/Downloads/dump.hprof

[marcphilipp]

Thanks for reporting! I have a few ideas what to try but won't get to it in the next few days.