Performance degradation with repeated query execution

[quasiben]

An example of perf degrading over time (Q9)

___________________________________________________________________________________________________
  _ iter _ wall time _ spill D_H _ D_H time _ host alloc total _ host stream delay _ recv-into-host _
  ___________________________________________________________________________________________________
  _ 0    _ 68.4s     _ 491 GB    _ 238.7s   _ 796 GB           _ 43.4s             _ 122K           _
  ___________________________________________________________________________________________________
  _ 1    _ 74.4s     _ 528 GB    _ 259.9s   _ 876 GB           _ 47.9s             _ 133K           _
  ___________________________________________________________________________________________________
  _ 2    _ 84.6s     _ 517 GB    _ 275.4s   _ 995 GB           _ 49.9s             _ 172K           _
  ___________________________________________________________________________________________________
  _ 3    _ 96.5s     _ 536 GB    _ 277.8s   _ 1027 GB          _ 52.1s             _ 178K           _
  ___________________________________________________________________________________________________
  _ 4    _ 90.0s     _ 531 GB    _ 276.4s   _ 1019 GB          _ 56.7s             _ 174K           _
  ___________________________________________________________________________________________________

We are spilling and this is probably what's driving time up/perf down but why do we need to spill more over time ? We think we are correctly cleaning up between queries so we don't think it's a garbage collection issue.