Can misaligned reposerver parallelism limits (lsremote vs proc) on multisource apps cause deadlock?

[scubadam]

Is it possible for a deadlock to occur if I have lower  lsremote parallelism such as:

  reposerver.git.lsremote.parallelism.limit: "5"
  reposerver.parallelism.limit: "30"

and say several multisource apps all syncing at once.
In fact say i have 40 apps with 2 soruces all updating at once as their appset updated a target ref for all at once.

My theory is that if 5 processes all start and each take out their first source lsremote semaphore it isn't perhaps released until after the next lsremote semaphore is needed? Hence all five deadlock each other. Then grpc timesouts or something happens and it leaves leaked goroutines and stuck pending repo requests metrics. Pods where this happens can't terminate gracefully and must wait termination grace-period and be force killed.

Is this possible? Complete misconfiguration aimed at protecting a ghe instance from the storm, but kind of backfired.
Instead I am trying

  reposerver.git.lsremote.parallelism.limit: "20"
  reposerver.parallelism.limit: "5"

But now obviously the queue builds up But to protect ghe that is the cost - I understand.

Just trying to confirm my observation.

[scubadam]

Actually I disproved this myself. Had a occurrence even with much higher lsremote limit.
I think the problem lies with long running git requests triggerd by redis restart or cache expiration and a request storm triggered by appsets each updating 50+ apps at once. I observed grpc timeouts in the repo server pod (from the client - app controller)
and I think the multisource app or lock on the repo cache or something causes the client initiated termination to NOT propagate to go routine terminations..
Hoping this recent change will maybe help
#25531
given its always multisource apps that get stuck.