Odd device/resource busy error with experiment-generator

[manodeep]

I am generating a bunch of experiments to test ESM1.6 layouts (in a jupyter notebook). Right now, I am tweaking the setup - so I can get a perturbation experiment of my liking. When I try to delete the root directory of the experiments, I seem to frequently run into a device/resource busy error. I have gotten around the issue by killing all the relevant (git) processes that are keeping the .git directory busy. Not sure what/how this is happening, but flagging here anyway.

Here is some potential debugging info:

[~/codes/scaling_studies/access-models-scaling @gadi-login-04]$ rm -Rf access-esm1.6_scaling_layouts
rm: cannot remove 'access-esm1.6_scaling_layouts/access-esm1.6-PI-config/.git/objects/pack': Directory not empty
[~/codes/scaling_studies/access-models-scaling @gadi-login-04]$ lsof +D access-esm1.6_scaling_layouts/
lsof: WARNING: can't stat() tracefs file system /sys/kernel/debug/tracing
      Output information may be incomplete.
lsof: WARNING: can't stat() lustre file system /private/lustre/gadiscr2
      Output information may be incomplete.
lsof: WARNING: can't stat() lustre file system /private/lustre/gadiscr1
      Output information may be incomplete.
lsof: WARNING: can't stat() lustre file system /private/lustre/gdata1b
      Output information may be incomplete.
lsof: WARNING: can't stat() lustre file system /private/lustre/gdata1a
      Output information may be incomplete.
lsof: WARNING: can't stat() lustre file system /private/lustre/gdata4
      Output information may be incomplete.
lsof: WARNING: can't stat() lustre file system /private/lustre/gdata6
      Output information may be incomplete.
lsof: WARNING: can't stat() lustre file system /private/lustre/gdata5
      Output information may be incomplete.
COMMAND     PID   USER   FD   TYPE DEVICE SIZE/OFF                NODE NAME
git     1243314 ms2335  cwd    DIR   0,56     4096 9264261886474905060 access-esm1.6_scaling_layouts/access-esm1.6-PI-config
git     1243314 ms2335  mem    REG   0,56   146196 9264261869287914135 access-esm1.6_scaling_layouts/access-esm1.6-PI-config/.git/objects/pack/.nfs80914516011a3a97000306db
git     1243413 ms2335  cwd    DIR   0,56     4096 9264261886474905060 access-esm1.6_scaling_layouts/access-esm1.6-PI-config
git     1243413 ms2335  mem    REG   0,56   146196 9264261869287914135 access-esm1.6_scaling_layouts/access-esm1.6-PI-config/.git/objects/pack/.nfs80914516011a3a97000306db
git     1243422 ms2335  cwd    DIR   0,56     4096 9264261886474905060 access-esm1.6_scaling_layouts/access-esm1.6-PI-config
git     1243422 ms2335  mem    REG   0,56   146196 9264261869287914135 access-esm1.6_scaling_layouts/access-esm1.6-PI-config/.git/objects/pack/.nfs80914516011a3a97000306db
git     1243705 ms2335  cwd    DIR   0,56     4096 9264261886474905060 access-esm1.6_scaling_layouts/access-esm1.6-PI-config
git     1243705 ms2335  mem    REG   0,56   146196 9264261869287914135 access-esm1.6_scaling_layouts/access-esm1.6-PI-config/.git/objects/pack/.nfs80914516011a3a97000306db
[~/codes/scaling_studies/access-models-scaling @gadi-login-04]$ cd access-esm1.6_scaling_layouts/access-esm1.6-PI-config
[~/codes/scaling_studies/access-models-scaling/access-esm1.6_scaling_layouts/access-esm1.6-PI-config @gadi-login-04]$ ll
total 12K
drwxr-x--- 3 ms2335 tm70 4.0K Oct 17 16:09 .
drwxr-x--- 3 ms2335 tm70 4.0K Oct 17 16:09 .git
drwxr-x--- 3 ms2335 tm70 4.0K Oct 17 16:06 ..
[~/codes/scaling_studies/access-models-scaling/access-esm1.6_scaling_layouts/access-esm1.6-PI-config @gadi-login-04]$ rm -Rf .git
rm: cannot remove '.git/objects/pack/.nfs80914516011a3a97000306db': Device or resource busy
[~/codes/scaling_studies/access-models-scaling/access-esm1.6_scaling_layouts/access-esm1.6-PI-config @gadi-login-04]$

In the output above, if I do kill -9 <pid> for all the listed pids, then I can remove the directory.