to cherrypick upstream `petsc` commits

[justinh2002]

See PR made by @justinh2002 to Upstream spack:

spack#50606

Relevant commit hash to be cherry picked into our spack repo:
commit 0a487df

[harshula]

Hi @justinh2002 , Can you please test branch: https://github.com/ACCESS-NRI/spack/tree/test-v0.22

[justinh2002]

jh7060@gadi-login-01:/g/data/au88/jh7060/spack/0.22/spack>spack install petsc ~examples
[+] /usr (external glibc-2.28-rmko4qyyetyn6neqs7tnuusqbllgwbnb)
[+] /usr (external glibc-2.28-i5vakeet5euon7ceaixzrpekyparsfxy)
==> openmpi@4.1.7 : has external module in ['openmpi/4.1.7']
[+] /apps/openmpi/4.1.7 (external openmpi-4.1.7-izun53dz3tkla25fxctopj2nzmeb3lw3)
==> python@3.9.2 : has external module in ['python3/3.9.2']
[+] /apps/python3/3.9.2 (external python-3.9.2-mgf3zv4w3xecoskhjwtnha3flmcbl4br)
[+] /g/data/au88/jh7060/spack/0.22/release/linux-rocky8-x86_64_v4/gcc-14.1.0/gcc-runtime-14.1.0-qukvuocm7w3d226pa4tnari6lgx4daoa
[+] /g/data/au88/jh7060/spack/0.22/release/linux-rocky8-x86_64_v4/gcc-13.2.0/gcc-runtime-13.2.0-tmmmxjmtktk4miyqzxusmdv7gojrg2sm
[+] /g/data/au88/jh7060/spack/0.22/release/linux-rocky8-x86_64_v4/gcc-14.1.0/gmake-4.4.1-ex2epk42mxz2coib6coiqapoztkdgrhg
[+] /g/data/au88/jh7060/spack/0.22/release/linux-rocky8-x86_64_v4/gcc-13.2.0/pkgconf-2.2.0-k24zmr2kgxzm6lckmtequjnshjbn2ehi
[+] /g/data/au88/jh7060/spack/0.22/release/linux-rocky8-x86_64_v4/gcc-13.2.0/zlib-ng-2.1.6-csssi246q4qg6ln76dgt5y4tnochai2a
[+] /g/data/au88/jh7060/spack/0.22/release/linux-rocky8-x86_64_v4/gcc-13.2.0/metis-5.1.0-ex3vgvt76s7gasqykngobxlpft7kodmg
[+] /g/data/au88/jh7060/spack/0.22/release/linux-rocky8-x86_64_v4/gcc-13.2.0/openblas-0.3.26-fb2vqjju3kd3cb6c5b5gyr4dbk527jqb
[+] /g/data/au88/jh7060/spack/0.22/release/linux-rocky8-x86_64_v4/gcc-14.1.0/libiconv-1.17-mfqg4idiarswd25zinw6ijl5yf4xi46t
[+] /g/data/au88/jh7060/spack/0.22/release/linux-rocky8-x86_64_v4/gcc-13.2.0/hdf5-1.14.3-pgpzqc5s5tpnnyonxdns5o7kxmekxjcg
[+] /g/data/au88/jh7060/spack/0.22/release/linux-rocky8-x86_64_v4/gcc-13.2.0/parmetis-4.0.3-4prinzwtc4intj72c7ur7fybgojgwagb
[+] /g/data/au88/jh7060/spack/0.22/release/linux-rocky8-x86_64_v4/gcc-13.2.0/hypre-2.31.0-kdkhaopbwvnbmoda2wzv2oym3i4hpwq6
[+] /g/data/au88/jh7060/spack/0.22/release/linux-rocky8-x86_64_v4/gcc-14.1.0/diffutils-3.10-pifxkotuodbeifp6unhodei4vpmkdgct
[+] /g/data/au88/jh7060/spack/0.22/release/linux-rocky8-x86_64_v4/gcc-13.2.0/superlu-dist-8.2.1-am46skic2blnkvs7zg7zt4ccv7vbu5b4
==> Installing petsc-3.21.1-zszggca2r4rqs4liaqgrhkqnyoavhcyw [18/18]
==> No binary for petsc-3.21.1-zszggca2r4rqs4liaqgrhkqnyoavhcyw found: installing from source
/apps/python3/3.9.2/bin/python3.9: error while loading shared libraries: libpython3.9.so.1.0: cannot open shared object file: No such file or directory
==> Using cached archive: /g/data/au88/jh7060/spack/0.22/sourcecache/_source-cache/archive/7f/7ff8b692bceb7d7a8f51e2f45ccb20af00ba9395d7e1eee8816d46eb1c4c4b27.tar.gz
==> No patches needed for petsc
==> petsc: Executing phase: 'configure'
==> petsc: Executing phase: 'build'
==> petsc: Executing phase: 'install'
==> petsc: Successfully installed petsc-3.21.1-zszggca2r4rqs4liaqgrhkqnyoavhcyw
  Stage: 11.53s.  Configure: 5m 0.37s.  Build: 1m 43.08s.  Install: 56.23s.  Post-install: 13.32s.  Total: 8m 17.65s
[+] /g/data/au88/jh7060/spack/0.22/release/linux-rocky8-x86_64_v4/gcc-14.1.0/petsc-3.21.1-zszggca2r4rqs4liaqgrhkqnyoavhcyw

[justinh2002]

documented issue here: spack#50506 , now manually checking the examples directory, install-lib target the 8k files

jh7060@gadi-login-01:/g/data/au88/jh7060/spack/0.22/spack>du -sh $(spack location -i petsc@3.21.1%gcc@14.1.0 arch=linux-rocky8-x86_64_v4)/share/petsc/examples
168K	
/g/data/au88/jh7060/spack/0.22/release/linux-rocky8-x86_64_v4/gcc-14.1.0/petsc-3.21.1-zszggca2r4rqs4liaqgrhkqnyoavhcyw/share/petsc/examples
jh7060@gadi-login-01:/g/data/au88/jh7060/spack/0.22/spack>find $(spack location -i petsc@3.21.1%gcc@14.1.0 arch=linux-rocky8-x86_64_v4)/share/petsc/examples | wc -l   
11