How to set tmpDir of pbalign when running fc_quiver

[phillip-mcclurg-driscolls]

Note: Forgive if this is a duplication, but I do not wee my previous issue listed after submission:

I am running the test_case by Greg Conception - I have run fc_unzip.py fc_unzip.cfg successfuly.

When running fc_quiver.py fc_unzip.py the execution stops because pbalign uses the default tmpDir /scratch - On our machine this is on a small local disk which will overload on a real case. Is there a way to reconfigure pbalign to change the tmpDir parameter in this case?

The only difference between my fc_unzip.cfg file and Greg's is that I've added a line in the [General] section:

use_tmpdir = /archive/jenkins_test/pmcclurg/tmp

which points to a much larger disk. Upon running

fc_quiver.py fc_unzip.cfg here is the output of the .stderr file for job 000000F:

touch {output.job_done}

samtools faidx /archive/jenkins_test/pmcclurg/FALCON_unzip/src/FALCON-integrate/FALCON-examples/run/greg200k-sv2/4-quiver/quiver_scatter/refs/000000F/ref.fa

• samtools faidx /archive/jenkins_test/pmcclurg/FALCON_unzip/src/FALCON-integrate/FALCON-examples/run/greg200k-sv2/4-quiver/quiver_scatter/refs/000000F/ref.fa
  pbalign --tmpDir=/scratch/ --nproc=24 --minAccuracy=0.75 --minLength=50 --minAnchorSize=12 --maxDivergence=30 --concordant --algorithm=blasr --algorithmOptions=--useQuality --maxHits=1 --hitPolicy=random --seed=1 /archive/jenkins_test/pmcclurg/FALCON_unzip/src/FALCON-integrate/FALCON-examples/run/greg200k-sv2/4-quiver/segregate_scatter/segregated/000000F/000000F.bam /archive/jenkins_test/pmcclurg/FALCON_unzip/src/FALCON-integrate/FALCON-examples/run/greg200k-sv2/4-quiver/quiver_scatter/refs/000000F/ref.fa aln-000000F.bam
• pbalign --tmpDir=/scratch/ --nproc=24 --minAccuracy=0.75 --minLength=50 --minAnchorSize=12 --maxDivergence=30 --concordant --algorithm=blasr --algorithmOptions=--useQuality --maxHits=1 --hitPolicy=random --seed=1 /archive/jenkins_test/pmcclurg/FALCON_unzip/src/FALCON-integrate/FALCON-examples/run/greg200k-sv2/4-quiver/segregate_scatter/segregated/000000F/000000F.bam /archive/jenkins_test/pmcclurg/FALCON_unzip/src/FALCON-integrate/FALCON-examples/run/greg200k-sv2/4-quiver/quiver_scatter/refs/000000F/ref.fa aln-000000F.bam
  [Errno 13] Permission denied: '/scratch/tmp7ZwBg8'
  Traceback (most recent call last):
  File "/archive/jenkins_test/pmcclurg/FALCON_unzip/fc_env_180607/lib/python2.7/site-packages/pbcommand/cli/core.py", line 137, in _pacbio_main_runner
  return_code = exe_main_func(*args, **kwargs)
  File "/archive/jenkins_test/pmcclurg/FALCON_unzip/fc_env_180607/lib/python2.7/site-packages/pbalign/pbalignrunner.py", line 284, in args_runner
  return PBAlignRunner(args, output_dataset_type=output_dataset_type).start()
  File "/archive/jenkins_test/pmcclurg/FALCON_unzip/fc_env_180607/lib/python2.7/site-packages/pbcore/util/ToolRunner.py", line 85, in start
  return self.run()
  File "/archive/jenkins_test/pmcclurg/FALCON_unzip/fc_env_180607/lib/python2.7/site-packages/pbalign/pbalignrunner.py", line 234, in run
  self._tempFileManager)
  File "/archive/jenkins_test/pmcclurg/FALCON_unzip/fc_env_180607/lib/python2.7/site-packages/pbalign/pbalignrunner.py", line 122, in _createAlignService
  service = BlasrService(args, fileNames, tempFileManager)
  File "/archive/jenkins_test/pmcclurg/FALCON_unzip/fc_env_180607/lib/python2.7/site-packages/pbalign/alignservice/align.py", line 122, in init
  self._tempFileManager.SetRootDir(self._options.tmpDir)
  File "/archive/jenkins_test/pmcclurg/FALCON_unzip/fc_env_180607/lib/python2.7/site-packages/pbalign/utils/tempfileutil.py", line 91, in SetRootDir
  rootDir = tempfile.mkdtemp(dir=rootDir)
  File "/archive/jenkins_test/pmcclurg/Python-2.7.9/lib/python2.7/tempfile.py", line 333, in mkdtemp
  _os.mkdir(file, 0700)
  OSError: [Errno 13] Permission denied: '/scratch/tmp7ZwBg8'
  touch quiver_done.exit
• touch quiver_done.exit
  2018-06-26 09:33:08,436 - root - WARNING - Call '/bin/bash user_script.sh' returned 512.
  2018-06-26 09:33:08,436 - root - WARNING - CD: '/archive/jenkins_test/pmcclurg/FALCON_unzip/src/FALCON-integrate/FALCON-examples/run/greg200k-sv2/4-quiver/quiver_run/000000F' -> '/archive/jenkins_test/pmcclurg/FALCON_unzip/src/FALCON-integrate/FALCON-examples/run/greg200k-sv2/4-quiver/quiver_run/000000F'
  2018-06-26 09:33:08,436 - root - WARNING - CD: '/archive/jenkins_test/pmcclurg/FALCON_unzip/src/FALCON-integrate/FALCON-examples/run/greg200k-sv2/4-quiver/quiver_run/000000F' -> '/archive/jenkins_test/pmcclurg/FALCON_unzip/src/FALCON-integrate/FALCON-examples/run/greg200k-sv2/4-quiver/quiver_run/000000F'
  2018-06-26 09:33:08,436 - root - CRITICAL - Error in /archive/jenkins_test/pmcclurg/FALCON_unzip/fc_env_180607/lib/python2.7/site-packages/pypeflow/do_task.py with args="{'json_fn': '/archive/jenkins_test/pmcclurg/FALCON_unzip/src/FALCON-integrate/FALCON-examples/run/greg200k-sv2/4-quiver/quiver_run/000000F/task.json',\n 'timeout': 30,\n 'tmpdir': None}"
  Traceback (most recent call last):
  File "/archive/jenkins_test/pmcclurg/Python-2.7.9/lib/python2.7/runpy.py", line 162, in _run_module_as_main
  "main", fname, loader, pkg_name)
  File "/archive/jenkins_test/pmcclurg/Python-2.7.9/lib/python2.7/runpy.py", line 72, in _run_code
  exec code in run_globals
  File "/archive/jenkins_test/pmcclurg/FALCON_unzip/fc_env_180607/lib/python2.7/site-packages/pypeflow/do_task.py", line 250, in
  main()
  File "/archive/jenkins_test/pmcclurg/FALCON_unzip/fc_env_180607/lib/python2.7/site-packages/pypeflow/do_task.py", line 242, in main
  run(**vars(parsed_args))
  File "/archive/jenkins_test/pmcclurg/FALCON_unzip/fc_env_180607/lib/python2.7/site-packages/pypeflow/do_task.py", line 236, in run
  run_cfg_in_tmpdir(cfg, tmpdir)
  File "/archive/jenkins_test/pmcclurg/FALCON_unzip/fc_env_180607/lib/python2.7/site-packages/pypeflow/do_task.py", line 212, in run_cfg_in_tmpdir
  run_bash(bash_template, myinputs, myoutputs, parameters)
  File "/archive/jenkins_test/pmcclurg/FALCON_unzip/fc_env_180607/lib/python2.7/site-packages/pypeflow/do_task.py", line 173, in run_bash
  util.system(cmd)
  File "/archive/jenkins_test/pmcclurg/FALCON_unzip/fc_env_180607/lib/python2.7/site-packages/pypeflow/io.py", line 29, in syscall
  raise Exception(msg)
  Exception: Call '/bin/bash user_script.sh' returned 512.

real 0m0.892s
user 0m1.787s
sys 0m5.446s

• finish
• echo 'finish code: 1'

[pb-cdunn]

In the next tarball (a day or two), use_tmpdir should fix this.

[pb-cdunn]

(This is really an issue Falcon_unzip, btw.)