diffexp and diffsplice errors

[olneykimberly]

I can not get the diffexp or the diffsplice modules to run without error.
I have tried with both the conda environment and the Docker SIF. I keep getting similar errors of package not found or r2py compatibility issues. Yes, I made sure to update the conda env via conda env update --name flair --file https://github.com/BrooksLabUCSC/flair/releases/download/v2.2.0+master/flair_diffexp_conda_env.yaml

Error message with the conda version:

Traceback (most recent call last):
  File "/tgen_labs/jfryer/kolney/tools/miniconda3/envs/flair/lib/python3.12/site-packages/flair/runDS.py", line 24, in <module>
    from rpy2.robjects import pandas2ri
  File "/tgen_labs/jfryer/kolney/tools/miniconda3/envs/flair/lib/python3.12/site-packages/rpy2/robjects/pandas2ri.py", line 13, in <module>
    pandas2ri.activate()
    ^^^^^^^^^^^^^^^^^^
AttributeError: partially initialized module 'rpy2.robjects.pandas2ri' has no attribute 'activate' (most likely due to a circular import)
Traceback (most recent call last):
  File "/tgen_labs/jfryer/kolney/tools/miniconda3/envs/flair/lib/python3.12/site-packages/flair/runDS.py", line 24, in <module>
    from rpy2.robjects import pandas2ri
  File "/tgen_labs/jfryer/kolney/tools/miniconda3/envs/flair/lib/python3.12/site-packages/rpy2/robjects/pandas2ri.py", line 13, in <module>
    pandas2ri.activate()
    ^^^^^^^^^^^^^^^^^^
AttributeError: partially initialized module 'rpy2.robjects.pandas2ri' has no attribute 'activate' (most likely due to a circular import)
Traceback (most recent call last):
  File "/tgen_labs/jfryer/kolney/tools/miniconda3/envs/flair/lib/python3.12/site-packages/flair/runDS.py", line 24, in <module>
    from rpy2.robjects import pandas2ri
  File "/tgen_labs/jfryer/kolney/tools/miniconda3/envs/flair/lib/python3.12/site-packages/rpy2/robjects/pandas2ri.py", line 13, in <module>
    pandas2ri.activate()
    ^^^^^^^^^^^^^^^^^^
AttributeError: partially initialized module 'rpy2.robjects.pandas2ri' has no attribute 'activate' (most likely due to a circular import)
Traceback (most recent call last):
  File "/tgen_labs/jfryer/kolney/tools/miniconda3/envs/flair/lib/python3.12/site-packages/flair/runDS.py", line 24, in <module>
    from rpy2.robjects import pandas2ri
  File "/tgen_labs/jfryer/kolney/tools/miniconda3/envs/flair/lib/python3.12/site-packages/rpy2/robjects/pandas2ri.py", line 13, in <module>
    pandas2ri.activate()
    ^^^^^^^^^^^^^^^^^^
AttributeError: partially initialized module 'rpy2.robjects.pandas2ri' has no attribute 'activate' (most likely due to a circular import)

I even tried updating the runDE.py and runDU.py scripts, but the errors persist again. I even tried installing an older version of rpy2, but that had compatibility issues.

Error message with the docker version:
singularity exec docker://brookslab/flair:latest flair diffexp -q flair.quantify.counts.tsv --out_dir test_dir

running DESEQ2 genes_deseq2
input file: OMG_please_work/workdir/filtered_gene_counts_ds2.tsv
/usr/local/lib/python3.12/dist-packages/rpy2/robjects/lib/ggplot2.py:70: UserWarning: This was designed against ggplot2 versions starting with 3.3. but you have 3.5.2
  warnings.warn(
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/flair/runDE.py", line 244, in <module>
    main()
  File "/usr/local/lib/python3.12/dist-packages/flair/runDE.py", line 103, in main
    runDESeq(outdir, group1, group2, matrix, prefix, formula)
  File "/usr/local/lib/python3.12/dist-packages/flair/runDE.py", line 145, in runDESeq
    importr('DESeq2')
  File "/usr/local/lib/python3.12/dist-packages/rpy2/robjects/packages.py", line 472, in importr
    raise PackageNotInstalledError(
rpy2.robjects.packages.PackageNotInstalledError: The R package "DESeq2" is not installed.
Warning messages:
1:
In (function (package, help, pos = 2, lib.loc = NULL, character.only = FALSE,  :

 library ‘/usr/lib/R/site-library’ contains no packages
2:
In (function (package, help, pos = 2, lib.loc = NULL, character.only = FALSE,  :

 library ‘/usr/lib/R/site-library’ contains no packages
3:
In (function (package, help, pos = 2, lib.loc = NULL, character.only = FALSE,  :

 library ‘/usr/lib/R/site-library’ contains no packages
4:
In (function (package, help, pos = 2, lib.loc = NULL, character.only = FALSE,  :

 library ‘/usr/lib/R/site-library’ contains no packages
5:
In (function (package, help, pos = 2, lib.loc = NULL, character.only = FALSE,  :

 library ‘/usr/lib/R/site-library’ contains no packages
6:
In (function (package, help, pos = 2, lib.loc = NULL, character.only = FALSE,  :

 library ‘/usr/lib/R/site-library’ contains no packages
running DESEQ2 isoforms_deseq2
input file: OMG_please_work/workdir/filtered_iso_counts_ds2.tsv
/usr/local/lib/python3.12/dist-packages/rpy2/robjects/lib/ggplot2.py:70: UserWarning: This was designed against ggplot2 versions starting with 3.3. but you have 3.5.2
  warnings.warn(
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/flair/runDE.py", line 244, in <module>
    main()
  File "/usr/local/lib/python3.12/dist-packages/flair/runDE.py", line 103, in main
    runDESeq(outdir, group1, group2, matrix, prefix, formula)
  File "/usr/local/lib/python3.12/dist-packages/flair/runDE.py", line 145, in runDESeq
    importr('DESeq2')
  File "/usr/local/lib/python3.12/dist-packages/rpy2/robjects/packages.py", line 472, in importr
    raise PackageNotInstalledError(
rpy2.robjects.packages.PackageNotInstalledError: The R package "DESeq2" is not installed.
Warning messages:
1:
In (function (package, help, pos = 2, lib.loc = NULL, character.only = FALSE,  :

 library ‘/usr/lib/R/site-library’ contains no packages
2:
In (function (package, help, pos = 2, lib.loc = NULL, character.only = FALSE,  :

 library ‘/usr/lib/R/site-library’ contains no packages
3:
In (function (package, help, pos = 2, lib.loc = NULL, character.only = FALSE,  :

 library ‘/usr/lib/R/site-library’ contains no packages
4:
In (function (package, help, pos = 2, lib.loc = NULL, character.only = FALSE,  :

 library ‘/usr/lib/R/site-library’ contains no packages
5:
In (function (package, help, pos = 2, lib.loc = NULL, character.only = FALSE,  :

 library ‘/usr/lib/R/site-library’ contains no packages
6:
In (function (package, help, pos = 2, lib.loc = NULL, character.only = FALSE,  :

 library ‘/usr/lib/R/site-library’ contains no packages
running DRIMSEQ isoforms_drimseq
input file: OMG_please_work/workdir/filtered_iso_counts_drim.tsv
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/flair/runDU.py", line 188, in <module>
    main()
  File "/usr/local/lib/python3.12/dist-packages/flair/runDU.py", line 110, in main
    rundrimseq(outdir, group1, group2, matrix, prefix, formula, threads)
  File "/usr/local/lib/python3.12/dist-packages/flair/runDU.py", line 134, in rundrimseq
    importr('DRIMSeq')
  File "/usr/local/lib/python3.12/dist-packages/rpy2/robjects/packages.py", line 472, in importr
    raise PackageNotInstalledError(
rpy2.robjects.packages.PackageNotInstalledError: The R package "DRIMSeq" is not installed.
Warning messages:
1:
In (function (package, help, pos = 2, lib.loc = NULL, character.only = FALSE,  :

 library ‘/usr/lib/R/site-library’ contains no packages
2:
In (function (package, help, pos = 2, lib.loc = NULL, character.only = FALSE,  :

 library ‘/usr/lib/R/site-library’ contains no packages

Please help! I'm so close.

[diekhans]

This looks like the wrong version of rpy2 is being brought in. We have remove the dependencies on rpy2 and pandas in master as they caused various problems. We are getting very close to a new FLAIR release. If you install it into the conda environment with pip install git+https://github.com/BrooksLabUCSC/flair.git that should work, and we can fix quickly if you find problems.

[olneykimberly]

Thank you so much for the help and quick reply!!
I ran pip install git+https://github.com/BrooksLabUCSC/flair.git as suggested, and now I have a new set of errors:

flair diffexp -q flair.quantify.counts.tsv --out_dir out_dir2
/tgen_labs/jfryer/kolney/tools/miniconda3/envs/flair/lib/python3.12/site-packages/scipy/stats/_axis_nan_policy.py:579: RuntimeWarning: Precision loss occurred in moment calculation due to catastrophic cancellation. This occurs when the data are nearly identical. Results may be unreliable.
  res = hypotest_fun_out(*samples, **kwds)
WARNING: ignoring environment value of R_HOME
failure: Rscript /tgen_labs/jfryer/kolney/tools/miniconda3/envs/flair/lib/python3.12/site-packages/flair/diffExp_deseq2.R --group1 LBD --group2 CONTROL --batch SMRT3337 --matrix out_dir2/workdir/filtered_gene_counts_ds2.tsv --outDir out_dir2 --prefix genes_deseq2 --formula out_dir2/workdir/formula_matrix.tsv 2><_io.TextIOWrapper name='out_dir2/workdir/genes_deseq2.txt' mode='w' encoding='UTF-8'>
Traceback (most recent call last):
  File "/tgen_labs/jfryer/kolney/tools/miniconda3/envs/flair/lib/python3.12/site-packages/pipettor/processes.py", line 412, in _raise_if_failed
    raise p.procExcept
pipettor.exceptions.ProcessException: process exited 1: Rscript /tgen_labs/jfryer/kolney/tools/miniconda3/envs/flair/lib/python3.12/site-packages/flair/diffExp_deseq2.R --group1 LBD --group2 CONTROL --batch SMRT3337 --matrix out_dir2/workdir/filtered_gene_counts_ds2.tsv --outDir out_dir2 --prefix genes_deseq2 --formula out_dir2/workdir/formula_matrix.tsv
Traceback (most recent call last):
  File "/tgen_labs/jfryer/kolney/tools/miniconda3/envs/flair/lib/python3.12/site-packages/flair/flair_diffExp.py", line 303, in run_deseq2
    pipettor.run(["Rscript", diffExp_deseq2, "--group1", groups[0], "--group2", groups[-1],
  File "/tgen_labs/jfryer/kolney/tools/miniconda3/envs/flair/lib/python3.12/site-packages/pipettor/__init__.py", line 17, in run
    Pipeline(cmds, stdin=stdin, stdout=stdout, stderr=stderr, logger=logger, logLevel=logLevel).wait()
  File "/tgen_labs/jfryer/kolney/tools/miniconda3/envs/flair/lib/python3.12/site-packages/pipettor/processes.py", line 459, in wait
    self._wait_guts()
  File "/tgen_labs/jfryer/kolney/tools/miniconda3/envs/flair/lib/python3.12/site-packages/pipettor/processes.py", line 452, in _wait_guts
    self._raise_if_failed()
  File "/tgen_labs/jfryer/kolney/tools/miniconda3/envs/flair/lib/python3.12/site-packages/pipettor/processes.py", line 415, in _raise_if_failed
    raise ex
  File "/tgen_labs/jfryer/kolney/tools/miniconda3/envs/flair/lib/python3.12/site-packages/pipettor/processes.py", line 412, in _raise_if_failed
    raise p.procExcept
pipettor.exceptions.ProcessException: process exited 1: Rscript /tgen_labs/jfryer/kolney/tools/miniconda3/envs/flair/lib/python3.12/site-packages/flair/diffExp_deseq2.R --group1 LBD --group2 CONTROL --batch SMRT3337 --matrix out_dir2/workdir/filtered_gene_counts_ds2.tsv --outDir out_dir2 --prefix genes_deseq2 --formula out_dir2/workdir/formula_matrix.tsv

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/tgen_labs/jfryer/kolney/tools/miniconda3/envs/flair/lib/python3.12/site-packages/flair/flair_cli.py", line 85, in main
    flair_module_run(opts, args.module, args.module_args)
  File "/tgen_labs/jfryer/kolney/tools/miniconda3/envs/flair/lib/python3.12/site-packages/flair/flair_cli.py", line 73, in flair_module_run
    flair_diffExp.diffExp()
  File "/tgen_labs/jfryer/kolney/tools/miniconda3/envs/flair/lib/python3.12/site-packages/flair/flair_diffExp.py", line 423, in diffExp
    calculate_sig(args)
  File "/tgen_labs/jfryer/kolney/tools/miniconda3/envs/flair/lib/python3.12/site-packages/flair/flair_diffExp.py", line 391, in calculate_sig
    run_deseq2("genes_deseq2", workdir, groups, batches, geneMatrixFile, outDir, formulaMatrixFile)
  File "/tgen_labs/jfryer/kolney/tools/miniconda3/envs/flair/lib/python3.12/site-packages/flair/flair_diffExp.py", line 307, in run_deseq2
    raise FlairError(f'running {prefix} failed, please check {stderr} for details') from exc
flair.FlairError: running genes_deseq2 failed, please check out_dir2/workdir/genes_deseq2.txt for details

[olneykimberly]

I've tried everything at this point.

I'm working on an HPC environment, and we use R containers; that's a big reason why I'm having trouble getting it to work. Is it possible to have R and the dependencies in a container so I can just use that container to run flair diffexp?

[diekhans]

Can you just use the FLAIR docker container: docker pull brookslab/flair:2.2.0 it includes R. Kimberly Olney, Ph.D ***@***.***> writes:

…

olneykimberly left a comment (BrooksLabUCSC/flair#599) I've tried everything at this point. I'm working on an HPC environment, and we use R containers; that's a big reason why I'm having trouble getting it to work. Is it possible to have R and the dependencies in a container so I can just use that container to run flair diffexp? -- Reply to this email directly or view it on GitHub: #599 (comment) You are receiving this because you commented. Message ID: ***@***.***>

[olneykimberly]

I have already tried. We use singularity, and I applied the following:

singularity exec \
>     --cleanenv \
>     -B /home,/scratch,/tgen_labs,/tgen_labs/jfryer/kolney/LBD_CWOW/isoform_CWOW/flair_outputs/quantify:/data \
>     docker://brookslab/flair:2.2.0 \
>     flair diffexp -q flair.quantify.counts.nodash.tsv --out_dir out_dir_test
INFO:    Using cached SIF image
WARNING, running DESeq2 on genes failed, please check out_dir_test/workdir/dge_stderr.txt for details
WARNING, running DESeq2 on isoforms failed, please check out_dir_test/workdir/dge_stderr.txt for details
WARNING, running DRIMSeq failed, please check out_dir_test/workdir/dge_stderr.txt for details
Flair diffexp took 0 minutes and 11 seconds and finished without issues.

FLAIR HAS FINISHED

cat out_dir_test/workdir/dge_stderr.txt
running DESEQ2 genes_deseq2
input file: out_dir_test/workdir/filtered_gene_counts_ds2.tsv
/usr/local/lib/python3.12/dist-packages/rpy2/robjects/lib/ggplot2.py:70: UserWarning: This was designed against ggplot2 versions starting with 3.3. but you have 3.5.2
  warnings.warn(
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/flair/runDE.py", line 244, in <module>
    main()
  File "/usr/local/lib/python3.12/dist-packages/flair/runDE.py", line 103, in main
    runDESeq(outdir, group1, group2, matrix, prefix, formula)
  File "/usr/local/lib/python3.12/dist-packages/flair/runDE.py", line 145, in runDESeq
    importr('DESeq2')
  File "/usr/local/lib/python3.12/dist-packages/rpy2/robjects/packages.py", line 472, in importr
    raise PackageNotInstalledError(
rpy2.robjects.packages.PackageNotInstalledError: The R package "DESeq2" is not installed.
Warning messages:
1:
In (function (package, help, pos = 2, lib.loc = NULL, character.only = FALSE,  :

 library ‘/usr/lib/R/site-library’ contains no packages
2:
In (function (package, help, pos = 2, lib.loc = NULL, character.only = FALSE,  :

 library ‘/usr/lib/R/site-library’ contains no packages
3:
In (function (package, help, pos = 2, lib.loc = NULL, character.only = FALSE,  :

 library ‘/usr/lib/R/site-library’ contains no packages
4:
In (function (package, help, pos = 2, lib.loc = NULL, character.only = FALSE,  :

 library ‘/usr/lib/R/site-library’ contains no packages
5:
In (function (package, help, pos = 2, lib.loc = NULL, character.only = FALSE,  :

 library ‘/usr/lib/R/site-library’ contains no packages
6:
In (function (package, help, pos = 2, lib.loc = NULL, character.only = FALSE,  :

 library ‘/usr/lib/R/site-library’ contains no packages
running DESEQ2 isoforms_deseq2
input file: out_dir_test/workdir/filtered_iso_counts_ds2.tsv
/usr/local/lib/python3.12/dist-packages/rpy2/robjects/lib/ggplot2.py:70: UserWarning: This was designed against ggplot2 versions starting with 3.3. but you have 3.5.2
  warnings.warn(
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/flair/runDE.py", line 244, in <module>
    main()
  File "/usr/local/lib/python3.12/dist-packages/flair/runDE.py", line 103, in main
    runDESeq(outdir, group1, group2, matrix, prefix, formula)
  File "/usr/local/lib/python3.12/dist-packages/flair/runDE.py", line 145, in runDESeq
    importr('DESeq2')
  File "/usr/local/lib/python3.12/dist-packages/rpy2/robjects/packages.py", line 472, in importr
    raise PackageNotInstalledError(
rpy2.robjects.packages.PackageNotInstalledError: The R package "DESeq2" is not installed.
Warning messages:
1:
In (function (package, help, pos = 2, lib.loc = NULL, character.only = FALSE,  :

 library ‘/usr/lib/R/site-library’ contains no packages
2:
In (function (package, help, pos = 2, lib.loc = NULL, character.only = FALSE,  :

 library ‘/usr/lib/R/site-library’ contains no packages
3:
In (function (package, help, pos = 2, lib.loc = NULL, character.only = FALSE,  :

 library ‘/usr/lib/R/site-library’ contains no packages
4:
In (function (package, help, pos = 2, lib.loc = NULL, character.only = FALSE,  :

 library ‘/usr/lib/R/site-library’ contains no packages
5:
In (function (package, help, pos = 2, lib.loc = NULL, character.only = FALSE,  :

 library ‘/usr/lib/R/site-library’ contains no packages
6:
In (function (package, help, pos = 2, lib.loc = NULL, character.only = FALSE,  :

 library ‘/usr/lib/R/site-library’ contains no packages
running DRIMSEQ isoforms_drimseq
input file: out_dir_test/workdir/filtered_iso_counts_drim.tsv
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/flair/runDU.py", line 188, in <module>
    main()
  File "/usr/local/lib/python3.12/dist-packages/flair/runDU.py", line 110, in main
    rundrimseq(outdir, group1, group2, matrix, prefix, formula, threads)
  File "/usr/local/lib/python3.12/dist-packages/flair/runDU.py", line 134, in rundrimseq
    importr('DRIMSeq')
  File "/usr/local/lib/python3.12/dist-packages/rpy2/robjects/packages.py", line 472, in importr
    raise PackageNotInstalledError(
rpy2.robjects.packages.PackageNotInstalledError: The R package "DRIMSeq" is not installed.
Warning messages:
1:
In (function (package, help, pos = 2, lib.loc = NULL, character.only = FALSE,  :

 library ‘/usr/lib/R/site-library’ contains no packages
2:
In (function (package, help, pos = 2, lib.loc = NULL, character.only = FALSE,  :

 library ‘/usr/lib/R/site-library’ contains no packages

[diekhans]

I am going to build a container with master branch for you. We need to test this for the next release, so you can be our tester.

[diekhans]

Hi Kimberly, We just published a Docker of an alpha release of the upcoming version of flair. Please give it a try and lets us know how it goes: brookslab/flair:3.0.0a1

[olneykimberly]

Praise be!! It works!! Thank you, thank you!

[diekhans]

Thank you for testing it; that was a big help! Kimberly Olney, Ph.D ***@***.***> writes:

…

olneykimberly left a comment (BrooksLabUCSC/flair#599) Praise be!! It works!! Thank you, thank you! -- Reply to this email directly or view it on GitHub: #599 (comment) You are receiving this because you modified the open/close state. Message ID: ***@***.***>