PDM incorrectly uses top-level virtualenv from within a nox session

[Amar1729]

• I have searched the issue tracker and believe that this is not a duplicate.
  • i think this issue is too old to be relevant, though its title sounds similar pdm installs packages although nox re-uses virtualenv #346

Make sure you run commands with -v flag before pasting the output.

Steps to reproduce

Unfortunately I can't seem to reproduce this bug from a completely clean project generated by pdm init, and I can't paste in all the files here as some of them are sensitive. I'll paste in what I can below. What's happening is that nox says it creates a new virtualenv under .nox/session-name (in this case mypy), but pdm does not reuse that as it generally would. Instead, it uses the top-level repository's .venv, then fails to find the command-line bin mypy.

/tmp/minimal-pdm-repro $ nox --list                                                                                                            ● main | 1

Sessions defined in /private/tmp/minimal-pdm-repro/noxfile.py:

* mypy -> Run mypy type-checker.
* ruff -> Run ruff linter. User can (and should) fix errors with `ruff --fix` as suggested.
* darglint -> Run darglint linter for checking docstrings.
* docs -> Build the documentation.

sessions marked with * are selected, sessions marked with - are skipped.

/tmp/minimal-pdm-repro $  nox -s mypy                                                                                                              ● main

nox > Running session mypy
nox > Creating virtual environment (virtualenv) using python in .nox/mypy
nox > pdm install --no-self --no-default -dG lint
python.use_venv is on, creating a virtualenv for this project...
Virtualenv is created successfully at /private/tmp/minimal-pdm-repro/.venv
Synchronizing working set with resolved packages: 6 to add, 0 to update, 0 to remove

  ✔ Install mypy-extensions 1.0.0 successful
  ✔ Install tomli 2.0.1 successful
  ✔ Install typing-extensions 4.8.0 successful
  ✔ Install darglint 1.8.1 successful
  ✔ Install ruff 0.0.292 successful
  ✔ Install mypy 1.5.1 successful

🎉 All complete!

nox > Program mypy not found.
nox > Session mypy failed.

Relevant snippets from files:

# noxfile.py
import os

# installed globally
import nox

os.environ.update({"PDM_IGNORE_SAVED_PYTHON": "1"})

@nox.session
def mypy(session):
    """Run mypy type-checker."""
    session.run_always("pdm", "install", "--no-self", "--no-default", "-dG", "lint", external=True)
    session.run("mypy", "--non-interactive", "--install-types", "src/", "tests/")

# noxfile has some other sessions defined, including ruff, darglint, docs, and tests.

# pyproject.toml
[project]
name = "minimal-pdm-repro"
version = "0.2.0"
description = "minimal PDM venv bug repro"
authors = [
    {name = "amar", email = "nowhere@example.com"},
]
readme = "README.md"
requires-python = ">=3.8,<3.10"

dependencies = []

[project.optional-dependencies]
test = [
    "pytest-cov>=4.1.0",
]

[tool.pdm.dev-dependencies]
lint = [
    "ruff>=0.0.292",
    "mypy>=1.5.1",
    "darglint>=1.8.1",
]
doc = [
    "Sphinx>=7.1.2",
    "sphinx-autodoc-typehints>=1.24.0",
]

[build-system]
requires = ["setuptools>=61, wheel"]
build-backend = "setuptools.build_meta"

# pdm.toml

# set by `pdm config --local python.use_venv true`
[python]
use_venv = true

Actual behavior

I've followed the docs on using nox with pdm, and seemingly have my noxfile.py configured properly.

However, pdm seems to only want to re-use the repo's top-level .venv, even though nox says it can reuse it. I've cleaned the repo multiple times (rm -r .venv .nox) but successive runs always create nox's virtualenvs, but only install deps to /.venv rather than /.nox/mypy.

Note in the paste above, the following line:

Virtualenv is created successfully at /private/tmp/minimal-pdm-repro/.venv

Expected behavior

In other projects, this is what happens:

/tmp/minimal-2 $  nox                                                                                                                                main

nox > Running session mypy
nox > Creating virtual environment (virtualenv) using python in .nox/mypy
nox > pdm install --no-self --no-default -dG lint
Inside an active virtualenv /private/tmp/minimal-2/.nox/mypy, reusing it.
Set env var PDM_IGNORE_ACTIVE_VENV to ignore it.
Synchronizing working set with resolved packages: 3 to add, 0 to update, 0 to remove

  ✔ Install mypy-extensions 1.0.0 successful
  ✔ Install typing-extensions 4.8.0 successful
  ✔ Install mypy 1.5.1 successful

🎉 All complete!

nox > mypy --non-interactive --install-types src/ tests/
Success: no issues found in 2 source files
nox > Session mypy was successful.

Note the Inside an active virtualenv .../.nox/mypy

Environment Information

# nox is provided by pipx
$ nox --version

2023.4.22

# python3 is provided by `pyenv` as 3.8
$ python3 -V

Python 3.8.18

# Paste the output of `pdm info && pdm info --env` below:
PDM version:
  2.9.3
Python Interpreter:
  /private/tmp/minimal-pdm-repro/.venv/bin/python (3.8)
Project Root:
  /private/tmp/minimal-pdm-repro
Local Packages:

{
  "implementation_name": "cpython",
  "implementation_version": "3.8.18",
  "os_name": "posix",
  "platform_machine": "arm64",
  "platform_release": "22.6.0",
  "platform_system": "Darwin",
  "platform_version": "Darwin Kernel Version 22.6.0: Fri Sep 15 13:41:30 PDT 2023; root:xnu-8796.141.3.700.8~1/RELEASE_ARM64_T8103",
  "python_full_version": "3.8.18",
  "platform_python_implementation": "CPython",
  "python_version": "3.8",
  "sys_platform": "darwin"
}

Let me know what other info I can provide. Not sure of the best way to go about debugging this.

[frostming]

In other projects, this is what happens:

What makes this project different from other projects?

[nejcskofic]

I had a similar issue, though my use case is a bit different. I'm using conda for virtual environment creation inside nox (to test on different python versions), but I'm using venv for development. When nox creates conda environment it does not clear VIRTUAL_ENV environment variable. When pdm is trying to resolve interpreter to use, it first looks for the VIRTUAL_ENV and after that for CONDA_PREFIX (code). Removing VIRTUAL_ENV before invoking install resolved my issue.

@Amar1729 I would take a look at environment variables that nox is passing to install. Those can be accessed via session.env inside your mypy function. Specifically you are interested in value of VIRTUAL_ENV. By the output you provided it looks like this is either missing or empty. If that is the case it may be nox issue. Try updating it to the latest version or report the issue there.