find_on_path distribution finder can cause "maximum recursion depth exceeded in comparison"

[cjerdonek]

A few days ago someone filed an issue on pip's tracker that looks like a setuptools issue:
pypa/pip#6288

Basically, it looks like the issue is that find_on_path() (the "distribution finder" corresponding to pkgutil.ImpImporter and FileFinder) can get into a situation where it's calling itself in an (infinite) recursive loop. Also, I believe this is happening in the find_on_path(only=False) case, rather than only=True.

[cjerdonek]

I wonder if this could be caused by a path visited by find_on_path() resolving as a symlink to a path higher in the directory hierarchy already visited by find_on_path(). It looks like find_on_path() does resolve symlinks here in the first line (the _normalize_cached() function does call os.path.realpath()):

setuptools/pkg_resources/__init__.py

Lines 2018 to 2020 in 702f2d9

	def find_on_path(importer, path_item, only=False):
	"""Yield distributions accessible on a sys.path directory"""
	path_item = _normalize_cached(path_item)

[Rajskc]

Hey @pganssle . Can you please let me know what can be done to solve this bug?

[pganssle]

@Rajskc I'm not actually sure. If either you or @cjerdonek can provide a minimal test case that reproduces the issue, it would be a good start towards solving the issue. Even better if you could make a PR with a failing test case (marked as xfail).

[cjerdonek]

@Rajskc Are you still experiencing the problem on your system? One thing that would be helpful is if you could do some debugging, for example by inserting some print statements to find out what paths it's getting stuck in an infinite loop on. It could also help to know some of the other values of the function arguments in the infinite loop.

Let me know if you need any suggestions for what to print out.

[Rajskc]

@cjerdonek Any suggestion would be very helpful as I am not quite familiar with this stuff.

[Rajskc]

@cjerdonek Hey! I haven't been able to work on this till now. Can you please help me here. I understand that it is getting stuck the loop below, does increasing the sys.setrecursionlimit to 1500 help here and is it a good method to solve this.

File "c:\users\m phanidhar\anaconda3\lib\site-packages\pip\_vendor\pkg_resources\__init__.py", line 1968, in find_on_path
    for dist in factory(fullpath):

[Previous line repeated 960 more times]

[cjerdonek]

I wouldn't recommend increasing the recursion limit..

[Rajskc]

Okay I will create a PR as suggested above.

[Rajskc]

hey @pganssle I am trying to create a pull request as you suggested. But the code is different in my system compared to the gitversion here. I have installed the latest setup tools version using pip. What can i do to solve this?

[pganssle]

@Rajskc At the moment, I think we're still blocked on reproducing the problem. The first step is preparing a minimal working example of a way for someone else to reproduce the problem. Here is a blog post on crafting minimal reports, and here's Stack Overflow's guide. Even if you already have the solution, I have found that most of the time I spend working on PRs that are eventually merged is spent writing tests, and tests are basically MWEs that run automatically.

When you are ready to make a PR, you should be working in the git repository, so you would need to fork the repo, then clone it locally, then make changes in a new branch. Here is a document on how to make a PR. You don't necessarily have to install the git version on your local system, because if you have added a test, you can install and run tox from the local environment.

For more information about the specifics of contributing to setuptools, see our contributing documentation (which I realize could be better for beginners).

Thanks for your interest in helping with this. Please note that the most important thing is submitting a MWE, so that even if you get frustrated with submitting a PR, someone else can pick up where you left off.

[tronghieu1987]

Hello,

I am experiencing this issue and desperately searching for a solution. Is there any progress on this? Or could I do anything to reproduce the error?

I saw a MWE was asked for, but I am not sure what a "reproducing this problem" actually means. I run into this error when doing 'pip install ' with any package.

Thank you,

[jaraco]

I am not sure what a "reproducing this problem" actually means. I run into this error when doing 'pip install ' with any package.

The problem is, when the maintainers here run pip install, we don't encounter the issue... so we're kinda stuck investigating. I've tried running pip install service-identity on my Windows machine and it completes just fine. This all suggests to me there's another issue at play here. We need to figure out what that is. Maybe you have symlinks installed. Maybe you're in a directory that has settings configured. Maybe you have environment variables set. Maybe you're behind a weird firewall. Maybe you're running on an unprivileged account. We need to eliminate all these variables and hone in on what are the factors that lead to the failure you've encountered. Now that we have two people encountering the issue, perhaps we could compare environments to ascertain what things might be in common.

cjerdonek has done some investigation trying to imagine what might be going wrong, but we don't have a good way to confirm (or to ensure this doesn't happen to someone else).

For next steps, I recommend trying another machine. If you're on a work machine, find a personal machine (or vice versa) to try the same operation. Or try installing a clean version of Python (maybe try 3.7 instead of 3.6). Or find a friend and see if they encounter the same problem. Alternately, install a clean version of Windows into a VM and try that. Or even better, if you can reproduce the issue outside of Windows, that will vastly expand the audience of maintainers that will be able to help track down the issue.

Failing all of that, I recommend digging into the code when the failure occurs (add print statements or use pdb to break into the code and inspect the state. You'll need to study the code enough to understand what it's trying to do to know when the behavior diverges from that expectation.

Best of luck.