Poetry based on pyproject.toml installing from hpobench development branch fails due to yahpo-gym subdir

[LGro]

With the following pyproject.toml in a Python 3.9 environment with Poetry version 1.2.0:

[tool.poetry]
name = "hpobench-yahpo-failure"
version = "0.1.0"
description = "Example repo to demonstrate setup error with hpobench yahpo integration."
authors = ["Grossberger Lukas <Lukas.Grossberger@mail.local>"]
readme = "README.md"
packages = [{include = "hpobench_yahpo_failure"}]

[tool.poetry.dependencies]
python = "^3.8"
hpobench = {git = "https://github.com/automl/HPOBench.git", branch = "development"}


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

the following error occurs:

Unable to determine package info for path: /tmp/.venv/src/yahpo_gym

Command ['/tmp/tmpsrspcer5/.venv/bin/python', '-'] errored with the following return code 1, and output: 
Traceback (most recent call last):
  File "<stdin>", line 9, in <module>
  File "/tmp/tmpsrspcer5/.venv/lib/python3.9/site-packages/build/__init__.py", line 208, in __init__
    _validate_source_directory(srcdir)
  File "/tmp/tmpsrspcer5/.venv/lib/python3.9/site-packages/build/__init__.py", line 109, in _validate_source_directory
    raise BuildException(f'Source {srcdir} does not appear to be a Python project: no pyproject.toml or setup.py')
build.BuildException: Source /tmp/.venv/src/yahpo_gym does not appear to be a Python project: no pyproject.toml or setup.py
input was : import build
import build.env
import pep517

source = '/tmp/.venv/src/yahpo_gym'
dest = '/tmp/tmpsrspcer5/dist'

with build.env.IsolatedEnvBuilder() as env:
    builder = build.ProjectBuilder(
        srcdir=source,
        scripts_dir=env.scripts_dir,
        python_executable=env.executable,
        runner=pep517.quiet_subprocess_runner,
    )
    env.install(builder.build_system_requires)
    env.install(builder.get_requires_for_build('wheel'))
    builder.metadata_path(dest)

No fallback setup.py file was found to generate egg_info.

which I don't get when just using vanilla pip to

pip install git+https://github.com/automl/HPOBench.git@development

This makes me think that poetry might miss the subdirectory specification in the hpobench dependencies and I'm wondering if it might be viable to switch in hpobench to the wheel from pypi.org instead of installing the yahpo-gym dependency from source in hpobench.

[LGro]

The issue might also go away with the release of python-poetry/poetry/pull/5172
EDIT: Still seems to be an issue with Poetry 1.2.1

[PhMueller]

Hi,

thanks for your feedback!
Unfortunately, I dont have any experience with poetry.
Could you provide me the commands you use to install hpobench?

Sure, i will check if we can install yahpo_gym also per wheel. Fyi, yahpo_gymwas moved to https://github.com/slds-lmu/yahpo_gym.
Therefore, the link in the dependencies ( extra_requirements/yahpo_gym.json) has changed to

{
  "yahpo_gym": ["yahpo_gym@git+https://github.com/slds-lmu/yahpo_gym@9ec63dddcdfc4ba9d6084a9fd17dcd63a1386233#egg=yahpo_gym&subdirectory=yahpo_gym"],
}

Maybe this already solves the issue?

One thing that makes me curious:
Why does poetry try to install yahpo_gym at all? Actually, you don't have to install the extra dependencies to use the benchmark.

Since each benchmark is located in a container (that is the main advantage of your package, 😉) the following should already work:

from hpobench.container.benchmarks.surrogates.yahpo_gym import YAHPOGymMOBenchmark 

If you could describe me your usecase a little bit more, i might be able to provide better support :-)

Cheers, Philipp

[LGro]

Hi, apologies for the lackluster reproduction instructions; here is another take ☺️

1. Set up a Python 3.9 environment (e.g. with Anaconda) and activate it
2. pip install poetry==1.2.1
3. Create a new directory
   • place the above mentioned content inside a file called pyproject.toml
   • create a package directory hpobench_yahpo_failure that contains an empty __init__.py
   • you should now be inside that directory and see the toml file and a subdirectory you just created
4. poetry install

I'm aware of the convenient container isolation and this is the use case, so I share your confusion about why poetry is looking for yahpo_gym in the first place.