Discussion of cmake-conan evolving towards Dependency Provider.

[hwhsu1231]

Recently, I found that a new experimental branch (experiment/cmake_project_include) was created, and the Conan team has submitted a few PRs related to it:

• experiment with CMAKE_PROJECT_INCLUDE #473
• Add file to configure Conan as a cmake dependency provider #474
• dep provider: fixes #476

Does cmake-conan decide to evolve towards Dependency Providers?

If that is the case, then I would be very happy to see it evolves in this direction. I think this solution may be better than the cmake-conan-setup (#449) that I previously proposed.

[memsharded]

It is not a decision yet, but we are experimenting with the different possibilities, trying to achieve the best possible integration.
Certainly the dependency providers looks good, but we want to learn first if there could be some blocker or problems before announcing anything, so it is expected that we will be trying things for a couple of weeks.

[hwhsu1231]

@memsharded - Have you considered the CAE problem of CMakeToolchain generator in this scheme?

[memsharded]

@memsharded - Have you considered the CAE problem of CMakeToolchain generator in this scheme?

Yes, hopefully it will not be a problem

[hwhsu1231]

Yes, hopefully it will not be a problem

@memsharded

Based on what CMake Docs said about Dependency Provider:

When a call is made to find_package() or FetchContent_MakeAvailable(), the call may be forwarded to a dependency provider which then has the opportunity to fulfill the request.

If I haven't misunderstood the mechanism, it seems that the xxx_provide_dependency() function is called only when we call find_package() command. For example:

cmake_minimum_required(VERSION 3.24)
project(cmake-conan-test LANGUAGES CXX)
#
# Call the xxx_provide_dependency() when the find_package(fmt) is invoked.
#
find_package(fmt)

That means the conan_toolchain.cmake file of CMakeToolchain generator will be generated after the project() command. It's because the conan install command is executed inside the conan_provide_dependency() function currently.

cmake-conan/conantools.cmake

Lines 127 to 151 in 1f33316

	function(conan_provide_dependency package_name)
	if(NOT CONAN_INSTALL_SUCCESS)
	message(STATUS "CMake-conan: first find_package() found, running 'conan install' to install deps")
	detect_host_profile(${CMAKE_BINARY_DIR}/conan_host_profile)
	if(CMAKE_BUILD_TYPE)
	message(STATUS "CMake-conan: Intalling single configuration ${CMAKE_BUILD_TYPE}")
	conan_install(-pr ${CMAKE_BINARY_DIR}/conan_host_profile --build=missing -g CMakeDeps)
	else()
	message(STATUS "CMake-conan: Intalling both Debug and Release")
	conan_install(-pr ${CMAKE_BINARY_DIR}/conan_host_profile -s build_type=Release --build=missing -g CMakeDeps)
	conan_install(-pr ${CMAKE_BINARY_DIR}/conan_host_profile -s build_type=Debug --build=missing -g CMakeDeps)
	endif()
	if (CONAN_INSTALL_SUCCESS)
	set(CONAN_GENERATORS_FOLDER "${CONAN_GENERATORS_FOLDER}" CACHE PATH "Conan generators folder")
	endif()
	else()
	message(STATUS "CMake-conan: find_package(${package_name}) found, 'conan install' aready ran")
	endif()
	if (CONAN_GENERATORS_FOLDER)
	list(PREPEND CMAKE_PREFIX_PATH "${CONAN_GENERATORS_FOLDER}")
	endif()
	find_package(${ARGN} BYPASS_PROVIDER)
	endfunction()

However, the key of solving the CAE problem is that we have to generate the conan_toolchain.cmake file and then include(conan_toolchain.cmake) before the project() command.

Right?

[hwhsu1231]

I conduct a simple experiment of Dependency Provider. The followings are the demo files:

• CMakeLists.txt:

  Click to expand details

  cmake_minimum_required(VERSION 3.24)
  
  message("[DEBUG] Before calling \"project()\" command...")
  project(cmake-conan-test LANGUAGES CXX)
  message("[DEBUG] After calling \"project()\" command...")
  
  message("[DEBUG] Before calling \"find_package(fmt)\" command...")
  find_package(fmt)
  message("[DEBUG] After calling \"find_package(fmt)\" command...")

• CMakePresets.json:

  Click to expand details

  {
    "version": 4,
    "cmakeMinimumRequired": {
      "major": 3,
      "minor": 23,
      "patch": 0
    },
    "configurePresets": [
      {
        "name": "win32",
        "displayName": "Windows",
        "description": "Windows",
        "generator": "Visual Studio 16 2019",
        "toolset": "v142,host=x64",
        "architecture": "x64",
        "binaryDir": "${sourceDir}/build/${presetName}",
        "cacheVariables": {
          "CMAKE_PROJECT_TOP_LEVEL_INCLUDES": "${sourceDir}/conan_provider.cmake"
        }
      }
    ]
  }

• conan_provider.cmake:

  Click to expand details

  message("[DEBUG] Start to include \"conan_provider.cmake\" file...")
  
  function(conan_provide_dependency package_name)
      message("[DEBUG] Start to call \"conan_provide_dependency()\" function...")
  endfunction()
  
  cmake_language(
      SET_DEPENDENCY_PROVIDER conan_provide_dependency
      SUPPORTED_METHODS FIND_PACKAGE
  )

The following is the log of running cmake --preset win32 command:

Click to expand details

D:\Test\cmake-conan-provider>cmake --preset win32
Preset CMake variables:

  CMAKE_PROJECT_TOP_LEVEL_INCLUDES="D:/Test/cmake-conan-provider/conan_provider.cmake"

[DEBUG] Before calling "project()" command...
-- Selecting Windows SDK version 10.0.22000.0 to target Windows 10.0.22621.
[DEBUG] Start to include "conan_provider.cmake" file...
-- The CXX compiler identification is MSVC 19.29.30147.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
[DEBUG] After calling "project()" command...
[DEBUG] Before calling "find_package(fmt)" command...
[DEBUG] Start to call "conan_provide_dependency()" function...
CMake Warning at CMakeLists.txt:8 (find_package):
  By not providing "Findfmt.cmake" in CMAKE_MODULE_PATH this project has
  asked CMake to find a package configuration file provided by "fmt", but
  CMake did not find one.

  Could not find a package configuration file provided by "fmt" with any of
  the following names:

    fmtConfig.cmake
    fmt-config.cmake

  Add the installation prefix of "fmt" to CMAKE_PREFIX_PATH or set "fmt_DIR"
  to a directory containing one of the above files.  If "fmt" provides a
  separate development package or SDK, be sure it has been installed.


[DEBUG] After calling "find_package(fmt)" command...
-- Configuring done
-- Generating done
-- Build files have been written to: D:/Test/cmake-conan-provider/build/win32

After extracting the [DEBUG] message, we can see that conan_provide_dependency() function is indeed called after project() command:

[DEBUG] Before calling "project()" command...
[DEBUG] Start to include "conan_provider.cmake" file...
[DEBUG] After calling "project()" command...
[DEBUG] Before calling "find_package(fmt)" command...
[DEBUG] Start to call "conan_provide_dependency()" function...
[DEBUG] After calling "find_package(fmt)" command...

If that is the case, I'm afraid that this scheme might not be able to solve the CAE problem...

[jcar87]

we are not currently considering having Conan generate conan_toolchain.cmake (CMakeToolchain). That needs to provided before the first invocation of cmake configuration, and as such, Conan would need to be invoked first in order to generate it.

In the current experiment we simply ensure that CMakeDeps has been called to satisfy dependencies by the time we enter the first invocation of find_package inside the consumer's CMake scripts. We are also assuming the existence of a conanfile.txt or conanfile.py listing the requirements.

I have tested my current version of conan_provider.cmake, alongside your CMakeLists.txt and CMakePresets.json (with a conanfile.txt expressing the dependency on fmt), and I can see the find_package call correctly finding the Conan-provided fmt.

Note that all of this is subject to change, we are simply experimenting with Conan 2.0 and CMake's new dependency provided feature.

cmake --preset=win32
Preset CMake variables:

  CMAKE_PROJECT_TOP_LEVEL_INCLUDES="C:/Users/julius/dev/conan-io/cmake-conan/conan_provider.cmake"

-- The CXX compiler identification is MSVC 19.34.31937.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.34.31933/bin/Hostx64/x64/cl.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- CMake-conan: first find_package() found, running 'conan install' to install deps
-- Conan-cmake: cmake_system_name=Windows
-- Conan-cmake: CMake compiler=MSVC
-- Conan-cmake: CMake cmpiler version=19.34.31937.0
-- Conan-cmake: [settings] compiler=msvc
-- Conan-cmake: [settings] compiler.version=193
-- Conan-cmake: Creating profile C:/Users/julius/dev/conan-io/cmake-conan/julius/consumer/build/win32/conan_host_profile
-- Conan-cmake: Profile:
include(default)
[settings]
os=Windows
compiler=msvc
compiler.version=193
[conf]
tools.cmake.cmaketoolchain:generator=Visual Studio 17 2022

-- CMake-conan: Intalling both Debug and Release
-- CMake-conan: conan install C:/Users/julius/dev/conan-io/cmake-conan/julius/consumer -of=C:/Users/julius/dev/conan-io/cmake-conan/julius/consumer/build/win32/conan -pr;C:/Users/julius/dev/conan-io/cmake-conan/julius/consumer/build/win32/conan_host_profile;-s;build_type=Release;--build=missing;-g;CMakeDeps

======== Input profiles ========
Profile host:
[settings]
arch=x86_64
build_type=Release
compiler=msvc
compiler.cppstd=14
compiler.runtime=dynamic
compiler.runtime_type=Release
compiler.version=193
os=Windows
[conf]
tools.cmake.cmaketoolchain:generator=Visual Studio 17 2022

Profile build:
[settings]
arch=x86_64
build_type=Release
compiler=msvc
compiler.cppstd=14
compiler.runtime=dynamic
compiler.runtime_type=Release
compiler.version=193
os=Windows


======== Computing dependency graph ========
Graph root
    conanfile.txt: C:/Users/julius/dev/conan-io/cmake-conan/julius/consumer\conanfile.txt
Requirements
    fmt/9.1.0#6708c9d84f98d56a6d9f2e6c2d5639ba - Cache

======== Computing necessary packages ========
Requirements
    fmt/9.1.0#6708c9d84f98d56a6d9f2e6c2d5639ba:bed280a9c51d41820ca64294cf373083ad852aa8#37ef2fe37424dcc4928575821a1b7f3c - Cache

======== Installing packages ========
fmt/9.1.0: Already installed! (1 of 1)
WARN: Usage of deprecated Conan 1.X features that will be removed in Conan 2.X:
WARN:     'cpp_info.names' used in: fmt/9.1.0
tools.cmake.cmaketoolchain:generator=Visual Studio 17 2022

Profile build:
[settings]
arch=x86_64
build_type=Release
compiler=msvc
compiler.cppstd=14
compiler.runtime=dynamiccompiler.runtime_type=Release
compiler.version=193
os=Windows


======== Computing dependency graph ========
Graph root
    conanfile.txt: C:/Users/julius/dev/conan-io/cmake-conan/julius/consumer\conanfile.txt
Requirements
    fmt/9.1.0#6708c9d84f98d56a6d9f2e6c2d5639ba - Cache

======== Computing necessary packages ========
Requirements
    fmt/9.1.0#6708c9d84f98d56a6d9f2e6c2d5639ba:d7e170053c010c715fb01e2b851bc035c96184c1#38106a0610ea359cbf4676e33c0c07d0 - Cache

======== Installing packages ========
fmt/9.1.0: Already installed! (1 of 1)
WARN: Usage of deprecated Conan 1.X features that will be removed in Conan 2.X:
WARN:     'cpp_info.names' used in: fmt/9.1.0

======== Finalizing install (deploy, generators) ========
conanfile.txt: Writing generators to C:/Users/julius/dev/conan-io/cmake-conan/julius/consumer/build/win32/conan
conanfile.txt: Generator 'CMakeDeps' calling 'generate()'
conanfile.txt: Generating aggregated env files
conanfile.txt: Generated aggregated env files: ['conanbuild.bat', 'conanrun.bat']
Install finished succesfully
-- CMake-conan: CONAN_GENERATORS_FOLDER=C:/Users/julius/dev/conan-io/cmake-conan/julius/consumer/build/win32/conan
-- Conan: Component target declared 'fmt::fmt'
-- Configuring done
-- Generating done
-- Build files have been written to: C:/Users/julius/dev/conan-io/cmake-conan/julius/consumer/build/win32

[hwhsu1231]

Since Conan Team insisted that we should include the conan_toolchain.cmake file in the the following previous issues:

• [feature] Add IMPORTED_LOCATION_<CONFIG> properties for Imported Targets created by CMakeDeps. conan#12077 (comment)

I wondered what will happen if we don't include the conan_toolchain.cmake file?

[jcar87]

Since Conan Team insisted that we should include the conan_toolchain.cmake file in the the following previous issues:

• [feature] Add IMPORTED_LOCATION_<CONFIG> properties for Imported Targets created by CMakeDeps. conan#12077 (comment)

I wondered what will happen if we don't include the conan_toolchain.cmake file?

It will depend on some specific use cases and workflows - the conan_toolchain does indeed set up some things that need to be defined ahead of the first invocation to project(). The dependency providers feature is limited to resolving calls to find_package - I believe there's enough information available about the os/compiler/compiler version at the point of the first invocation, that we could rely on this to invoke conan install, and resolve the call to find_package - but there will be limitations: this only works for find_package, but wouldn't work for find_program, find_library, or to provide things like the compiler itself via a conan package (can't have it both ways: either you call Conan from CMake with a compiler you've already identified, or you call Conan with the tool requirements you want CMake to use in a subsequent call). Some of these we may be able to mitigate over time, but there will indeed be limitations.

[memsharded]

A proposal for 2.0 based on dependency providers and providing transparent (no need to modify CMakeLists.txt) is in https://github.com/conan-io/cmake-conan/tree/develop2, testing and feedback welcome.

[hwhsu1231]

Excuse me, Conan Team.

I'm glad to see that the plan of cmake-conan evolving towards Dependecny Provider succeeds. However, as @jcar87 mentioned in his comment #475 (comment), there exist some limitations in the Dependency Provider. I'm not sure whether these limitations can be mitigated in the future. Therefore, I want to see cmake-conan-setup as a fallback plan.

Besides, although the Conan 2.0 is already released, I still use Conan 1.59.0 currently. It's because I already applied the cmake-conan-setup in some of my personal projects, but the currently latest conan.cmake is not compatible with Conan 2.0. That enforces me to stay in the Conan 1.59.0 version.

Therefore, I hope that the develop branch can be kept maintained thereafter.

First of all, could you review/merge this PR (#454) that I submitted before into develop branch? (I've been waiting it for a long time...)