advice as to why the same cppyy code can not be run multiple times in succession in the same run

[SimonDev666]

Hello,

i have embedded the python interpreter within my c++ program / app (QT) using pybind11.
i setup a simple python script file which internally sets up and then calls a simple c++ function using cppyy.
Runs without a problem.
now copy the same code, keeping to scope rules, so it does the same thing in an excution twice, then it fails on the second call that worked on the first call.
here is the error:

"Test001
Hello, World!
['/home/simon/Desktop/Code/VirtEnv/lib/python3.12/site-packages/', '/home/simon/Code/QT/Test_pybind11/Test_pybind11_cppyy_001/', '', '/usr/lib/python312.zip', '/usr/lib/python3.12', '/usr/lib/python3.12/lib-dynload', '/usr/local/lib/python3.12/dist-packages', '/usr/lib/python3/dist-packages', '/usr/lib/python3.12/dist-packages']
here
done
free(): invalid size
Test002
*** Break *** abort
#0 0x00007f08d6d10893 in __GI___wait4 (pid=10578, stat_loc=stat_loc
entry=0x7ffc9b684228, options=options
entry=0, usage=usage
entry=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
#1 0x00007f08d6d109e7 in _GI___waitpid (pid=, stat_loc=stat_loc.........
#51 0x00007f08d737f94f in PyObject_CallFunction () from /lib/x86_64-linux-gnu/libpython3.12.so.1.0
#52 0x00007f08d74cf10a in PyImport_Import () from /lib/x86_64-linux-gnu/libpython3.12.so.1.0
#53 0x00007f08d74cf31f in PyImport_ImportModule () from /lib/x86_64-linux-gnu/libpython3.12.so.1.0
#54 0x000055dee4fe9934 in pybind11::module::import (name=0x55dee501113e "Python_001") at /usr/include/pybind11/pybind11.h:1279
#55 0x000055dee4fe85f6 in main (argc=1, argv=0x7ffc9b688888) at /home/simon/Code/QT/Test_pybind11/Test_pybind11_cppyy_001/main.cpp:65"

here is the main c++ program / app (this is in my code and is compiled within QT creator):
" std::cout << "Test001\n";
{
//
py::scoped_interpreter guard{};
//
QString qPythonDir("/home/simon/Code/QT/Test_pybind11/Test_pybind11_cppyy_001/");
//
QString qPythonCommand("");
auto locals = py::dict();
//
qPythonCommand += "import sys\n";
qPythonCommand += "sys.path.insert(0, '";
qPythonCommand += qPythonDir.toStdString().c_str();
qPythonCommand += "')\n";
qPythonCommand += "sys.path.insert(0, '/home/simon/Desktop/Code/VirtEnv/lib/python3.12/site-packages/')\n";
qPythonCommand += "print(sys.path)\n";
py::exec(qPythonCommand.toStdString().c_str(), py::globals(), locals);
//
py::module module = py::module_::import("Python_001");
//
module.attr("runCPP")();
}
std::cout << "Test002\n";
{
//
py::scoped_interpreter guard{};
//
QString qPythonDir("/home/simon/Code/QT/Test_pybind11/Test_pybind11_cppyy_001/");
//
QString qPythonCommand("");
auto locals = py::dict();
//
qPythonCommand += "import sys\n";
qPythonCommand += "sys.path.insert(0, '";
qPythonCommand += qPythonDir.toStdString().c_str();
qPythonCommand += "')\n";
qPythonCommand += "sys.path.insert(0, '/home/simon/Desktop/Code/VirtEnv/lib/python3.12/site-packages/')\n";
qPythonCommand += "print(sys.path)\n";
py::exec(qPythonCommand.toStdString().c_str(), py::globals(), locals);
//
py::module module = py::module_::import("Python_001");
//
module.attr("runCPP")();
}
"

here is the python script (Python_001.py) called by pybind11:
"import cppyy

cppyy.cppdef(r"""
#include
void hello() {
std::cout << "Hello, World!" << std::endl;
}
""")

def runCPP():
print("here")
cppyy.gbl.hello()
print("done")
"

my cmake file looks like:
"cmake_minimum_required(VERSION 3.16)
project(Test_pybind11_cppyy_001 LANGUAGES CXX)
set(CMAKE_AUTOUIC ON)
set(CMAKE_AUTOMOC ON)
set(CMAKE_AUTORCC ON)
set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
find_package(QT NAMES Qt6 Qt5 REQUIRED COMPONENTS Core)
find_package(Qt${QT_VERSION_MAJOR} REQUIRED COMPONENTS Core)
find_package(pybind11 REQUIRED)
add_executable(Test_pybind11_cppyy_001
main.cpp
../../EMB.h
)
#target_link_libraries(Test_pybind11_cppyy_001 Qt${QT_VERSION_MAJOR}::Core)
target_link_libraries(Test_pybind11_cppyy_001 Qt${QT_VERSION_MAJOR}::Core pybind11::embed)
include(GNUInstallDirs)
install(TARGETS Test_pybind11_cppyy_001
LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR}
RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR}"

can anyone tell me what i am doing incorrect please.
this is all running on the latest Kali Linux, using the latest QT, with all python libaries uptodate as possible.

[SimonDev666]

more to the about error - as i step through using a debugger, on the cppyy call in Test002, i get the following from the debugger:

"Test001
Hello, World!
['/home/simon/Desktop/Code/VirtEnv/lib/python3.12/site-packages/', '/home/simon/Code/QT/Test_pybind11/Test_pybind11_cppyy_001/', '', '/usr/lib/python312.zip', '/usr/lib/python3.12', '/usr/lib/python3.12/lib-dynload', '/usr/local/lib/python3.12/dist-packages', '/usr/lib/python3/dist-packages', '/usr/lib/python3.12/dist-packages']
here
done
free(): invalid size
44 ./nptl/pthread_kill.c: No such file or directory
81 ../sysdeps/unix/sysv/linux/internal-signals.h: No such file or directory
50 ./nptl/pthread_kill.c: No such file or directory
89 ../sysdeps/unix/sysv/linux/internal-signals.h: No such file or directory
69 ./nptl/pthread_kill.c: No such file or directory
"

so i then copied the Python_001.py into Python_002.py and used that in Test002. I get the same error.
if comment out the "cppy.gbl.hello()" in Python_002.py called by Test002 - everything works, just not calling anything cppyy.
it seems to be whilst in Python_002.py i use cppyy.cppdef to wrap around the c++ source to be run, and then call using Pybind11 the 'runCPP' function that uses the "cppy.gbl.hello()" fails. comment that specific line out, then 'runCPP' works.

has anyone any ideas - any advice - any help - please

[SimonDev666]

more to the error - if it is an error

if i start up python from the command line, ensuring the virtual env is activative.
if i run the Python_001.py script direct from within python, i can run it any number of times. i even tried running both Python_001.py and Python_002.py - all work fine.

so the issues seems to be running python whilst embedded is the problem, using pybind11 as the glue between the two. But pybind11 runs fine for multiple test uses of embedded python in one execution.

confused. pybind11 states its closes down python once its guard goes out of scope. i even tried changing the hello c++ functions in both python_001 & 002 to 'hello1()' and 'hello2()' - same error as before. really confused.

any advice or help please.

[wlav]

I don't understand what you are trying to do here? cppyy is a bindings generator between Python and C++. As such, it has state in both languages. With the scoped_interpreter of pybind11, you tear down the Python portion, leaving the C++ side as-is. Second time, you're trying to reinitialize the Python portion, which tries to reinitialize the C++ side as it should when starting from scratch, but that side is already initialized. Furthermore, to connect the two, the C++ side will have outstanding references to the Python side that were forcibly removed when the prior scoped_interpreter went out of scope, which is the most probable cause of the crash (I can't be sure w/o seeing the full stack trace). I can't see how this could possibly work, without you also offloading and reinitializing the C++ side.

Note that Python itself never offloads dynamic libraries, as the dynamic linker will put a reference on any library into which a symbol was resolved. Python has no control over that. As such, you might get this to work for small, self-contained cases, but not if you intend any connection to some large library such as Qt.

Again, I don't know what the goal is here, but one thing that might work is to import cppyy separately such that it isn't offloaded by the scoped_interpreter's destructor.

[SimonDev666]

thanks for the reply.

this is part of my dissertation for my MSc in Cyber Security. i have developed a visual plug-and-play environment that allows a user to both develop their own code and utilise others solutions (encapsulating their entire command line application). its a bit like metasploit but far more visual, far more extendable and far more experimentable in terms of playing around with captured and created data.

i am embedding python into my QT / C++ solution. i would like to call new c++ code from within this environment, without going through all the issues of compiling and then bring an '.so' into my system explicitly. the python part of this equation is easy.

i understand the issues of scope and am banking on that. the python works well. starting up, stopping and restarting, all under user control. each time python is called with a job of work, there will be no other python running - GIL issues - but this may change in the future with 3.13.

i would like to use cppyy within my embedded python environment to do the above. either take some existing c / c++ code and run it with a set of wrappers, or allow the user to write their own.

so i simulated this isolated environment. first with pybind11 and then just python / c api to utilise cppyy.
i got the desired results for a simple c++ function from with cppyy called from embedded python from within my c++ solution. your examples all worked, but even after stopping python 'via the scoped guard in pybind11' or the 'Py_FinalizeEx with python / c api', a second try almost directly after the first call (all in the same run - one after another), fails.

so i tried something new. now the second run in a new function in my code (first function was 'Test001', this second function 'Test002' - both in c++ in my code). Test001 runs as expected but Test02 fails. i stripped Test002 down to just importing cppyy, leaving Test001 as it was (cppdef a c++ function), with no other code, and it still fails

so i went further. now both Test001 and Test002 (totally independent functions, each starting and stopping their own embedded python sessions as before, running one after each other, not in parallel), each setting up their own sys.paths in python, each only containing "import cppyy". fails when it gets the Test002.

i am totally stumped as to what to do to fix it. i have tested my code multiple times and am happy with all the non-cppyy parts, but i must be doing something wrong somewhere. i just can not see it.
This is what Test001 and Test002 look like

//
void Test001()
{
//
py::scoped_interpreter guard{};
//
QString qPythonDir("/home/simon/Code/QT/Test_pybind11/Test_pybind11_cppyy_001/");
QString qPythonCommand("");

//
auto locals = py::dict();

//
qPythonCommand += "import sys\n";
qPythonCommand += "sys.path.insert(0, '";
qPythonCommand += qPythonDir.toStdString().c_str();
qPythonCommand += "')\n";
qPythonCommand += "sys.path.insert(0, '/home/simon/Desktop/Code/VirtEnv/lib/python3.12/site-packages/')\n";
qPythonCommand += "print(sys.path)\n";
py::exec(qPythonCommand.toStdString().c_str(), py::globals(), locals);

//
qPythonCommand = \
    R"PY(

import cppyy
)PY";

//
py::exec(qPythonCommand.toStdString().c_str(), py::globals(), locals);

}

any advice would be very much appreciated please.

[wlav]

As said above, the Python side isn't the problem, so that's not where the fix can be, it's on the C++ side. As such, I really don't think any of the above approaches have a chance of working.

I tried with cppyy kept alive globally (by running pybind11 from cppyy through C++), but found that that indeed doesn't work with scoped_interpreter.

Two other options: p3.12 subinterpreters (with the main intepreter keeping cppyy alive) or embedding Cling rather than cppyy. I mention the latter as it's not clear to me that you care about Python per se, but rather seem to want to compile C++ code at run-time, with cppyy just a (normally anyway) convenience.

Note that Cling isn't setup to be transactional (that's why cppyy isn't), but you can create and destroy Cling interpreters and have multiple Cling interpreters alive at the same time as long as you're careful not to create linkable symbols in one interpreter that you use in another.

Also, if the code isn't meant to be interactively used (ie. if it's always correct rather than typed on some in-application prompt), you can just embed OrcJIT.

[SimonDev666]

again thanks for the reply.

you are right about performing c++ scripting within Python. it just that your solution held out so many possiblities for this approach and so many more. plus there was little development need to move data back and forward when from python to my solution.

in my day research job i use matlab, which is both very expensive and is not interactive where c++ comes into play. my client was quite keen on my approach and the use of cppyy + python all within a dynamic visual + controlling framework. the university was also. i will have to rethink some of my concepts.

you mentioned using cling (root??). is there any chance of pointing me to a simple guide to both building cling and embedding it (initially for linux and later for windows)??

many thanks for all your advice and carry on with the excellant cppyy project.

[wlav]

Yes, you can use Cling w/o cppyy to get the same features, but yes, the marshaling is then yours to do (of course that's true now as well from Python, but a bit harder since Python's reflection is easier to use).

Cling is here: https://github.com/root-project/cling
Or the newer CppInterop: https://github.com/compiler-research/CppInterOp
Or native clang-repl: https://clang.llvm.org/docs/ClangRepl.html

I'm still not 100% sure on the use case and in particular not on how calls happen, but one way to start experimenting today, is by taking function pointers in your own code, with cppyy only used to create compiled code, not to do the calls. Compilation is globally locked (by Cling) so having that under the GIL doesn't matter. The locking needs of the generated C++ are whatever they are.

The following is a bit self-referencing, but you can execute it in C++ as well using CPyCppyy/API.h to experiment and if it does what you want, move to Cling proper or Clang-Repl later, as they allow all this w/o Python. The main point is in extracting the function pointers, which, once you have them, you can use for whatever from your C++ program:

import cppyy
  
cppyy.cppdef("""\
void hello1() {
    std::cerr << "hello from 1" << std::endl;
}""")

cppyy.cppdef("""\
void hello2() {
    std::cerr << "hello from 2" << std::endl;
}""")

hello1 = cppyy.gbl.gInterpreter.ProcessLine("hello1;")   # evaluates the function pointer
hello2 = cppyy.gbl.gInterpreter.ProcessLine("hello2;")   # id

cppyy.cppexec("((void (*)())%ld)()" % hello1)   # this code would live in C++ land, so no GIL
cppyy.cppexec("((void (*)())%ld)()" % hello2)   # id.

With just Cling Or Clang-Repl, you can also spin up completely fresh interpreters, wiping away old state (Cling will be a bit slower). Neither truly allows resource tracking (AFAIK), like eg. OrcJIT would, but since you can have multiple interpreters active at the same time, the only limitation is the startup time, which if you don't use the PCH and include C++ headers explicitly instead, isn't all that much.

[SimonDev666]

again, thanks for all the advice. i had based all my assumptions on using cppyy as i would in any python session life.
can i ask 2 quite distinct questions please.

question 1.
so you are suggesting that i create my desired cpp interpreted solutions in python using cppyy, then close that python session down and then run these cppyy orientated functions from my side and not python?? how would that work if the user then changed that code to be interpreted; how would cppyy deal with that. is there a way of shutting / closing the cppyy environment down, then allowing for it to be re-started for a new interpreter session doing something completely different or an updated intrepretation.
from what i am understanding i would be pushing water uphill for a great deal of expense, but if its possible as this is only a research dissertation project.

question 2.
i have managed to get cling to compile - no easy feat with my knowledge of such complex cmake processes. i have the default cling environment. i am struggling to go to the next stage and embed its interpreter in a simple c++ solution using cmake (cmakelists).
would it be possible to paste here my recipe of how it did this and you show me how to put together a simple cmakelist that i could use in QT (nothing specific to QT - just using its creator for this process). the idea here would be to make a shared object which i would load when needed and then disposed of - infact this is how most of my functionality code will exist - loaded when needed, making a thin client. of course this would be just for linux - i will have to address windows down the line later.

many thanks for any advice that can be given.

[wlav]

For the first, using cppyy is just a convenience (if not hack), to allow you in essence to pip install cling as brought in by cppyy. You should NOT close the Python session, just use it as a JIT: send it code, return function pointers. (Aside, I forgot that you don't need to access gInterpreter, this cppyy.addressof(cppyy.gbl.hello1) also returns the function pointer address. The latter was done for the benefit of Numba.) And then yes, manage and dispatch on the function pointers from your own code. The point is that these are pure (compiled) C++, so no GIL.

Then no, there really isn't any way to shutdown/restart cppyy. It might be technically possible (or even needed, for sub-interpreter support), but it would require invasive modifications that you're welcome to do (yay, Open Source :) ), but it's going to be an absolute ton of work. Now, if for the start/stop/restart your main motivation is to prevent name classes, I'd suggest using namespaces with unique names.

For the second, yes, that's why I suggested sticking to cppyy through Python a bit longer as it's easier to install. You can then assess whether this is a viable path in the first place. And yes, creating a shared library that can be offloaded (or even creating/discarding interpreters) should be possible this way (it's definitely possible for OrcJIT sec; I'm doing that in Arkouda). I don't think Windows will be different here, except for memory allocations (when linking, you need to export malloc and friends if you intend to delete objects in functions created by functions in a different shared library; this includes, annoyingly, iostream output buffers).

You can share the cmake code, but I really doubt I know more of it than you.

[SimonDev666]

again, thanks for the reply - these really do help

my use-case for wanting to use c++& python as a scripting features in my code, alone with many other future potential language encapsulations, is to allow a user to experiment whilst in my environment and do this visually, to capture results, to save results, to feed results from one solution into another.

Python was easy embed. c++ can be easy but the process can be quite brittle and not always for the faint-hearted, plus the need for many specialist support functions to take data backup and forward. the user may create a solution in a piece meal approach, stop, start, debug, repeat, hence my questions about playing around with the session element of cppyy (a similar idea to that of the life-time in python sessions). i am not particularly worried about the time element yet, but its always nice to have a good functioning flow.

core to my dissertation is not only to allow a user to create their own solutions from raw source code, but to allow them to play around with other's code and to allow them to encapsulate entire command line applications from others. so hence my fixation on python and c++ - others will come later.

i have been playing around with a number of repl solutions about allow the illusion of an interpreter, but this still does not address all of the support issues. cling seems to have a rich set of features, which you use in cppyy, for this.

so it would seem cling or even that clangrepl, may be a direction, if only i can develop a simple demonstrator in a standalone manor (embedding that interpreter in my code).

so here is my cling build recipe to date:
"
git clone https://github.com/root-project/llvm-project.git
cd llvm-project
git checkout cling-latest
cd ..
git clone https://github.com/root-project/cling.git
mkdir cling-build && cd cling-build
cmake -DLLVM_EXTERNAL_PROJECTS=cling -DLLVM_EXTERNAL_CLING_SOURCE_DIR=../cling/ -DLLVM_ENABLE_PROJECTS="clang" -DLLVM_TARGETS_TO_BUILD="host;NVPTX" -DCMAKE_BUILD_TYPE=Release ../llvm-project/llvm
cmake --build . --target cling
cmake --build . --target clang
"
on my kali linux vm, i do this in a directory called 'Experiment'. it takes a considerable time to run and do its thing, but i can run the 'cling' interpreter after. it creates 3 directories: cling, cling-build and llvm-project.

after spending 2 days trying to get cling to embed, i have in utter frustation deleted my play area with my cmake + c++ files, swearing i would never touch cling again - but then i calmed down and regreted my actions.
these are my 2 source (amongst so many others) exampled solutions i tried to follow (and failed to achieve):
https://rawgit.com/vgvassilev/cling/master/www/index.html
https://github.com/root-project/cling/tree/master/tools/demo

at its simpliest, the embedding to the cling interpreter would like this in c++:
#include "cling/Interpreter/Interpreter.h"

int main(int argc, char** argv)
{
const char* LLVMRESDIR = "/usr/local/"; //path to llvm resource directory
cling::Interpreter interp(argc, argv, LLVMRESDIR);

interp.declare("int p=0;");
}

what does 'LLVMRESDIR' point to?? i got something to compile, but fell over at the 'interp' statement.
and because i am using qt across multiple platforms, i would prefere to use cmake

many thanks for at least allowing me to ask all this and if you can help, it would be very much appreciated and save me from tearing any more of my hair out.

[wlav]

Pinging @aaronj0 here, in case he has a cmake fragment to embed clang-repl?

Yes, building LLVM can take a significant amount of time; recommend to always to a full parallel build by allowing 2x the jobs as you have cores available.

As for the LLVM resources dir, it's the directory to the Clang include files for CUDA, AVX, etc. Search for eg. avx2intrin.h. to find it. It's separate from the standard system files b/c tools such as g++ use builtin intrinsics instead.

Beyond that, interactive C++ is significantly different in use that interactive Python. Two important considerations here are the easy way of redefining entities in the latter (or even modifying them); but also the fact that preceding C++ code affects code that comes later and hence combining two pieces of C++ code interactively isn't guaranteed to be deterministic.

[SimonDev666]

thanks for the reply

i understand, or at least some undertanding, of the issues with respect to the interactivity lose - its never going to be anything like that of python or Visual studio or QT in terms of auto-complete - there is the potential for many stop / restarts in the process - but it will allow a user to taken code from what-ever source they choose or code themself to fit into the flow of the visual concept. a session recipe may be made of many languages, and in the visual flow - one node feeding another node, in turn feeding another node, they do not have to fit as in a normal programming flow, just in the data they expect (flow based programming (FBP) is a close example) - an example, so i could capture some raw network traffic, save it and then feed that captured traffic into another node (a piece of code or another encapsulated command-line program) for futher processing - so the flow could be a functional flow, as in most programs we write, or a data flow, with different progams extracting some further meaning that another connected node can further perform processing - so i design a recipe to a soution from many different sub-solutions to achieve some desired objective

the c++ element in an intrepreter is just another node. it could be a teaching aid for students wan to learn some concept - the number of python and c++ code snippets given to me during my MSc i wished i could have quickly tried out - in my clients world we extract meaning from multiple souces of data, which leads to huge monothlic code chunks, which if could be broken down into small parts but still bolten together to achieve some desired objective, would save massive amounts of time.

again, given the cling recipe i posted above, how i could create a simple standalone demo, from which i could then make new solutions in terms of the cmake - i had hoped cppyy would fill this role through using python, but that seems not possible - i do have other potential approachs but nothing with the support functions for data transfer in and out of using cling

if you can provide any advice or know of anyone how could advise, i would very much appreciate it - many thanks

[wlav]

@vgvassilev do you have an example code of the most basic Cling setup?

[vgvassilev]

I think the demo @SimonDev666 mentioned should be the easiest start. We have built tutorials for embedding clang-repl https://github.com/compiler-research/pldi-tutorials-2023

maybe that’d help?

[SimonDev666]

Hello vgvassilev,
my recipe for building cling is above + the very simple dmo code
i hope this is acceptable

i will have a look at the web link you sent

many thanks