sample cbindgen

[youknowone]

Please check if you can use Modules/_base64/_base64.rs.h for clinic input

[emmatyping]

Please check if you can use Modules/_base64/_base64.rs.h for clinic input

I expect not as-is, but clinic could be modified to handle cbindgen generated headers, so I wouldn't worry to much about that.

Independently of that however, I may have been unclear when saying I'd want a proof of concept, so sorry for being unclear! I may be misunderstanding cbindgen, but I think if we'd use cbindgen, we'd want to use it so we can define types in Rust? In other words, define PyObject, PyMethodDef, etc. in Rust, then generate the header to be used by CPython. cc @Gankra (hopefully you don't mind the ping!)

As-is I don't think we need _base64.rs.h from the C side as far as I can tell. It could potentially be used for clinic input I suppose. Simplifying a lot here because there's a lot of complexity and actually a few ways this can work, but CPython's import system will look up an import mymodule statement by:

1. looking for mymodule[.abi-tag].so
2. loading that shared library via dlopen()
3. looking up (via dlsym()) PyInit_mymodule and get a function pointer
4. call that function pointer

It doesn't need to know the signature nor name of the module initialization function as those are pre-defined (see https://docs.python.org/3.15/c-api/extension-modules.html#c.PyInit_modulename).

Sorry again for the confusion!

[youknowone]

This is a third option of question in pre-PEP: What about Argument Clinic?

Rather than either integrating Rust support into clinic or creating a Rust procedural macro capable of parsing a similar DSL, generating C header is totally possible and clinic can use it as C input as before. This generation is only will be useful for clinic and no usage out of it. (Please let me know if clinic doesn't work in that way)

Because type and function generation have different dependencies and benefits, it is better to think of them as “cbindgen types” and “cbindgen functions”, rather than everything together simply as “cbindgen.” This is about cbindgen functions for clinic.

but I think if we'd use cbindgen, we'd want to use it so we can define types in Rust? In other words, define PyObject, PyMethodDef, etc. in Rust, then generate the header to be used by CPython

This is also totally right. And about "cbindgen types". They can be fully separated and have its own profits of reliable type definitions.

[emmatyping]

This is a third option of question in pre-PEP: What about Argument Clinic?

Rather than either integrating Rust support into clinic or creating a Rust procedural macro capable of parsing a similar DSL, generating C header is totally possible and clinic can use it as C input as before.

Aha! That makes much more sense. Thank you for the explanation, that helped a lot. So this would allow us to automatically generate the PyMethodDefs and argument parsing code through clinic needed for Rust functions in an extension. An interesting idea!

The only other part of this that isn't clear to me (sorry for having so many questions!) is how we'd tie this back to the Rust module definition. Clinic works by parsing the comments above implementation functions and generating both a function handling argument parsing, and also a macro for the PyMethodDef entry for that function. The PyMethodDef points to the function handling argument parsing, and that function calls the implementation function. Since these only exist in a header, and the PyMethodDef is defined in a macro, I don't think we can use clinic as-is. Normally, these would be included into the C file where the original function is defined.

For an example, the sha2.sha256 function arguments are defined here: https://github.com/python/cpython/blob/a35c683da55e77c96828fd0421640787337cfc64/Modules/sha2module.c#L563-L577
Here's the clinic output: https://github.com/python/cpython/blob/a35c683da55e77c96828fd0421640787337cfc64/Modules/clinic/sha2module.c.h#L171-L254
And here's where that header is included: https://github.com/python/cpython/blob/a35c683da55e77c96828fd0421640787337cfc64/Modules/sha2module.c#L87

I suppose we could modify clinic to either output Rust code, or output to a C file instead of a header?

[youknowone]

Thank you! Thanks to your explanation, I realized that I had been making an incorrect assumption about the role and behavior of clinic.

_sha2_sha256 must have a C ABI, but _sha2_sha256_impl is preferably a Rust function for better ergonomics. As you said, applying this without modifying clinic would be difficult. Since cbindgen only works on #[no_mangle] extern "C" fn, it would never generate the _sha2_sha256_impl function.

If the goal is minimally patching clinic, it must be patched to take Rust file as input but still can produce a C (header) file as output, that would seem like a viable solution. On the Rust side, if we define extern "C" fn _sha2_sha256 and extern "C" static _sha2_sha256__doc__, then we should be able to use the C functions clinic generates.

[emmatyping]

If the goal is minimally patching clinic, it must be patched to take Rust file as input but still can produce a C (header) file as output

That header wouldn't be compiled into anything however. Right now clinic only works for the sha2 module because sha2module.c includes clinic/sha2module.c.h (the clinic output header) and thus the functions in the header get compiled into sha2module.o. If we output a header with the argument parsing function it isn't going to get included in any Rust compilation unit. We could I suppose output to a .c file then link that into the static library that is generated? That still wouldn't give us the PyMethodDef structures however, as those are macros. I think it would be better to properly support Rust output in clinic, or use a proc-macro.

[youknowone]

That's right. making proper Rust output looks a lot better. Thank you for the review!

[emmatyping]

Thank you for talking through the solution with me and your patience!

[Gankra]

(Everything y'all discussed makes sense, and yes totally fine to ping me!)