🐛 fix(stubs): resolve stub type hints for C/Rust extensions

[gaborbernat]

C/Rust extension functions re-exported from parent packages — like cbor2._cbor2.dumps exposed as cbor2.dumps — had no working stub resolution. Sphinx's built-in _find_type_stub_spec() only looks for .pyi files adjacent to the .so with the same filename, so stubs at the parent package level (e.g. cbor2/__init__.pyi) were never found. This meant type aliases like EncoderHook were expanded to raw Callable[...] signatures, annotations on C extension functions were lost entirely, and cross-references were missing from rendered docs. Additionally, C extension classes that define constructors via __new__ (with __init__ inherited from object) had no constructor parameter documentation because process_docstring only examined __init__. 🐛

The fix introduces a parent-package walk-up that follows the PEP 561 re-export pattern: when no stub is found next to the extension module, it traverses sys.modules upward to find the parent whose __init__.pyi documents the re-exported public API. A unified _get_stub_context() call retrieves the stub's local namespace, TypeAlias names (detected via AST including typing.TypeAlias, typing_extensions.TypeAlias, and type statements), and the owning module name in a single lookup — avoiding duplicate path discovery. The owning module name is then used as globalns during eval()-based annotation resolution, ensuring the correct namespace is used when the function's __module__ points at the C extension child rather than the stub-owning parent. ✨

A crossref flag on MyTypeAliasForwardRef distinguishes stub-derived aliases (which emit :py:type: roles for Sphinx cross-referencing) from autodoc_type_aliases display names (which must not be wrapped to avoid double-quoting in text output). The collect_documented_type_aliases function no longer matches unqualified type names from other modules, eliminating a cross-contamination risk in multi-module projects. Constructor logic in process_docstring now falls back to __new__ when __init__ is object.__init__ and __new__ is overridden, and uses inspect.signature(cls) instead of inspect.signature(cls.__new__) to get the correct parameter list for C extensions where the latter returns (*args, **kwargs).

When injecting annotation-based types, preexisting :type: directives in the docstring (including multi-line continuations) are now removed and replaced rather than duplicated. Inline :param type name: formats are stripped to :param name: before injection, ensuring all parameter types use consistent CSS styling via the sphinx_autodoc_typehints-type class. Preexisting :rtype: directives are also replaced with the annotation-derived return type, fixing a bug where stale docstring return types would shadow the actual function signature.

[agronholm]

While this PR seems to improve things, I'm seeing strange behavior:

• For dumps/dump, not all type hints are resolved, and the links look different from normal
• The type aliases are expanded even though the plugin's documentation says they shouldn't be when they're included in the docs

Compare the results – the first is from dumps(), the other from CBOREncoder.encode():

I'm running with -W -n but no errors are emitted at build time.

[agronholm]

Right, I still had explicit type hints in my docstring for CBOREncoder, so that explains why it looks correct.

[agronholm]

I removed all the explicit type hints now. CBOREncoder shows no type hints now. Is it not picking up the hints from CBOREncoder.__new__() in the type stubs?

[agronholm]

Ah, I see: obj = obj.__init__ if inspect.isclass(obj) else obj
It's not even trying to get the annotations from __new__().

[agronholm]

As for the failing resolution of CBOREncoder and CBORDecoder, it seems like it's caused by combined_localns only containing imports and not anything defined locally in the stub. Am I on the right track?

[agronholm]

I also noticed that collect_documented_type_aliases() always returns an empty dict. This was probably not intentional either?

EDIT: It would seem like this function was not updated for type stub support.

[gaborbernat]

@agronholm can you try again? Pushed some more fixes.

[agronholm]

Now it seems like all type hints are gone completely. Did this work for you locally?

[agronholm]

My bad, I installed the wrong branch. With the correct branch, type hints are there but the resolution of local names still doesn't work.

[gaborbernat]

Could you share a specific example of which local names aren't resolving? For instance, is it that type aliases like EncoderHook are being expanded instead of preserved, or that cross-references to local classes like CBOREncoder aren't linking properly?

On my end, building the cbor2 rust branch docs with this PR produces zero warnings under sphinx-build -W -n, and EncoderHook/TagHook render as named cross-referenced aliases rather than expanded Callable[...].

[agronholm]

Could you share a specific example of which local names aren't resolving? For instance, is it that type aliases like EncoderHook are being expanded instead of preserved, or that cross-references to local classes like CBOREncoder aren't linking properly?

On my end, building the cbor2 rust branch docs with this PR produces zero warnings under sphinx-build -W -n, and EncoderHook/TagHook render as named cross-referenced aliases rather than expanded Callable[...].

I already showed screenshots here, did I not? I also shared my analysis on why that is happening.

[gaborbernat]

Those screenshots were before the last fix, locally I see:

So my question what issues you're seeing post latest fixes.

[agronholm]

I had local changes that interfered with the name resolution. With those gone, I'm seeing the same as you are, so that's one issue down. The classes are still missing their type hints and the type alias expansion is still happening though.

[agronholm]

I tested your latest changes. Here are my findings:

1. Links to class names work now
2. Type aliases are now correctly kept
3. Type aliases do not get properly linked
4. Type annotations from the __new__() method are not collected/used

So I would call this good progress, but there is still some work left to be done.

[agronholm]

If you fixed something with those changes, I can't say what. If you keep force-pushing, it's hard for me to see what actually changed.

[gaborbernat]

If you fixed something with those changes, I can't say what. If you keep force-pushing, it's hard for me to see what actually changed.

Should be good now 🤔 (at least in my local tests seems sane).

[agronholm]

I can confirm that the links to type aliases work properly now. The only remaining problem is then the missing type hints in the initialization parameters of CBOREncoder and CBORDecoder.

[agronholm]

As a data point, changing __new__() to __init__() in the stub didn't help with the initialization parameter annotations.

[gaborbernat]

@agronholm any remaining feature gaps? 🤔

[agronholm]

@agronholm any remaining feature gaps? 🤔

You mean other than the initialization parameters in the classes missing their type hints?

[gaborbernat]

Do you mean these, or we're talking about something else?

[agronholm]

This is what I'm seeing (nuked the build dir, reinstalled from this branch, ran sphinx-build):

[agronholm]

Strange we're getting different results from the same code. What specific function should I be looking at to debug?

[gaborbernat]

Let me double check 🤔 can you post the exact command sequence you ran and cwd?

[agronholm]

I ran sphinx-build docs/ build/sphinx -W -n from the project root.

[agronholm]

I'm debugging this, and I can confirm it's picking up the type hints from the stub's __new__() method. But they're not showing up anywhere.

[gaborbernat]

I ran sphinx-build docs/ build/sphinx -W -n from the project root.

Yeah but how you've created envs, how did you install the tool? 🤔

[agronholm]

The env: python3.12 -m venv venv312
Installing the project: pip install --group doc -e .
Installing the tool: pip install git+https://github.com/gaborbernat/sphinx-autodoc-typehints.git@654

[gaborbernat]

Debugging 👍

[agronholm]

I'm stepping through process_docstring() for name='cbor2.CBOREncoder' and there's nothing wrong with how the type hints end up:

(Pdb) pp type_hints
{'canonical': <class 'bool'>,
 'date_as_datetime': <class 'bool'>,
 'datetime_as_timestamp': <class 'bool'>,
 'default': typing.Optional[MyTypeAliasForwardRef('EncoderHook')],
 'encoders': collections.abc.Mapping[type, MyTypeAliasForwardRef('EncoderHook')] | None,
 'fp': typing.IO[bytes],
 'indefinite_containers': <class 'bool'>,
 'return': 'Self',
 'string_referencing': <class 'bool'>,
 'timezone': datetime.tzinfo | None,
 'value_sharing': <class 'bool'>}

[agronholm]

Okay, so I figured it was going wrong when process_docstring() called _inject_types_to_docstring() with signature value that was just *args, **kwargs. No wonder the parameters weren't matched to their type hints. But how did it work for you? Different Python version?

[agronholm]

I tried on 3.10 and 3.14 too and they too give a signature of (*args, **kwargs) for the __new__() method, so that's not the explanation.

[agronholm]

Are you getting a different result when inspecting the signature of CBOREncoder.__new__?

[agronholm]

If I get the signature from the CBOREncoder class itself and not the __new__ method, then it looks correct.

[gaborbernat]

I am not entirely sure. I'll have another look at it tomorrow.

[gaborbernat]

Check with latest commit 🤔 but I'm truly off now until tomorrow 😆

[agronholm]

Seems to work now! Thank you for your quick response to this.

[gaborbernat]

Cool 🆒

[henryiii]

See #656, I think this breaks NamedTuple.