Move the traces and metrics code under a common observability package

[xBis7]

This patch is refactoring the metrics and the traces packages and moves them under a common package named observability.

This is done so that it will be easier to add common files in the future.

The need for these changes came from this discussion

#56150 (comment)

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[jason810496]

Nice! Thanks for the cleanup!

LGTM for the Airflow-Core and Providers change, but I'm not sure about TaskSDK part.
I will leave it to someone more familiar to answer.

[ferruzzi]

Approved pending Ash weighing in on the SDK part.

[xBis7]

@jason810496 @ferruzzi Thank you for the review!

[xBis7]

@jason810496 @ferruzzi @ashb It's ready for review. The CI is green.

The step for running the task sdk integration tests seems to be flaky.

https://github.com/apache/airflow/actions/runs/18505099702/job/52734949908?pr=56187

I ran it locally and it worked, so I triggered the CI again and it passed.

To help you understand the changes faster

• The new observability package has been moved under shared
• The unit tests have also been moved there
• The otel integration tests haven't been moved because I don't think there is a mechanism to run them on the CI if the change is made
• airflow-core and task-sdk both have their own version of shared directory. The files aren't copies but symlinks to the original ones
• airflow-core imports from its shared symlinks
• task-sdk imports from its own shared symlinks
• providers import from the task-sdk/shared symlinks
• in the task-sdk and the providers, the imports are wrapped with try-except for backwards compat

[jason810496]

Thanks for the update! The overall change LGMT, but there are still question about the dependencies for Airflow-Core and TaskSDK.

[amoghrajesh]

@xBis7 the failure on your PR:

================================================================================================= ERRORS =================================================================================================
__________________________________________________________ ERROR collecting airflow-core/tests/unit/api_fastapi/core_api/routes/test_routes.py ___________________________________________________________
airflow-core/tests/unit/api_fastapi/core_api/routes/test_routes.py:19: in <module>
    from airflow.api_fastapi.core_api.routes.public import authenticated_router, public_router
airflow-core/src/airflow/api_fastapi/core_api/routes/public/__init__.py:24: in <module>
    from airflow.api_fastapi.core_api.routes.public.assets import assets_router
airflow-core/src/airflow/api_fastapi/core_api/routes/public/assets.py:127: in <module>
    def get_assets(
airflow-core/src/airflow/api_fastapi/common/router.py:37: in decorator
    self.add_api_route(
/usr/python/lib/python3.10/site-packages/fastapi/routing.py:1011: in add_api_route
    route = route_class(
/usr/python/lib/python3.10/site-packages/fastapi/routing.py:636: in __init__
    get_parameterless_sub_dependant(depends=depends, path=self.path_format),
/usr/python/lib/python3.10/site-packages/fastapi/dependencies/utils.py:132: in get_parameterless_sub_dependant
    return get_dependant(
/usr/python/lib/python3.10/site-packages/fastapi/dependencies/utils.py:290: in get_dependant
    sub_dependant = get_dependant(
/usr/python/lib/python3.10/site-packages/fastapi/dependencies/utils.py:259: in get_dependant
    endpoint_signature = get_typed_signature(call)
/usr/python/lib/python3.10/site-packages/fastapi/dependencies/utils.py:196: in get_typed_signature
    signature = inspect.signature(call, eval_str=True)
/usr/python/lib/python3.10/inspect.py:3254: in signature
    return Signature.from_callable(obj, follow_wrapped=follow_wrapped,
/usr/python/lib/python3.10/inspect.py:3002: in from_callable
    return _signature_from_callable(obj, sigcls=cls,
/usr/python/lib/python3.10/inspect.py:2463: in _signature_from_callable
    return _signature_from_function(sigcls, obj,
/usr/python/lib/python3.10/inspect.py:2307: in _signature_from_function
    annotations = get_annotations(func, globals=globals, locals=locals, eval_str=eval_str)
/usr/python/lib/python3.10/inspect.py:175: in get_annotations
    return_value = {key:
/usr/python/lib/python3.10/inspect.py:176: in <dictcomp>
    value if not isinstance(value, str) else eval(value, globals, locals)
<string>:1: in <module>
    ???
E   NameError: name 'HTTPAuthorizationCredentials' is not defined

is unrelated and is due to a fastAPI bump it seems.

[xBis7]

is unrelated and is due to a fastAPI bump it seems.

@amoghrajesh I see, it's on main as well.

[amoghrajesh]

@xBis7 yes, the fix is here: #59048

[amoghrajesh]

The fix is merged, rebasing it.

[xBis7]

The fix is merged, rebasing it.

@amoghrajesh I was about to do it. Thanks.

[xBis7]

@amoghrajesh @potiuk Both failures seem random. I ran the redis step locally and it worked.

[amoghrajesh]

Just rebased again, hopefully its green and ready to merge!

[xBis7]

@amoghrajesh @potiuk Thank you for all the help! @jason810496 @ferruzzi @jscheffl Thank you for the reviews!

[potiuk]

Wooohooo

[jason810496]

#protm

[ferruzzi]

Well done, and thanks for sticking with it. That was a long trek, but it turned out great.

[xBis7]

@ferruzzi Thank you!

[dstandish]

Do we not need to maintain the airflow.traces package due to backcompat?