[Question] Bundle-specific Python Path Isolation in Airflow 3.x Git Bundles

[uplsh580]

[Question] Bundle-specific Python Path Isolation in Airflow 3.x Git Bundles

Environment:

• Airflow Version: 3.1.7
• Deployment: Git Bundles
• Setup: Multi-tenant environment where each Git Repository (Bundle) belongs to a specific team.

---

Context (Path Architecture):
Our infrastructure deploys code into different paths depending on the Airflow component:

1. DAG Processor: {BUNDLE_ROOT}/{bundle_name}/tracking_repo/{user_code_root}/
2. Worker: {BUNDLE_ROOT}/{bundle_name}/version/{commit_id}/{user_code_root}/

The Directory Structure (Inside {user_code_root}):

{user_code_root}/
├── airflow_lib/       # Internal library (Shared across DAGs in the same bundle)
│   ├── __init__.py
│   ├── constants/
│   └── util/
├── dags/              # Actual DAG files
│   ├── my_dag.py
│   └── sub_dir/
│       └── nested_dag.py
└── README.md

The Problem:
When a DAG (e.g., my_dag.py) is parsed by the DAG Processor or executed by the Worker, it fails with ModuleNotFoundError: No module named 'airflow_lib'.

This is because the {user_code_root} directory—which contains the airflow_lib package—is not automatically added to sys.path. Since the path is dynamic (varies by commit_id on Workers) and inconsistent (differs between Processor and Worker), we cannot use a static global PYTHONPATH.

Key Challenges:

1. Component Path Discrepancy: Any solution must work for both the tracking_repo path on the Processor and the version/{commit_id} path on the Worker.

2. Namespace Collisions: Multiple bundles (teams) might have an airflow_lib folder. Adding all bundle roots to sys.path would cause collision and version "cross-talk."

3. No Manual Code Changes: We want to avoid forcing hundreds of developers to add sys.path.append logic to every DAG file.

Questions:

1. Is there a way to configure Airflow 3.x to automatically recognize the Bundle's specific root ({user_code_root}) as a Python source root during parsing and execution?

2. Are there any hooks or listeners (like on_task_instance_run or DAG policies) that are recommended for injecting component-specific paths dynamically?

3. How should we handle the fact that the import root changes between the DAG Processor (tracking) and Worker (versioned) while maintaining the same import statement from airflow_lib import ...?

We are looking for a clean, infrastructure-level solution that aligns with the AIP-66 design philosophy. Thank you!

[uplsh580]

Update: I found the solution!

The issue was caused by the subdir parameter in my Git Bundle configuration.
The Root Cause:
In my initial configuration, I had the subdir argument set to "dags":

{
  "name": "TESTBED2",
  "classpath": "airflow.providers.git.bundles.git.GitDagBundle",
  "kwargs": {
    "git_conn_id": "GITHUB__airflow-testbed2",
    "subdir": "dags", // This was the culprit
    "tracking_ref": "AIRFLOW-DEPLOY/develop",
    "refresh_interval": 60
  }
}

When subdir is specified in the Git Bundle configuration, Airflow treats that specific subdirectory as the root for DAG discovery and, crucially, adds only that subdirectory to the sys.path.

Since my internal library airflow_lib/ was located at the same level as the dags/ folder (siblings), it was excluded from the Python path. This led to the ModuleNotFoundError during both DAG parsing and task execution.

The Fix:
By removing the subdir parameter (or setting it to the root of the repository), Airflow now recognizes the {user_code_root} as the primary source root. This allows the Python interpreter to correctly resolve the airflow_lib package while still being able to find the DAGs within the dags/ directory.

Key Takeaway:
If you have internal packages or utilities at the root of your repository that are shared across DAGs, do not set the subdir parameter to your DAGs folder in the bundle configuration. Keep the bundle root as the source root to ensure sibling packages are importable.

[jherrmannNetfonds]

I had the same issue and tried to fix this by modifying the python path. But the issue is that the DAG processor probes PYTHONPATH directories during startup before the DAG bundle git checkout finishes. Python caches those paths as empty (None in sys.path_importer_cache). The cache is never invalidated, so later imports fail even though the files are now there.
My workaround for now is to call importlib.invalidate_caches() in one DAG to force Python to re-scan the directories.