Improve dag_bundle_config_list Configuration in Airflow Helm Chart

[uplsh580]

Overview

Related Issue: #60630
Improve dag_bundle_config_list Configuration in Airflow Helm Chart

This PR improves the dag_bundle_config_list configuration in the Airflow Helm Chart by allowing users to define DAG bundle configurations in a structured YAML format instead of requiring a JSON string directly in the values file.

Changes

• Added a new dag_bundles section under dagProcessor in values.yaml that allows users to define DAG bundles in a structured YAML format
• Created a Helm template helper function dag_bundles that automatically converts the YAML structure to the required JSON string format
• Set default value for dagProcessor.dag_bundles with a local DAG bundle configuration
• Updated config.dag_processor.dag_bundle_config_list to use the helper function
• Moved dag_bundles configuration under dagProcessor section for better logical grouping

Benefits

1. Better Readability: Long JSON strings in YAML files are difficult to read and maintain. The new structured format is much more readable.
2. Less Error-Prone: Manual JSON string construction is prone to syntax errors. The YAML format reduces the chance of errors.
3. Better IDE Support: IDEs can provide proper syntax highlighting, validation, and autocomplete for YAML structures.
4. Easier Maintenance: Adding, removing, or modifying bundle configurations is now straightforward.
5. Better Organization: DAG bundle configuration is now logically grouped under the dagProcessor section.

Usage

Users can now define DAG bundles like this:

dagProcessor:
  dag_bundles:
    - name: dags-folder
      classpath: "airflow.dag_processing.bundles.local.LocalDagBundle"
      kwargs: {}
    - name: bundle1
      classpath: "airflow.providers.git.bundles.git.GitDagBundle"
      kwargs:
        git_conn_id: "GITHUB__repo1"
        subdir: "dags"
        tracking_ref: "main"
        refresh_interval: 60
    - name: bundle2
      classpath: "airflow.providers.git.bundles.git.GitDagBundle"
      kwargs:
        git_conn_id: "GITHUB__repo2"
        subdir: "dags"
        tracking_ref: "develop"
        refresh_interval: 120

---

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)
• Cursor

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[uplsh580]

Validation

Limitation

Values are transmitted in a single-line format.

[jscheffl]

Some tests are missing - can you add some please?

[uplsh580]

Some tests are missing - can you add some please?

Thank you for the feedback. I've updated the PR with the following changes:

• Added unit tests for dagProcessor.dagBundleConfigList.
• Marked the setting as deprecated in the comments to avoid overwhelming users with multiple parallel options, as suggested.
• Standardized variable names to align with the style of other values in the Helm chart.

[choo121600]

One small nit suggestion:
we’ve agreed to refer to DAG as Dag going forward.
It would be great if you could update the occurrences accordingly :)

[VOTE] results as of https://lists.apache.org/thread/0651j4vdgzmlhgndmotvznlg97kyh328

[uplsh580]

One small nit suggestion:
we’ve agreed to refer to DAG as Dag going forward.
It would be great if you could update the occurrences accordingly :)

[VOTE] results as of https://lists.apache.org/thread/0651j4vdgzmlhgndmotvznlg97kyh328

Great catch, thank you for sharing the discussion link. I've gone through the PR and adjusted DAG to Dag to stay consistent with the community's decision. Thanks for your guidance!

[jscheffl]

Looks good for me.