Proposal: Improve `dag_bundle_config_list` Configuration in Airflow Helm Chart

[uplsh580]

Description

Currently, the dag_bundle_config_list configuration in the Airflow Helm chart(1.18.0) requires users to provide a JSON string directly in the values file. This approach has several drawbacks:

1. Poor Readability: Long JSON strings in YAML files are difficult to read and maintain, especially when configuring multiple DAG bundles.

2. Error-Prone: Manual JSON string construction is prone to syntax errors (missing quotes, commas, brackets, etc.).

3. Limited IDE Support: IDEs cannot provide proper syntax highlighting, validation, or autocomplete for JSON strings embedded in YAML.

4. Maintenance Burden: Adding, removing, or modifying bundle configurations requires careful string manipulation.

Current Approach

config:
  dag_processor:
    dag_bundle_config_list: '[{"name": "bundle1", "classpath": "airflow.providers.git.bundles.git.GitDagBundle", "kwargs": {"git_conn_id": "GITHUB__repo1", "subdir": "dags", "tracking_ref": "main", "refresh_interval": 60}}, {"name": "bundle2", "classpath": "airflow.providers.git.bundles.git.GitDagBundle", "kwargs": {"git_conn_id": "GITHUB__repo2", "subdir": "dags", "tracking_ref": "develop", "refresh_interval": 120}}]'

As you can see, this becomes unwieldy even with just two bundles, and it gets significantly worse with more configurations.

Proposed Solution

Allow users to define DAG bundle configurations in a structured YAML format, and use a Helm template helper function to automatically convert it to the required JSON string format.

Proposed Structure

dag_bundles:
  - name: bundle1
    classpath: "airflow.providers.git.bundles.git.GitDagBundle"
    kwargs:
      git_conn_id: "GITHUB__repo1"
      subdir: "dags"
      tracking_ref: "main"
      refresh_interval: 60
  - name: bundle2
    classpath: "airflow.providers.git.bundles.git.GitDagBundle"
    kwargs:
      git_conn_id: "GITHUB__repo2"
      subdir: "dags"
      tracking_ref: "develop"
      refresh_interval: 120
  - name: localbundle
    classpath: "airflow.dag_processing.bundles.local.LocalDagBundle"

config:
  dag_processor:
    dag_bundle_config_list: '{{ include "dag_bundles" }}'

Use case/motivation

No response

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[uplsh580]

This proposal is open for discussion and feedback. I welcome suggestions for improvements or alternative approaches.
And, I would appreciate the opportunity to revise it myself.

[kunaljoshi-creator]

Defining dag_bundle_config_list as an embedded JSON string in values.yaml is difficult to read and maintain, particularly for multiple bundles. A structured YAML configuration that is rendered to JSON via a Helm helper would improve readability, reduce errors, and provide better IDE support.

For implementation, it would be good to consider backward compatibility with the existing string-based configuration, as well as clearly documenting precedence if both options are provided.

This seems like a reasonable usability improvement for the Helm chart.

[uplsh580]

For implementation, it would be good to consider backward compatibility with the existing string-based configuration, as well as clearly documenting precedence if both options are provided.

@kunaljoshi-creator
That's a great point. I agree with considering backward compatibility and precedence during implementation.