Add support for disabling SA token automount for Scheduler (apache#59156)

* Add support for manual mounting of Service Account tokens in Scheduler when using Pod launching executors (apache#59099)

* fix bug in helper function for determining if executor is Pod launching (apache#59099)

* Remove emojis in production guide (apache#59099)

* Auto formatting by prek and some manual cleanup (apache#59099)

* Removed the detailed security enhancements from the main documentation page and created newsfragment 59156.significant.rst to properly document these changes in the changelog as suggested.  (apache#59099)

* Reduced redundancy by removing detailed examples and linking to the dedicated service-account-token-examples.rst document for comprehensive configuration examples and use cases. (apache#59099)

* Add support for manual mounting of Service Account tokens in Scheduler when using Pod launching executors (apache#59099)

* fix bug in helper function for determining if executor is Pod launching (apache#59099)

* Remove emojis in production guide (apache#59099)

* Auto formatting by prek and some manual cleanup (apache#59099)

* Removed the detailed security enhancements from the main documentation page and created newsfragment 59156.significant.rst to properly document these changes in the changelog as suggested.  (apache#59099)

* Reduced redundancy by removing detailed examples and linking to the dedicated service-account-token-examples.rst document for comprehensive configuration examples and use cases. (apache#59099)

* Fix title underlines in SA token examples RST file. (apache#59099)

* Fix spell checks and removed wrong Helm values schema spec for Worker. (apache#59099)

* Minor fix in descriptions. (apache#59099)

---------

Co-authored-by: Dan Osterrath <9108541+dan-osterrath@users.noreply.github.com>

Author: dan-osterrath

chart/README.md

@@ -47,6 +47,11 @@ cluster using the [Helm](https://helm.sh) package manager.
 * Supported database backend: ``PostgreSQL``, ``MySQL``
 * Autoscaling for ``CeleryExecutor`` provided by KEDA
 * ``PostgreSQL`` and ``PgBouncer`` with a battle-tested configuration
+* **Security enhancements**:
+   * Container-specific Service Account Token Volume configuration implementing Principle of Least Privilege
+   * Only scheduler containers receive API access; init and sidecar containers operate without tokens
+   * Defense-in-depth security with both ServiceAccount and Pod-level controls
+   * Compatibility with security policies like Kyverno and compliance frameworks
 * Monitoring:
    * StatsD/Prometheus metrics for Airflow
    * Prometheus metrics for PgBouncer

chart/docs/index.rst

@@ -44,6 +44,7 @@ Helm Chart for Apache Airflow
     :caption: Guides
 
     production-guide
+    service-account-token-examples
 
 .. toctree::
     :hidden:

chart/docs/production-guide.rst

@@ -703,3 +703,174 @@ this behavior is deprecated in favor of explicitly defining ``.Values.imagePullS
 
 You can read more about advanced ways of setting configuration variables in the
 :doc:`apache-airflow:howto/set-config`.
+
+Service Account Token Volume Configuration
+------------------------------------------
+
+When using pod-launching executors (``CeleryExecutor``, ``CeleryKubernetesExecutor``, ``KubernetesExecutor``, ``LocalKubernetesExecutor``),
+you can configure how Kubernetes service account tokens are mounted into pods. This provides enhanced security control
+and compatibility with security policies like Kyverno.
+
+Background
+^^^^^^^^^^
+
+By default, Kubernetes automatically mounts service account tokens into pods via the ``automountServiceAccountToken`` setting.
+However, for security reasons, you might want to disable automatic mounting and manually configure service account token volumes instead.
+
+This feature addresses Bug #59099 where ``scheduler.serviceAccount.automountServiceAccountToken: false`` was ignored
+when using the KubernetesExecutor. The solution implements a defense-in-depth approach with both ServiceAccount-level
+and Pod-level controls.
+
+Container-Specific Security
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The Service Account Token Volume is mounted **only** in containers that require Kubernetes API access, implementing the
+**Principle of Least Privilege**:
+
+* **Scheduler Container**: Receives Service Account Token (needs API access for pod management)
+* **Init Container "wait-for-airflow-migrations"**: No Service Account Token (only performs database migrations)
+* **Sidecar Container "scheduler-log-groomer"**: No Service Account Token (only performs log cleanup operations)
+
+This container-specific approach ensures that:
+
+- **Database Migration Container**: Only accesses the database for schema updates, no Kubernetes API access required
+- **Log Groomer Container**: Only performs filesystem operations for log cleanup, no API access required
+- **Scheduler Container**: Requires API access for launching and managing pods with pod-launching executors
+
+**Security Benefits:**
+
+* **Reduced Attack Surface**: Containers without API access cannot interact with the Kubernetes API even if compromised
+* **Compliance**: Meets security policy requirements that mandate minimal privilege assignment
+* **Audit Trail**: Clear separation of which containers have API access for security auditing
+* **Defense-in-Depth**: Multiple layers of security controls at both ServiceAccount and container levels
+
+Configuration Options
+^^^^^^^^^^^^^^^^^^^^^
+
+The service account token volume configuration is available for the scheduler component and includes the following options:
+
+.. code-block:: yaml
+
+  scheduler:
+    serviceAccount:
+      automountServiceAccountToken: false  # Disable automatic token mounting
+      serviceAccountTokenVolume:
+        enabled: true                      # Enable manual token volume
+        mountPath: /var/run/secrets/kubernetes.io/serviceaccount  # Mount path for the token
+        volumeName: kube-api-access        # Name of the projected volume
+        expirationSeconds: 3600            # Token expiration time in seconds
+        audience: ~                        # Token audience (optional)
+
+Security Implications
+^^^^^^^^^^^^^^^^^^^^^
+
+**When to use manual token volumes:**
+
+* When security policies require explicit control over service account token mounting
+* When using security policy engines like Kyverno that restrict automatic token mounting
+* When implementing defense-in-depth security strategies
+* When you need custom token expiration times or audiences
+* When compliance frameworks mandate container-specific privilege assignment
+
+**Security benefits:**
+
+* **Explicit control**: Manual configuration makes token mounting intentional and visible
+* **Policy compliance**: Compatible with security policies that restrict ``automountServiceAccountToken: true``
+* **Defense-in-depth**: Provides both ServiceAccount-level and Pod-level security controls
+* **Custom expiration**: Allows setting shorter token lifetimes for enhanced security
+* **Container isolation**: Only scheduler container receives API access, reducing attack surface
+* **Principle of Least Privilege**: Each container receives only the minimum required permissions
+
+**Why Init and Sidecar Containers Don't Need API Access:**
+
+* **Init Container (wait-for-airflow-migrations)**:
+
+  - **Purpose**: Performs database schema migrations using ``airflow db migrate``
+  - **Required Access**: Database connection only
+  - **Security Rationale**: Database operations don't require Kubernetes API interaction
+
+* **Sidecar Container (scheduler-log-groomer)**:
+
+  - **Purpose**: Cleans up old log files from the filesystem
+  - **Required Access**: Local filesystem access only
+  - **Security Rationale**: Log cleanup is purely filesystem-based, no API calls needed
+
+* **Scheduler Container**:
+
+  - **Purpose**: Manages DAG scheduling and launches task pods
+  - **Required Access**: Kubernetes API for pod creation, monitoring, and cleanup
+  - **Security Rationale**: Pod-launching executors require API access for container orchestration
+
+Use Cases and Examples
+^^^^^^^^^^^^^^^^^^^^^^
+
+For comprehensive configuration examples, security scenarios, and detailed use cases,
+see :doc:`service-account-token-examples`.
+
+Supported Executors
+^^^^^^^^^^^^^^^^^^^
+
+The service account token volume configuration is only effective for pod-launching executors:
+
+* ``CeleryExecutor`` - When launching Celery worker pods
+* ``CeleryKubernetesExecutor`` - For both Celery workers and Kubernetes task pods
+* ``KubernetesExecutor`` - When launching task pods in Kubernetes
+* ``LocalKubernetesExecutor`` - For Kubernetes task pods in local mode
+
+For other executors (``LocalExecutor``, ``SequentialExecutor``), this configuration has no effect
+as they don't launch additional pods.
+
+Migration from Automatic to Manual Token Mounting
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+To migrate from automatic to manual token mounting:
+
+1. **Test the configuration** in a non-production environment first
+2. **Update your values.yaml**:
+
+   .. code-block:: yaml
+
+     scheduler:
+       serviceAccount:
+         automountServiceAccountToken: false  # Disable automatic mounting
+         serviceAccountTokenVolume:
+           enabled: true                      # Enable manual mounting
+
+3. **Deploy the changes** using Helm upgrade
+4. **Verify** that the scheduler can still launch pods successfully
+5. **Monitor** for any authentication issues in the logs
+
+Troubleshooting
+^^^^^^^^^^^^^^^
+
+**Common Issues:**
+
+* **Authentication failures**: Ensure ``serviceAccountTokenVolume.enabled`` is set to ``true`` when ``automountServiceAccountToken`` is ``false``
+* **Permission denied**: Verify that the service account has the necessary RBAC permissions
+* **Token expiration**: Check if ``expirationSeconds`` is too short for your workload patterns
+
+**Debugging:**
+
+Check the scheduler logs for authentication-related errors:
+
+.. code-block:: bash
+
+  kubectl logs deployment/airflow-scheduler -n <namespace>
+
+Verify the projected volume is mounted correctly:
+
+.. code-block:: bash
+
+  kubectl describe pod <scheduler-pod-name> -n <namespace>
+
+Backward Compatibility
+^^^^^^^^^^^^^^^^^^^^^^
+
+This feature maintains full backward compatibility:
+
+* Existing deployments with ``automountServiceAccountToken: true`` (default) continue to work unchanged
+* The ``serviceAccountTokenVolume`` configuration is only applied when explicitly enabled
+* Default values ensure no breaking changes for existing installations
+
+For more information about Kubernetes service account tokens and projected volumes, see the
+`Kubernetes documentation on service account tokens <https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#serviceaccount-token-volume-projection>`_.