The YellowDog Provider for Apache Airflow provides:
- A Hook to support YellowDog Application credentials as Airflow Connections
- Operator classes to create and manage YellowDog Work Requirements and Worker Pools
- Sensor classes to monitor the states of YellowDog Work Requirements and Worker Pools
The main documentation is found at: https://docs.yellowdog.ai/airflow-provider/index.html
Requires Apache Airflow v2.10.0 or later, Python 3.10 or later, and the YellowDog SDK.
Install from PyPI as package yellowdog-airflow-provider, e.g.:
pip install -U yellowdog-airflow-provider
Please see the documentation at https://docs.yellowdog.ai/airflow-provider/index.html.
The script start-airflow-container.sh will start a local container running Airflow, including installing the YellowDog Airflow provider and creating YellowDog Connections.
Run this script from the top-level repo. directory, i.e.:
bin/start-airflow-command.sh
Please inspect the script for instructions on its customisation and use.
There's an example DAG that exercises all the YellowDog Provider components at yd-provider-test.py. The DAG's directory is automatically loaded by the Airflow container above.
Inspect/update the variables at beginning of the DAG module to customise the DAG for your environment.