According to Apache Airflow Core Concepts , XComs are essentially a key-value store stored in the Airflow metadata database. A task "pushes" a value, and another task "pulls" it. An XCom object is identified by: A unique name for the data. Task ID: The ID of the task that pushed the data. DAG ID: The ID of the DAG the task belongs to. Run ID: The specific execution instance.
| Feature | Use Case | Persistence | | :--- | :--- | :--- | | | Passing dynamic data between specific tasks within a DAG run. | Persists for the duration of the DAG run (usually cleaned up eventually). | | Variables | Storing static configuration or global settings (e.g., API keys, environment names). | Persists globally until manually deleted. | | External Storage | Moving large datasets (files, large DataFrames). | Persists until externally deleted. |
with DAG( "fraud_detection", xcom_exclusive_keys= "fetch_transactions": ["raw_txns"], "validate": ["valid_txns", "error_count"], "feature_engineering": ["features"], "fraud_model": ["score"], , xcom_backend="myapp.xcom.S3ExclusiveXCom", ) as dag:
Adopting these exclusive techniques requires adherence to best practices to ensure you are not introducing new problems while solving old ones. airflow xcom exclusive
To activate this exclusive backend across your entire cluster, update your airflow.cfg file with the following setting: [core] xcom_backend = plugins.custom_backend.S3XComBackend Use code with caution. Best Practices Checklist
Or use the built-in Redis backend (install apache-airflow-providers-redis ):
Scenario:
You can manually push data at any point within your task execution context using context['ti'].xcom_push(key='my_key', value=data) . The Downstream Pull
To help tailor this guide further for your platform architecture, could you let me know: What are you currently using?
If you are using traditional operators (like PythonOperator or BashOperator ), never pull all XComs globally. Always restrict the xcom_pull function by specifying the exact task_ids . According to Apache Airflow Core Concepts , XComs
Mastering Airflow XCom Exclusive Data Sharing: A Comprehensive Guide
Many operators automatically push their return value under the key return_value . For example, a BashOperator that executes echo "hello" will push "hello" to XCom without any additional code.