- MetaDatabase
- API Server
- Scheduler
- DAG-Processor
- Triggerer
✔ Airflow Scheduler
The Scheduler is the core component that orchestrates your workflows. It is responsible for:
What it does
-
Evaluates DAGs: Determines what tasks need to be run.
-
Creates task instances based on schedules or callbacks.
-
Queues tasks for execution by workers (Celery/Kubernetes/Local).
-
Monitors task states (success, failure, retries).
-
Handles SLAs, task dependencies, backfills, and scheduling intervals.
Key point
The scheduler decides when tasks should run and sends them to the executor.
✔ Airflow Triggerer
The Triggerer is used only for deferrable operators, introduced to handle async triggers efficiently.
What it does
-
Runs async triggers that wait for events (e.g., sensor completion, external conditions) without blocking worker slots.
-
Uses asyncio to handle thousands of concurrent lightweight tasks.
-
Wakes the scheduler/executor when an event is completed.
Key point
The Triggerer waits for external events asynchronously, freeing worker resources.
Examples of things the triggerer waits for:
-
File sensor waiting for object in S3
-
Time-based sleep
-
External API response
-
Custom deferrable operator
✔ Airflow DAG Processor
The DAG Processor handles parsing and processing DAG files.
What it does
-
Watches your DAG directory for new/changed DAG files.
-
Parses DAG Python files in isolation, producing serialized DAGs stored in the Metadata DB.
-
Offloads DAG parsing from the Scheduler for performance and safety.
Key point
The DAG Processor parses and prepares DAGs, while the Scheduler uses the parsed DAGs to schedule tasks.
🎯 Summary Chart
| Component | Main Purpose | Key Responsibilities |
|---|---|---|
| Scheduler | Decides when tasks run | Create task instances, queue tasks, enforce dependencies |
| Triggerer | Handles async waiting | Manage deferrable operators and event-based waiting |
| DAG Processor | Parses DAGs | Scan DAG folder, parse files, serialize DAGs |
🧠 Simple Analogy
-
DAG Processor = “The compiler” → reads DAG code and prepares it.
-
Scheduler = “The planner” → decides which tasks to run and when.
-
Triggerer = “The waiter” → waits for asynchronous events efficiently.