Data Engineer | 40 questions | 2026
Airflow Interview Questions and Answers
Airflow interview questions for freshers 19
Q1FresherMost askedWhat is Apache Airflow and what is a DAG?
AnswerHide
Airflow is an open-source platform to write, schedule and watch workflows. You write the workflows as Python code. A DAG (directed acyclic graph) is one workflow. It is a set of tasks with dependencies, and no loops.
from datetime import datetime
from airflow.decorators import dag, task
@dag(schedule="@daily", start_date=datetime(2025, 1, 1), catchup=False)
def daily_report():
@task
def extract():
return [1, 2, 3]
@task
def load(rows):
print(sum(rows))
load(extract())
daily_report()
Airflow decides when and in what order tasks run. The heavy work should run in other systems, such as Spark, a warehouse or a container.
What they are checking: Do you know the core idea, with a real example?
Q2FresherMost askedWhat are the main components of Airflow architecture?
AnswerHide
Airflow has these parts:
- Scheduler: reads the DAG files, finds which task instances are due, and queues them.
- Executor: decides how and where tasks run. It is part of the scheduler process.
- Workers: processes or pods that run the tasks.
- Webserver / API server: the UI where you watch and trigger runs.
- Metadata database: PostgreSQL or MySQL. It stores DAG runs, task states, connections and variables.
- DAG folder: the Python files that hold the workflows.
- Triggerer: runs deferrable operators in the background (async).
The scheduler and workers talk through the metadata database. With Celery, a broker such as Redis is also used.
What they are checking: Do you know how the parts fit together?
Q3FresherMost askedWhat is an operator, a task and a task instance in Airflow?
AnswerHide
- Operator: a template for one unit of work, such as
BashOperator,PythonOperator,S3ToRedshiftOperator. - Task: an operator used inside a DAG, with its arguments and a
task_id. - Task instance: one run of a task for one DAG run (one logical date). It has a state such as
success,failed,running.
from airflow.operators.bash import BashOperator
t = BashOperator(task_id="say_hi", bash_command="echo hi")
So the operator is the class. The task is the object in the DAG. The task instance is that task at one point in time.
What they are checking: Do you know the basic words?
Q4FresherMost askedWhat are sensors in Airflow and how do they work?
AnswerHide
Sensors are operators that wait for something to happen. For example, a file lands in S3, or another DAG finishes. Then the sensor succeeds.
from airflow.providers.amazon.aws.sensors.s3 import S3KeySensor
wait = S3KeySensor(task_id="wait_file", bucket_name="my-bucket",
bucket_key="in/{{ ds }}/data.csv",
mode="reschedule", poke_interval=300, timeout=6*3600)
There are two modes. In poke mode, the sensor keeps a worker slot busy and sleeps between checks. In reschedule mode, it frees the slot between checks. For long waits, use reschedule or a deferrable sensor. Always set a timeout.
What they are checking: Do you know waiting patterns and how they use resources?
Q5FresherMost askedWhat is XCom and when should you not use it?
AnswerHide
XCom (cross-communication) lets tasks pass small pieces of data to each other. The data is stored in the metadata database.
from airflow.decorators import task
@task
def a():
return {"rows": 120} # pushed as XCom automatically
@task
def b(info):
print(info["rows"]) # pulled via argument
b(a())
Use it for small values: ids, paths, counts and status flags. Do not pass DataFrames or big files. Write the data to S3 or a database and pass only its location. Large XComs make the metadata DB big and slow the scheduler. A custom XCom backend can store them in object storage.
What they are checking: Do you know the limit of XCom?
Q6FresherMost askedExplain catchup and backfill in Airflow.
AnswerHide
Catchup: the DAG's start_date is in the past and catchup=True. Then the scheduler creates a run for every missed interval, from the start date until now. With catchup=False, it schedules only the latest interval.
Backfill: you run a DAG by hand for a range of dates. You do this for old data or after a fix:
airflow dags backfill -s 2025-01-01 -e 2025-01-31 my_dag
In Airflow 3, you can also start a backfill from the UI and the API. Backfills need tasks that are idempotent (running twice gives the same result). They must use the logical date, not the current time. Set max_active_runs so a backfill does not overload the system.
What they are checking: Do you know how scheduling handles past dates?
Q7FresherMost askedHow do retries, retry_delay and SLAs work in Airflow?
AnswerHide
Set retries for each task or in default_args:
from datetime import timedelta
default_args = {
"retries": 3,
"retry_delay": timedelta(minutes=5),
"retry_exponential_backoff": True,
"execution_timeout": timedelta(hours=1),
}
A failed task goes to up_for_retry. It runs again after the delay, until the retries are used up. Then it is failed. execution_timeout kills a task that runs too long.
SLA is a target for how late a task or DAG may finish. If it is missed, Airflow sends a callback or email, but it does not stop the task. Task-level SLAs were deprecated in Airflow 2.x. They were removed in 3.0 and replaced by deadline-style alerts, so check your version. Use on_failure_callback for alerts.
What they are checking: Do you know the settings that make a pipeline reliable?
Q8FresherMost askedFree with sign-inHow do you set task dependencies in Airflow?
AnswerHide
Use the bit-shift operators or helper functions.
Q9FresherMost askedFree with sign-inWhat is the difference between Airflow Variables and Connections?
AnswerHide
- Connections store credentials and endpoints for outside systems (host, login, password, extras). Operators and hooks refer to them by
conn_id. - Variables are global key-value settings, such as a bucket name or a feature flag.
Q10FresherMost askedFree with sign-inWhat is the TaskFlow API?
AnswerHide
TaskFlow is the decorator style (Airflow 2.0+) for writing DAGs with plain Python functions. @task turns a function into a task. When you pass a return value to another task, Airflow makes the dependency and the XCom for you.
Q11FresherMost askedFree with sign-inWhat is the difference between PythonOperator and BashOperator, and what other operators have you used?
AnswerHide
PythonOperator runs a Python function inside the worker. BashOperator runs a shell command. Both are generic. Provider packages add operators for specific services:
Q12FresherMost askedFree with sign-inHow do you trigger one DAG from another, or make a DAG wait for another DAG?
AnswerHide
- Trigger:
TriggerDagRunOperatorstarts another DAG. It can passconfand can wait for the run to finish. - Wait:
ExternalTaskSensorwaits for a task or DAG in another DAG run. The logical dates must match. - Data-aware scheduling (Datasets/Assets, Airflow 2.4+): the producer task declares
outlets. The consumer DAG usesschedule=[dataset]. It runs when the data is updated, and you need no sensors.
Q13FresherMost askedFree with sign-inWhat is Jinja templating in Airflow? Give examples of macros.
AnswerHide
Templated operator fields are filled in at runtime with Jinja. This adds run details to your code.
Q14FresherMost askedFree with sign-inHow do you monitor Airflow and get alerts on failures?
AnswerHide
Use several layers of monitoring:
Q15FresherMost askedFree with sign-inWhat is the difference between Airflow and a cron job?
AnswerHide
Cron only starts a command at a set time. Airflow adds more. It has dependencies between tasks, retries and alerts. It has backfills and a UI with logs and history. It uses the logical date as a parameter. It can scale to many workers.
Q16FresherMost askedFree with sign-inWhat are pools in Airflow?
AnswerHide
Pools limit how many tasks can run at the same time against a shared resource, such as a database or an API. The limit works across all DAGs. A pool has a number of slots. Each task takes slots (pool_slots) while it runs.
Q17FresherMost askedFree with sign-inWhat is the difference between depends_on_past and wait_for_downstream?
AnswerHide
depends_on_past=True: a task instance runs only if the same task succeeded in the previous DAG run.wait_for_downstream=True: a task instance also waits until the tasks right after it in the previous run have succeeded.
Q18FresherMost askedFree with sign-inHow do you pass parameters to a DAG at trigger time?
AnswerHide
Define params with default values on the DAG. Override them when you trigger the DAG by hand. You can use the UI form, the CLI --conf, or the API.
Q19FresherMost askedFree with sign-inHow do you handle secrets and credentials in Airflow?
AnswerHide
Do not put passwords in DAG files or plain Variables. Use Connections with a secrets backend (AWS Secrets Manager, GCP Secret Manager, HashiCorp Vault). Then Airflow looks up the secrets at runtime.
Airflow interview questions for experienced 13
Q20ExperiencedMost askedWhat is the difference between executors in Airflow: Local, Celery and Kubernetes?
AnswerHide
The executor decides how tasks are run.
| Executor | How it runs tasks | Good for |
|---|---|---|
| Sequential | One at a time, uses SQLite | testing only |
| Local | Parallel processes on the scheduler machine | small setups |
| Celery | Pool of long-running workers via Redis/RabbitMQ | steady load, many short tasks |
| Kubernetes | A new pod for each task | isolation, bursty load, per-task resources |
Celery starts tasks fast, but you must manage the workers all the time. Kubernetes gives clean isolation and autoscaling. But each task pays the pod start-up time. There is also CeleryKubernetes, which uses both. Recent versions can also use several executors at once.
What they are checking: Can you choose the right way to scale?
Q21ExperiencedMost askedHow does Airflow scheduling work with start_date and schedule? Why does a DAG run after the interval ends?
AnswerHide
A DAG run covers one data interval. It starts only after that interval has ended.
Example: a @daily DAG has the data interval 2025-10-01 00:00 to 2025-10-02 00:00. It starts at 2025-10-02 00:00. Its logical date (ds) is 2025-10-01, the start of the interval. This is because the run processes the data of that interval.
start_date: the start of the first interval. Nothing runs before it.schedule: a cron string, a preset (@daily) or a timetable.- Templated values such as
{{ data_interval_start }}and{{ ds }}help tasks fetch the right slice of data.
So a new DAG with today's start_date seems not to run. The first interval has not finished yet.
What they are checking: This is one of the most confusing Airflow ideas. Can you explain it clearly?
Q22ExperiencedMost askedWhat does it mean for an Airflow task to be idempotent and how do you achieve it?
AnswerHide
An idempotent task gives the same result if you run it once or many times for the same logical date. So retries and backfills are safe.
- Use
{{ ds }}or the data interval as a parameter. Never usedatetime.now(). - Overwrite the partition for that date, or use upsert/
MERGE. Do not just append. - Write to a temporary place, then swap it in one step (atomic).
- Delete and insert the date's data in one transaction.
- Avoid side effects that cannot be repeated, or protect them with a check.
DELETE FROM fact_sales WHERE load_date = '{{ ds }}';
INSERT INTO fact_sales SELECT * FROM stage WHERE load_date = '{{ ds }}';
What they are checking: Do you know the key design rule?
Q23ExperiencedMost askedFree with sign-inWhat are best practices for writing Airflow DAGs?
AnswerHide
- Keep top-level DAG code light. No API calls, DB queries or heavy imports at parse time.
- Make tasks idempotent and atomic. Use logical dates and templates.
- Use small tasks with clear limits, not one giant function.
- Keep Airflow as the orchestrator. Push heavy work to Spark, warehouses or containers.
- Keep...
Q24ExperiencedMost askedFree with sign-inWhat are trigger rules in Airflow?
AnswerHide
A trigger rule decides when a task runs, based on the states of its upstream tasks. The default is all_success.
Q25ExperiencedMost askedFree with sign-inWhat is dynamic task mapping in Airflow?
AnswerHide
Dynamic task mapping (Airflow 2.3+) creates a changing number of task instances at runtime. The number depends on the output of an upstream task. You use expand.
Q26ExperiencedMost askedFree with sign-inHow do you test an Airflow DAG?
AnswerHide
Test in layers:
Q27ExperiencedMost askedFree with sign-inExplain the concurrency settings: parallelism, max_active_runs and max_active_tasks.
AnswerHide
parallelism(core config): the most task instances that can run at once in the whole installation.max_active_tasks(per DAG): the most task instances that can run at once for one DAG, across its runs.max_active_runs(per DAG): the most DAG runs at once. It helps with backfills and stops runs from overlapping.max_active_tis_per_dag(per task): the limit for one task across runs.worker_concurrency(Celery): tasks for each worker.- Pools: a limit for each resource.
Q28ExperiencedMost askedFree with sign-inWhat is the difference between PythonOperator and a custom operator or hook? When do you write your own?
AnswerHide
A hook is a reusable way to talk to an outside system (connection handling, API calls). Examples are S3Hook and PostgresHook. An operator is a task template. It usually uses a hook to do its job.
Q29ExperiencedMost askedFree with sign-inWhat are deferrable operators and the triggerer?
AnswerHide
Deferrable operators give a long wait to the triggerer. This is a background (async) process that runs many waiting triggers in one event loop. While the task is deferred, it frees its worker slot. So thousands of waits cost almost nothing.
Q30ExperiencedMost askedFree with sign-inWhat happens if the scheduler or a worker dies during a task?
AnswerHide
The scheduler and workers are separate. The state lives in the metadata database. So a restart continues the work.
Q31ExperiencedMost askedFree with sign-inWhat is the Airflow metadata database and what problems can it cause?
AnswerHide
It stores DAGs, runs, task instances, XComs, connections, variables and references to logs. It is the single source of truth, so everything depends on it.
Q32ExperiencedFree with sign-inWhat is the difference between schedule, schedule_interval and timetable?
AnswerHide
schedule_interval was the older argument. It took a cron string, a preset or a timedelta. In Airflow 2.4+, the single schedule argument replaces it. schedule accepts a cron expression, a preset like @daily, a timedelta, a list of datasets, or a Timetable object. Custom timetables handle rules that cron cannot write, such as 'every business day' or odd calendars.
Scenario based Airflow interview questions 8
Q33ExperiencedMost askedScenarioFree with sign-inYour DAG is not showing up in the Airflow UI. What do you check?
AnswerHide
Work from the file toward the scheduler.
Q34ExperiencedMost askedScenarioFree with sign-inA DAG is enabled but its scheduled runs are not starting. What could be wrong?
AnswerHide
Check these in order:
Q35ExperiencedMost askedScenarioFree with sign-inTasks are stuck in the 'queued' state. How do you troubleshoot it?
AnswerHide
Queued means the scheduler gave the task to the executor, but no worker has started it.
Q36FresherMost askedScenarioFree with sign-inHow do you handle a failed task in the middle of a DAG without rerunning everything?
AnswerHide
Fix the cause. Then clear only the failed task, and its downstream tasks if you want, in the UI or CLI. Airflow reruns just those task instances.
Q37ExperiencedMost askedScenarioFree with sign-inHow would you design a daily pipeline: extract from an API, load to S3, transform with Spark, load to a warehouse?
AnswerHide
Use four idempotent tasks, keyed on the logical date. Airflow orchestrates them, and other systems do the work.
Q38ExperiencedMost askedScenarioFree with sign-inYou need to rerun a failed pipeline for the last 30 days after fixing a bug. How do you do it safely?
AnswerHide
Use a backfill, or clear the past runs, with limits, on idempotent tasks.
Q39ExperiencedMost askedScenarioFree with sign-inTasks are stuck in the 'running' state but nothing is happening. What do you do?
AnswerHide
The task process may have died without telling Airflow. Or it may be waiting on something forever.
Q40ExperiencedMost askedScenarioFree with sign-inA DAG takes too long to parse and the scheduler is slow. How do you fix it?
AnswerHide
Parse time is the time spent running your DAG files. So cut the work done at module level.