Airflow Interview Questions and Answers

Airflow is an open-source platform to write, schedule and watch workflows. You write the workflows as Python code. A DAG (directed acyclic graph) is one workflow. It is a set of tasks with dependencies, and no loops.

from datetime import datetime
from airflow.decorators import dag, task

@dag(schedule="@daily", start_date=datetime(2025, 1, 1), catchup=False)
def daily_report():
    @task
    def extract():
        return [1, 2, 3]

    @task
    def load(rows):
        print(sum(rows))

    load(extract())

daily_report()

Airflow decides when and in what order tasks run. The heavy work should run in other systems, such as Spark, a warehouse or a container.

What they are checking: Do you know the core idea, with a real example?

Airflow has these parts:

• Scheduler: reads the DAG files, finds which task instances are due, and queues them.
• Executor: decides how and where tasks run. It is part of the scheduler process.
• Workers: processes or pods that run the tasks.
• Webserver / API server: the UI where you watch and trigger runs.
• Metadata database: PostgreSQL or MySQL. It stores DAG runs, task states, connections and variables.
• DAG folder: the Python files that hold the workflows.
• Triggerer: runs deferrable operators in the background (async).

The scheduler and workers talk through the metadata database. With Celery, a broker such as Redis is also used.

What they are checking: Do you know how the parts fit together?

• Operator: a template for one unit of work, such as BashOperator, PythonOperator, S3ToRedshiftOperator.
• Task: an operator used inside a DAG, with its arguments and a task_id.
• Task instance: one run of a task for one DAG run (one logical date). It has a state such as success, failed, running.

from airflow.operators.bash import BashOperator
t = BashOperator(task_id="say_hi", bash_command="echo hi")

So the operator is the class. The task is the object in the DAG. The task instance is that task at one point in time.

What they are checking: Do you know the basic words?

Sensors are operators that wait for something to happen. For example, a file lands in S3, or another DAG finishes. Then the sensor succeeds.

from airflow.providers.amazon.aws.sensors.s3 import S3KeySensor
wait = S3KeySensor(task_id="wait_file", bucket_name="my-bucket",
                   bucket_key="in/{{ ds }}/data.csv",
                   mode="reschedule", poke_interval=300, timeout=6*3600)

There are two modes. In poke mode, the sensor keeps a worker slot busy and sleeps between checks. In reschedule mode, it frees the slot between checks. For long waits, use reschedule or a deferrable sensor. Always set a timeout.

What they are checking: Do you know waiting patterns and how they use resources?

XCom (cross-communication) lets tasks pass small pieces of data to each other. The data is stored in the metadata database.

from airflow.decorators import task

@task
def a():
    return {"rows": 120}      # pushed as XCom automatically

@task
def b(info):
    print(info["rows"])        # pulled via argument

b(a())

Use it for small values: ids, paths, counts and status flags. Do not pass DataFrames or big files. Write the data to S3 or a database and pass only its location. Large XComs make the metadata DB big and slow the scheduler. A custom XCom backend can store them in object storage.

What they are checking: Do you know the limit of XCom?

Catchup: the DAG's start_date is in the past and catchup=True. Then the scheduler creates a run for every missed interval, from the start date until now. With catchup=False, it schedules only the latest interval.

Backfill: you run a DAG by hand for a range of dates. You do this for old data or after a fix:

airflow dags backfill -s 2025-01-01 -e 2025-01-31 my_dag

In Airflow 3, you can also start a backfill from the UI and the API. Backfills need tasks that are idempotent (running twice gives the same result). They must use the logical date, not the current time. Set max_active_runs so a backfill does not overload the system.

What they are checking: Do you know how scheduling handles past dates?

Set retries for each task or in default_args:

from datetime import timedelta
default_args = {
    "retries": 3,
    "retry_delay": timedelta(minutes=5),
    "retry_exponential_backoff": True,
    "execution_timeout": timedelta(hours=1),
}

A failed task goes to up_for_retry. It runs again after the delay, until the retries are used up. Then it is failed. execution_timeout kills a task that runs too long.

SLA is a target for how late a task or DAG may finish. If it is missed, Airflow sends a callback or email, but it does not stop the task. Task-level SLAs were deprecated in Airflow 2.x. They were removed in 3.0 and replaced by deadline-style alerts, so check your version. Use on_failure_callback for alerts.

What they are checking: Do you know the settings that make a pipeline reliable?

Use the bit-shift operators or helper functions.

• Connections store credentials and endpoints for outside systems (host, login, password, extras). Operators and hooks refer to them by conn_id.
• Variables are global key-value settings, such as a bucket name or a feature flag.

TaskFlow is the decorator style (Airflow 2.0+) for writing DAGs with plain Python functions. @task turns a function into a task. When you pass a return value to another task, Airflow makes the dependency and the XCom for you.

PythonOperator runs a Python function inside the worker. BashOperator runs a shell command. Both are generic. Provider packages add operators for specific services:

• Trigger: TriggerDagRunOperator starts another DAG. It can pass conf and can wait for the run to finish.
• Wait: ExternalTaskSensor waits for a task or DAG in another DAG run. The logical dates must match.
• Data-aware scheduling (Datasets/Assets, Airflow 2.4+): the producer task declares outlets. The consumer DAG uses schedule=[dataset]. It runs when the data is updated, and you need no sensors.

Templated operator fields are filled in at runtime with Jinja. This adds run details to your code.

Use several layers of monitoring:

Cron only starts a command at a set time. Airflow adds more. It has dependencies between tasks, retries and alerts. It has backfills and a UI with logs and history. It uses the logical date as a parameter. It can scale to many workers.

Pools limit how many tasks can run at the same time against a shared resource, such as a database or an API. The limit works across all DAGs. A pool has a number of slots. Each task takes slots (pool_slots) while it runs.

• depends_on_past=True: a task instance runs only if the same task succeeded in the previous DAG run.
• wait_for_downstream=True: a task instance also waits until the tasks right after it in the previous run have succeeded.

Define params with default values on the DAG. Override them when you trigger the DAG by hand. You can use the UI form, the CLI --conf, or the API.

Do not put passwords in DAG files or plain Variables. Use Connections with a secrets backend (AWS Secrets Manager, GCP Secret Manager, HashiCorp Vault). Then Airflow looks up the secrets at runtime.

The executor decides how tasks are run.

Celery starts tasks fast, but you must manage the workers all the time. Kubernetes gives clean isolation and autoscaling. But each task pays the pod start-up time. There is also CeleryKubernetes, which uses both. Recent versions can also use several executors at once.

What they are checking: Can you choose the right way to scale?

A DAG run covers one data interval. It starts only after that interval has ended.

Example: a @daily DAG has the data interval 2025-10-01 00:00 to 2025-10-02 00:00. It starts at 2025-10-02 00:00. Its logical date (ds) is 2025-10-01, the start of the interval. This is because the run processes the data of that interval.

• start_date: the start of the first interval. Nothing runs before it.
• schedule: a cron string, a preset (@daily) or a timetable.
• Templated values such as {{ data_interval_start }} and {{ ds }} help tasks fetch the right slice of data.

So a new DAG with today's start_date seems not to run. The first interval has not finished yet.

What they are checking: This is one of the most confusing Airflow ideas. Can you explain it clearly?

An idempotent task gives the same result if you run it once or many times for the same logical date. So retries and backfills are safe.

• Use {{ ds }} or the data interval as a parameter. Never use datetime.now().
• Overwrite the partition for that date, or use upsert/MERGE. Do not just append.
• Write to a temporary place, then swap it in one step (atomic).
• Delete and insert the date's data in one transaction.
• Avoid side effects that cannot be repeated, or protect them with a check.

DELETE FROM fact_sales WHERE load_date = '{{ ds }}';
INSERT INTO fact_sales SELECT * FROM stage WHERE load_date = '{{ ds }}';

What they are checking: Do you know the key design rule?

• Keep top-level DAG code light. No API calls, DB queries or heavy imports at parse time.
• Make tasks idempotent and atomic. Use logical dates and templates.
• Use small tasks with clear limits, not one giant function.
• Keep Airflow as the orchestrator. Push heavy work to Spark, warehouses or containers.
• Keep...

A trigger rule decides when a task runs, based on the states of its upstream tasks. The default is all_success.

Dynamic task mapping (Airflow 2.3+) creates a changing number of task instances at runtime. The number depends on the output of an upstream task. You use expand.

Test in layers:

• parallelism (core config): the most task instances that can run at once in the whole installation.
• max_active_tasks (per DAG): the most task instances that can run at once for one DAG, across its runs.
• max_active_runs (per DAG): the most DAG runs at once. It helps with backfills and stops runs from overlapping.
• max_active_tis_per_dag (per task): the limit for one task across runs.
• worker_concurrency (Celery): tasks for each worker.
• Pools: a limit for each resource.

A hook is a reusable way to talk to an outside system (connection handling, API calls). Examples are S3Hook and PostgresHook. An operator is a task template. It usually uses a hook to do its job.

Deferrable operators give a long wait to the triggerer. This is a background (async) process that runs many waiting triggers in one event loop. While the task is deferred, it frees its worker slot. So thousands of waits cost almost nothing.

The scheduler and workers are separate. The state lives in the metadata database. So a restart continues the work.

It stores DAGs, runs, task instances, XComs, connections, variables and references to logs. It is the single source of truth, so everything depends on it.

schedule_interval was the older argument. It took a cron string, a preset or a timedelta. In Airflow 2.4+, the single schedule argument replaces it. schedule accepts a cron expression, a preset like @daily, a timedelta, a list of datasets, or a Timetable object. Custom timetables handle rules that cron cannot write, such as 'every business day' or odd calendars.

Work from the file toward the scheduler.

Check these in order:

Queued means the scheduler gave the task to the executor, but no worker has started it.

Fix the cause. Then clear only the failed task, and its downstream tasks if you want, in the UI or CLI. Airflow reruns just those task instances.

Use four idempotent tasks, keyed on the logical date. Airflow orchestrates them, and other systems do the work.

Use a backfill, or clear the past runs, with limits, on idempotent tasks.

The task process may have died without telling Airflow. Or it may be waiting on something forever.

Parse time is the time spent running your DAG files. So cut the work done at module level.