Python Interview Questions and Answers for Data Engineers

All three combine DataFrames, but in different ways.

• pd.merge(left, right, on=..., how=...): works like a SQL join on column values. how can be inner, left, right or outer.
• df.join(other): a shortcut that joins on the index by default.
• pd.concat([df1, df2], axis=0|1): stacks frames on top of each other (rows) or side by side (columns). It lines them up by index. It does not match keys.

import pandas as pd
c = pd.DataFrame({"id": [1, 2], "name": ["A", "B"]})
o = pd.DataFrame({"id": [1, 1, 3], "amt": [10, 20, 30]})
print(pd.merge(c, o, on="id", how="left"))
print(pd.concat([c, c], ignore_index=True))

Use validate="one_to_many" in merge to catch duplicate keys by mistake.

What they are checking: Everyday pandas joins

groupby splits the data by one or more keys, runs an aggregation on each group, and joins the results back (split, apply, combine).

import pandas as pd
df = pd.DataFrame({"dept": ["A", "A", "B"], "sal": [10, 20, 30]})
out = df.groupby("dept").agg(total=("sal", "sum"), avg=("sal", "mean"), n=("sal", "size")).reset_index()
print(out)

Named aggregation (above) gives clean column names. size counts all rows, including nulls. count counts only non-null values. Use transform when you want a result with the same shape as the original rows. For example: df["pct"] = df.sal / df.groupby("dept").sal.transform("sum").

What they are checking: Core pandas skills

First measure the missing values. Then decide for each column whether to drop, fill or flag them.

import pandas as pd, numpy as np
df = pd.DataFrame({"a": [1, np.nan, 3], "b": ["x", None, "z"]})
print(df.isna().sum())               # count nulls per column
df2 = df.dropna(subset=["a"])         # drop rows
df3 = df.fillna({"a": df["a"].median(), "b": "unknown"})
df["a"] = df["a"].ffill()              # forward fill for time series

The right choice depends on what the value means. A missing amount might be 0. A missing key should be rejected. Keep an is_missing flag if the gap carries meaning. Never fill values silently in a pipeline. Log how many values you changed.

What they are checking: Data cleaning judgment

They differ in order, whether you can change them, and how fast lookup is.

What they are checking: Python basics

A generator gives values one at a time, only when asked (lazily). It does not build the whole collection in memory.

def read_lines(path):
    with open(path) as f:
        for line in f:
            yield line.rstrip("\n")

def parse(lines):
    for ln in lines:
        yield ln.split(",")

for row in parse(read_lines("data.csv")):
    pass  # process one row at a time

Chained generators make a streaming pipeline that uses a constant amount of memory. You can process files bigger than RAM. There are two downsides. A generator can be read only once. You also cannot use an index on it.

What they are checking: Memory-efficient processing

Use pandas with the pyarrow engine (or fastparquet). You can also use PyArrow directly.

import pandas as pd
df = pd.DataFrame({"id": [1, 2], "city": ["KTM", "PKR"]})
df.to_parquet("cities.parquet", engine="pyarrow", compression="snappy")
back = pd.read_parquet("cities.parquet", columns=["id"])

Parquet stores data by column, compresses it, and keeps the types. So reading only some columns is fast. For partitioned datasets use df.to_parquet("out/", partition_cols=["city"]). For files bigger than memory, read one row group at a time with pyarrow.parquet.ParquetFile(...).iter_batches().

What they are checking: Column-based file handling

Sort by the timestamp. Then call drop_duplicates on the key and keep the last row.

import pandas as pd
df = pd.DataFrame({"id": [1, 1, 2], "ts": ["2025-01-01", "2025-02-01", "2025-01-05"], "v": ["a", "b", "c"]})
latest = (df.sort_values("ts")
            .drop_duplicates(subset="id", keep="last"))
print(latest)

Another way is groupby: df.loc[df.groupby("id")["ts"].idxmax()]. Convert timestamps with pd.to_datetime first. Otherwise, odd formats sort as plain text.

What they are checking: Common coding task

Create a client or resource. Then use get_object / put_object, or the helpers upload_file / download_file.

Split the work into three functions: extract, transform and load. This makes each step easy to test. Log the row counts.

loc selects by label (index values and column names). iloc selects by integer position.

pivot changes long data to wide data without aggregating. It needs unique index and column pairs. pivot_table does the same, but it also aggregates duplicates. melt changes wide data to long data.

The Global Interpreter Lock (GIL) is a lock in CPython. It lets only one thread run Python bytecode at a time. It keeps reference counts safe.

import pandas as pd

def csv_to_parquet(src, dst, min_amount=100):
    df = pd.read_csv(src, usecols=["id", "amount", "date"], parse_dates=["date"])
    out = df[df["amount"] >= min_amount]
    out.to_parquet(dst, index=False)
    return len(df), len(out)

Add checks: the file exists and the expected...

A shallow copy makes a new container, but it points to the same inner objects. A deep copy copies everything, level by level.

== compares values (it calls __eq__). is compares identity: do both names point to the same object?

A list comprehension builds the whole list at once, with square brackets. A generator expression, with parentheses, gives items lazily.

Use pd.json_normalize to flatten nested dictionaries. Use record_path for nested lists.

Adding a row at a time to a DataFrame copies the whole frame each time. This is O(n squared), which is very slow. DataFrame.append was removed in pandas 2.0. Instead, collect the pieces in a list and call pd.concat once. Or build the frame from a list of dicts.

Never hard-code or commit credentials. Read settings from the environment or a secrets manager.

Type hints document the expected types, and tools such as mypy can check them. Dataclasses write the repeated setup code for classes that mostly hold data.

A module is a single .py file. A package is a directory of modules. It has an __init__.py file, or it is a namespace package. A script is a file you run directly.

Use DataFrame.to_sql with a SQLAlchemy engine. Write in chunks.

import pandas as pd
from sqlalchemy import create_engine
engine = create_engine("sqlite:///demo.db")
df = pd.DataFrame({"id": [1, 2], "amt": [10.5, 20.0]})
df.to_sql("sales", engine, if_exists="append", index=False, chunksize=10_000, method="multi")

Notes: if_exists can be fail, replace or append. to_sql only inserts rows. For upserts (update or insert), load into a staging table. Then run INSERT ... ON CONFLICT or MERGE in SQL. For large PostgreSQL loads, COPY through psycopg is much faster. Keep credentials in environment variables or a secrets manager.

What they are checking: Loading to a database

Use smaller dtypes and load less data.

• Make numbers smaller: pd.to_numeric(s, downcast="integer").
• Turn strings with few distinct values into category.
• Use nullable or string[pyarrow] dtypes for text.
• Read only the columns you need with usecols.
• Delete intermediate DataFrames and read in chunks.

Vectorized operations run in compiled code on whole columns. They are the fastest. map and apply call a Python function for each element or row, so they are much slower.

Because of the GIL, threads do not run Python code in parallel. So use threads for I/O-bound work and processes for CPU-bound work.

Use a paginator. It follows the continuation tokens for you.

Parse to datetime. Make the time zone explicit. Convert only at the edges.

Use the logging module with levels. Catch specific exceptions. For each error, decide whether to retry, skip or fail.

A decorator wraps a function to add behavior. A context manager handles setup and cleanup around a with block.

Write transformations as pure functions. They take a DataFrame and return a DataFrame. Then test them with small hand-made inputs using pytest.

Downloads wait on the network (I/O-bound), so use a thread pool.

Read it in chunks. Aggregate or write out each chunk. Never hold all the data at once.

import pandas as pd
totals = {}
for chunk in pd.read_csv("big.csv", chunksize=500_000, usecols=["region", "amount"],
                         dtype={"region": "category"}):
    part = chunk.groupby("region")["amount"].sum()
    for k, v in part.items():
        totals[k] = totals.get(k, 0) + v
print(totals)

Also do these things. Load only the columns you need (usecols). Choose smaller dtypes. Parse dates explicitly. Convert the file to Parquet once. If the data keeps growing, move to DuckDB, Polars (lazy mode), Dask or Spark.

What they are checking: Practical large-file handling

Validate at the edges: right after reading and right before writing. Check the schema, nulls, uniqueness, ranges and links between tables (referential integrity). Fail the job, or set bad rows aside.

import pandas as pd
df = pd.DataFrame({"id": [1, 2, 2], "age": [25, -3, 40]})
errors = []
if df["id"].duplicated().any():
    errors.append("duplicate ids")
bad = df[~df["age"].between(0, 120)]
if not bad.empty:
    errors.append(f"{len(bad)} rows with invalid age")
print(errors)

Useful libraries: Pandera, Great Expectations, Pydantic (row by row) and dbt tests. Good practice: write rejected rows to a quarantine table with a reason. Track row counts in and out. Alert when the failure rate crosses a limit. Do not drop data silently.

What they are checking: Reliability mindset

Idempotent means that running a job twice leaves the same final state as running it once.

Profile first (%timeit, cProfile). Then fix the biggest cost.

Fix the data first. Then change the design so a rerun does no harm.

Load the batch into a staging table. Then merge it into the target table with one SQL statement, inside a transaction.

Loop until there is no next page. Reuse one session. Respect rate limits. Retry temporary errors with backoff (wait longer after each failure).

Make both sides smaller first. Then merge. Watch out for duplicate keys.

Save the highest value of a column that only grows (such as updated_at or an id) from the last successful run. Then query only the rows beyond it. This saved value is called a watermark.