PySpark Interview Questions and Answers

Apache Spark is a distributed engine that processes data in memory. It works for batch jobs, SQL, streaming and machine learning.

Why it is usually faster than MapReduce:

• It keeps temporary data in memory. MapReduce writes every stage to disk (HDFS).
• It builds a DAG (a graph of steps) for the whole job and optimizes it. MapReduce has fixed map then reduce steps.
• Tasks run as threads inside long-lived executors, so no new JVM starts for each task.
• Catalyst and Tungsten optimize SQL plans and memory layout.

Spark still spills to disk when data does not fit. It can read and write HDFS, S3 and other stores. It is only a compute engine, not a storage system.

What they are checking: Whether you know the real technical reasons, not just the slogan 'in memory'

A Spark application has one driver, many executors, and a cluster manager that gives out resources.

• Driver: runs main() and creates the SparkSession. It turns your code into a logical plan and a DAG, splits it into stages and tasks, and schedules them.
• Executors: JVM processes on worker nodes. They run tasks, hold cached data, and write shuffle files.
• Cluster manager: gives out resources. Examples are Standalone, YARN, Kubernetes (and Mesos in older setups).

Flow: the driver asks the cluster manager for executors. Then it sends tasks to them and collects the results. If the driver dies, the application dies. Losing an executor is fine, because lost tasks are recomputed from lineage.

What they are checking: Whether you can say who plans the work and who runs it

A transformation describes a new dataset made from an existing one. It is lazy, so it does not run yet. An action starts the work and returns a result or writes output.

• Transformations: select, filter, map, withColumn, join, groupBy, union. They return a new DataFrame or RDD and run nothing.
• Actions: count, collect, show, take, first, write, foreach. Each one starts a job.

df2 = df.filter(df.age > 30)   # nothing runs
print(df2.count())             # job runs here

Transformations are also split into narrow and wide (shuffle) types. Each action runs the whole chain again unless the data is cached.

What they are checking: The basic words of lazy evaluation

Spark saves transformations as a plan. It runs them only when you call an action.

Why it matters:

• The optimizer sees the whole plan. It can push filters down, skip unused columns, and combine steps (Catalyst).
• Narrow transformations run together in one pass over the data.
• Nothing is computed for results you never use.

Things to remember: an error in a transformation may only show up at the action. Two actions on the same DataFrame compute it twice unless you use cache().

What they are checking: Do you know why Spark code runs late, and what that means for debugging

All three are distributed collections. They differ in how much structure and optimization they have.

In PySpark you almost always use DataFrames. They run on the optimized engine and do not serialize each row through Python. Use RDDs only when you need low-level control or have unstructured data. A DataFrame is a Dataset of Row.

What they are checking: Whether you know why DataFrames are the default

A narrow transformation needs data from only one input partition for each output partition. A wide transformation needs data from many partitions, so it causes a shuffle.

• Narrow: map, filter, select, withColumn, union, coalesce. No data moves, and the steps run together.
• Wide: groupBy, join (unless broadcast or co-partitioned), distinct, repartition, orderBy, reduceByKey.

Each wide transformation creates a stage boundary. Shuffles write to local disk and move data over the network. They are the main cost in most jobs, so optimize them first.

What they are checking: Do you see the link between transformation type, shuffle and stage boundary

A shuffle moves data between partitions so that rows with the same key end up together. Wide transformations cause it.

Why it costs a lot:

• The map side writes shuffle files to local disk.
• The reduce side fetches blocks over the network.
• Data must be serialized and compressed, and it may spill to disk.
• Skew (uneven data) can make a few tasks do most of the work.

How to reduce it: filter early and select only the columns you need. Use broadcast joins and pre-aggregate. Avoid extra repartition calls. Tune spark.sql.shuffle.partitions (default 200).

What they are checking: Do you know the main performance bottleneck and how to attack it

They are three levels of the execution plan.

• Job: created by one action (for example count() or write).
• Stage: a group of tasks that can run without a shuffle. A new stage starts at each shuffle.
• Task: the smallest unit. There is one per partition per stage, and it runs on one executor core.

Example: read -> filter -> groupBy -> count is one job with two stages. Stage 1 reads, filters and does a partial aggregation. Stage 2 reads the shuffled data and finishes the aggregation. The Spark UI shows all three, so start debugging there.

What they are checking: Whether you can match code to what you see in the Spark UI

Both change the number of partitions. repartition does a full shuffle and can increase or decrease the number. coalesce merges existing partitions without a full shuffle and can only decrease the number.

• repartition(n) gives partitions of roughly equal size. It can also take columns, like repartition(200, "user_id").
• coalesce(n) is cheaper, but partitions may be uneven.

df.repartition(100)      # shuffle, even sizes
df.coalesce(10)          # no shuffle, fewer files

Typical use: coalesce before writing, to reduce small files. repartition is for more parallelism, or to put the same keys together before a join or partitioned write.

What they are checking: A classic tuning question with a trade-off

A broadcast join sends a small table to every executor. The large table is then joined locally, with no shuffle.

from pyspark.sql.functions import broadcast
result = big_df.join(broadcast(small_df), "country_code")

Use it when one side is small enough to fit in executor memory. Spark does it automatically below spark.sql.autoBroadcastJoinThreshold (default 10 MB). The other choice is a sort-merge join, which shuffles both sides. Do not broadcast a large table. It can cause out-of-memory errors on the driver or executors.

What they are checking: The most common join optimization

cache() is persist() with the default storage level. persist() lets you choose the level.

• DataFrame cache() uses MEMORY_AND_DISK.
• Other options include persist(StorageLevel.DISK_ONLY), MEMORY_ONLY and MEMORY_AND_DISK_SER.

Use spark.read and df.write with a format.

The DAG (directed acyclic graph) is the graph of steps made from your transformations. The driver's DAG scheduler builds it when you call an action.

Both are RDD wide operations on key-value pairs. reduceByKey first combines values inside each partition (map-side combine). Then it shuffles only the partial results. groupByKey shuffles every value and then groups them. This uses much more network and memory.

Parquet is columnar, compressed and stores a schema. Analytics queries read less data and run faster.

A UDF is a user-defined Python function that you use on DataFrame columns.

Spark supports inner, left (outer), right (outer), full (outer), left_semi, left_anti, and cross.

• left_semi: rows of the left side that have a match. It returns only left columns, with no duplicates from the right.
• left_anti: rows of the left side with no match. Good for 'new records only'.

Use isNull, na.fill, na.drop and coalesce.

collect() pulls all rows from all executors into the driver's memory as a Python list. On a large DataFrame this can cause a driver out-of-memory error and a long network transfer.

map returns exactly one output element for each input element. flatMap returns zero or more output elements for each input, and joins them into one flat list.

The Spark UI shows how the job ran. Use port 4040 for a running app, and the history server afterwards.

AQE changes the plan while the job runs, using real statistics collected after each shuffle stage. It is on by default since Spark 3.2 (spark.sql.adaptive.enabled).

Skew means some keys have far more rows than others. A few tasks then run much longer and may run out of memory, while the rest sit idle.

How to spot it: on the stage page in the Spark UI, the max task time and shuffle read are far above the median.

First add up the revenue. Then rank the products inside each category with a window.

Spark picks one of these physical strategies:

Cache a DataFrame when it is expensive to compute and used by several actions or loops. Examples are ML loops, or a table reused by several later branches.

Catalyst is the query optimizer of Spark SQL. It uses rules and also cost estimates. It turns a query into an efficient physical plan in four phases:

Salting spreads one hot key over several partitions. Add a random number to the key on the big side. Then copy the small side once for every salt value.

partitionBy("col") writes data into one directory for each column value, such as year=2025/month=10/. Queries that filter on that column skip whole directories (partition pruning).

Aim for partitions of about 100 to 200 MB each. Also aim for at least 2 to 4 tasks per available core. This uses the whole cluster without too much scheduling work.

Spark tracks lineage, which is the chain of transformations that built each partition. If a partition or executor is lost, Spark recomputes only the lost partitions from the source.

Structured Streaming treats a stream as a table that keeps growing. It runs the same DataFrame API step by step in micro-batches.

A pandas UDF (vectorized UDF) gets and returns pandas Series or DataFrames in batches, using Apache Arrow. This avoids serializing row by row.

The mode decides where the driver runs.

They are open table formats. They add a transaction log and metadata on top of Parquet files in object storage.

Use the JDBC writer, with the driver jar on the classpath.

Treat driver out-of-memory (OOM) and executor OOM as two separate problems.

Driver: collect() or toPandas() on a large result, a huge broadcast, or metadata for too many partitions. Fix: write to storage, or use take or limit. Raise spark.driver.memory only as a last step.

Use the Spark UI before you change anything.

dropDuplicates removes exact or key-based duplicates. It does not let you choose which row stays. For 'latest per key', use a window function.

Small files overload the file system and slow down later reads. Fix it when writing, and with compaction.

Start from the workload and the node size of the cluster. Then tune using the Spark UI.

First find out why executors die. Then reduce the pressure.

Process only new or changed data, and make the job idempotent (running it twice gives the same result).

The difference is usually the environment, the data size, or serialization.