Spark Associate Domains Explained

The short version

The exam has seven domains. The DataFrame API is 30%, architecture and Spark SQL are 20% each, and troubleshooting and tuning and Structured Streaming are 10% each. Spark Connect and the pandas API on Spark are 5% each. The weightings and topics come from the Databricks exam guide that covers the live version as of October 8, 2026. The explanations and traps below are our own general guidance, not vendor statements.

1. Apache Spark Architecture and Components (20%)

What it covers. The role of the driver, the executors, the cluster manager, cores and memory. The execution hierarchy, which runs from application to jobs, stages, and tasks. How partitions and shuffles work. Transformations, actions, and lazy evaluation. DataFrame and Dataset concepts, the SparkSession lifecycle, caching, storage levels, and garbage collection. The Spark modules, including Spark SQL, Structured Streaming, the pandas API on Spark, and MLlib.

In practice. When you call `filter`, nothing runs. When you call `count`, Spark builds a plan, splits it into stages at shuffle boundaries, and sends tasks to executors. Knowing where the boundary falls tells you why one job has two stages and another has five.

Typical traps.

  • Believing a transformation does work immediately. It only extends the plan.
  • Confusing narrow and wide transformations. A wide one, such as a join or `groupBy`, needs a shuffle.
  • Thinking `cache()` runs at once. It marks the data, and the first action fills it.
  • Mixing up the driver and the executors, for example saying `collect()` pulls data to an executor.

What to practice. Run a small job with the Spark UI open and match stages to the lines of code. Cache a DataFrame, run two actions, and see what changes.

2. Using Spark SQL (20%)

What it covers. Reading and writing sources such as files and JDBC, including overwrite and partitioning by column. Running SQL directly on ORC, JSON, CSV, text, and Delta files. Save modes. Saving to persistent tables with sorting and partitioning. Registering DataFrames as temporary views.

In practice. A DataFrame and a SQL query are two front ends to the same engine. You can create a view from a DataFrame, query it with `spark.sql`, and get a DataFrame back.

Typical traps.

  • Forgetting the save modes: append, overwrite, ignore, and error if exists (the default).
  • Treating a temporary view as stored data. It is only a name for a query plan and disappears with the session.
  • Mixing up `partitionBy` on a writer, which sets the folder layout, with repartitioning in memory.
  • Assuming SQL and the DataFrame API give different performance for the same logic.

What to practice. Write the same transformation both ways. Write a partitioned Parquet output with each save mode and look at the folder that results.

3. Developing DataFrame/DataSet API Applications (30%)

What it covers. Adding, dropping, splitting, and renaming columns, filtering, and exploding arrays. Deduplication and validation. Aggregates such as count, approximate distinct count, mean, and summary. Date handling, such as converting epoch values to date strings and extracting parts. Joins (inner, left, broadcast, multiple keys, cross), unions, reading and writing with schemas, sorting, printing a schema, converting to lists. User-defined functions, with or without stateful operators, including StateStores, plus broadcast variables and accumulators.

In practice. This is the daily work of a Spark developer, and the largest domain. Expect code on the screen and a question about what it returns.

Typical traps.

  • `union` matches columns by position, while `unionByName` matches by name.
  • `distinct` looks at whole rows, while `dropDuplicates` can take a subset.
  • `withColumnRenamed` quietly does nothing if the old name does not exist.
  • A Python UDF is a black box to the optimizer, and a built-in function is usually the better choice.
  • Accumulator values are reliable only after an action has run.

What to practice. Keep a one-page sheet of look-alike calls, and run each pair on a tiny DataFrame.

4. Troubleshooting and Tuning (10%)

What it covers. Tuning with partitioning, repartitioning, coalescing, spotting data skew, and reducing shuffles. Adaptive Query Execution and its benefits. Reading driver and executor logs to diagnose out-of-memory errors and underused clusters.

In practice. A job is slow or fails. You look at the Spark UI for a task that takes far longer than its siblings, which points to skew, or at executor logs for memory errors.

Typical traps.

  • Using `repartition` when `coalesce` is enough. Coalesce only reduces the partition count and avoids a full shuffle.
  • Fixing a symptom (more memory) instead of a cause (a skewed key).
  • Forgetting that the default for shuffle partitions is 200 and may not suit your data size.

What to practice. Create a skewed dataset and watch the long task in the Spark UI.

5. Structured Streaming (10%)

What it covers. The programming model, micro-batch processing, exactly-once semantics, and fault tolerance. Writing streaming DataFrames with output modes and sinks. Selection, projection, windows, and aggregation. Deduplication with and without a watermark.

In practice. A streaming query is a batch query that never ends. Spark processes new data in small batches and keeps state between them. A checkpoint location records progress so a restarted query resumes where it stopped.

Typical traps.

  • Mixing up the output modes: append, complete, and update.
  • Thinking a watermark creates windows. It only tells Spark how long to keep state for late data.
  • Forgetting the checkpoint location.

What to practice. Stream files from a folder into the console and change the output mode.

6. Using Spark Connect to deploy applications (5%)

What it covers. The features of Spark Connect and the deployment modes: client, cluster, and local.

In practice. Spark Connect splits the client from the server, so a thin client can talk to a remote Spark session over gRPC. A connection string starts with `sc://`.

Typical traps. Confusing the Connect client with a classic driver, and mixing up where the driver runs in client and cluster mode.

What to practice. Write down, for each mode, where the driver runs and who owns it.

7. Using Pandas API on Spark (5%)

What it covers. The advantages of the pandas API on Spark and how to create and invoke a pandas UDF.

In practice. You keep pandas-style code, and Spark runs it in parallel. A pandas UDF processes a batch of values as a pandas Series.

Typical traps. Treating pandas on Spark as plain pandas. Row order and index behave differently on a distributed dataset. Another trap is mixing up a pandas UDF with a regular row-at-a-time UDF.

What to practice. Convert a short pandas script to the pandas API on Spark and note what changed.

More guides for DATABRICKS-SPARK-ASSOCIATE

Next step

Try your weakest domain first in our sample questions, or read the study plan to fit these domains into a schedule. The Databricks Spark Associate practice tests tag every question with one of these seven domains.

Try these practice exams