Spark Associate Cheat Sheet: Key Facts

The short version

This sheet is a fast review of the Databricks Spark Associate Developer exam: the format, then each of the seven domains with key ideas, look-alike pairs, decision rules and traps. It is written in our own words and does not cover everything on the exam. For the full topic list, use the Databricks exam guide, which is the authority.

Exam facts at a glance

Item	Detail (as published by Databricks)
Scored questions	45, multiple choice
Possible unscored items	Up to 10, not marked
Time limit	90 minutes
Code language	Python
Test aids	None, including API documentation
Delivery	Online proctored or test center
Passing score	Not published
Validity	2 years
Retake wait	14 days

Details are on the exam format page.

Domain weightings

Domain	Weight
Architecture and Components	20%
Spark SQL	20%
DataFrame/DataSet API	30%
Troubleshooting and Tuning	10%
Structured Streaming	10%
Spark Connect	5%
Pandas API on Spark	5%

1. Architecture and Components

Key concepts

The driver plans and coordinates. Executors run tasks on partitions. The cluster manager hands out resources.
Hierarchy: application, jobs, stages, tasks. An action starts a job. A shuffle splits a job into stages. One task works on one partition.
Transformations are lazy and only build a plan. Actions trigger work.
Narrow transformations (`filter`, `select`, `map`-like) need no shuffle. Wide ones (`groupBy`, most joins, `distinct`, `repartition`) do.

Look-alike pairs

`cache()` uses the default storage level. `persist()` lets you choose one.
Transformation returns a DataFrame. Action returns a value or writes data.

Decision rules

If the scenario says “reused several times”, think caching.
If it says “stage boundary” or “new stage”, think shuffle.

Traps: thinking `collect()` is cheap on big data; thinking `cache()` fills the cache at once.

2. Spark SQL

Key concepts

Save modes: append, overwrite, ignore, errorifexists (the default).
`partitionBy` on a writer sets the folder layout on disk.
A temporary view lives for the session; it stores no data.
You can query files directly in SQL by format and path.

Look-alike pairs

Temporary view vs persistent table: session-only name vs saved table.
`spark.sql(…)` returns a DataFrame; a DataFrame can be registered as a view.

Decision rules

“Replace what is there” means overwrite. “Add to it” means append.
“Keep existing data and skip silently” means ignore.

Traps: using `repartition` when the question is about folder layout.

3. DataFrame API

Key concepts

`withColumn` adds or replaces. `withColumnRenamed` renames and does nothing if the name is missing. `drop` removes.
Joins: inner, left, cross, broadcast hint for a small side.
`groupBy().agg(…)` for aggregates. `approx_count_distinct` trades accuracy for speed.
Schemas can be inferred or declared. Declared is safer and faster for large files.

Look-alike pairs

`union` (by position) vs `unionByName` (by name).
`distinct()` (whole row) vs `dropDuplicates(subset)`.
`na.drop()` (any null) vs `na.drop(“all”)` (all null).
`orderBy` and `sort` are aliases.
Python UDF (opaque to the optimizer) vs built-in function (preferred) vs pandas UDF (batch-wise).
Broadcast variable (read-only copy on executors) vs accumulator (counter written by tasks, read on the driver).

Decision rules

If a built-in function exists, choose it over a UDF.
If one join side is small, think broadcast.

Traps: reading accumulator values before an action; forgetting that a method returns a new DataFrame and the original is unchanged.

4. Troubleshooting and Tuning

Key concepts

`repartition(n)` shuffles and can increase or decrease partitions. `coalesce(n)` only decreases and avoids a full shuffle.
Data skew: one key holds far more rows, so one task runs much longer. The Spark UI shows it as a long straggler task.
Adaptive Query Execution adjusts plans at runtime, for example by coalescing small shuffle partitions and handling skewed joins.
`spark.sql.shuffle.partitions` defaults to 200.
Driver and executor logs help diagnose out-of-memory errors.

Decision rules

“Fewer output files” suggests `coalesce`.
“One task far slower than the rest” suggests skew.
“Driver out of memory after an action” suggests `collect()` on too much data.

5. Structured Streaming

Key concepts

A streaming query is a micro-batch query that keeps running. A checkpoint location stores progress.
Output modes: append (new rows only), update (changed rows), complete (whole result table).
A watermark tells Spark how late data may arrive, so old state can be dropped.
Windows group events by time. Deduplication can be done with or without a watermark; without one, state grows.

Traps: thinking a watermark defines windows; forgetting that `complete` mode needs an aggregation.

6. Spark Connect

A thin client talks to a remote Spark server, with the client separated from the server.
Deployment modes: client (driver on the submitting machine), cluster (driver inside the cluster), local (everything in one JVM).

7. Pandas API on Spark

Pandas-style code that runs distributed.
A pandas UDF handles batches as pandas objects, which is usually faster than a row-at-a-time UDF.
Index and row order differ from single-machine pandas.

Last-day checklist

Reread your list of look-alike pairs one more time.
Check the exam guide date once more.
Run the system check if you take it online, and clear your desk.
Sleep. A rested brain reads code snippets better than a crammed one.
Plan your time: about two minutes per question. Flag a hard one and move on.

More guides for DATABRICKS-SPARK-ASSOCIATE

Next step

Test what you remember with the sample questions, or go deeper with domains explained. Back on the overview, you can find all pages. For timed practice, try our Databricks Spark Associate practice tests.

Frequently asked questions

Can I rely on this sheet alone?

No. It is a review aid, not a full syllabus. Use the Databricks exam guide as your checklist.

Will the exam ask about Scala?

Databricks says code snippets are in Python.

Try these practice exams