Deal of The Day! Hurry Up, Grab the Special Discount - Save 25% - Ends In 00:00:00 Coupon code: SAVE25
Welcome to Pass4Success

- Free Preparation Discussions

Databricks Certified Associate Developer for Apache Spark 3.5 Exam - Topic 4 Question 20 Discussion

Posted by Val on Aug 12, 2026 at 08:00 PM

A Spark DataFrame df is cached using the MEMORY_AND_DISK storage level, but the DataFrame is too large to fit entirely in memory.

What is the likely behavior when Spark runs out of memory to store the DataFrame?

Suggested Answer: C

When using the MEMORY_AND_DISK storage level, Spark attempts to cache as much of the DataFrame in memory as possible. If the DataFrame does not fit entirely in memory, Spark will store the remaining partitions on disk. This allows processing to continue, albeit with a performance overhead due to disk I/O.

As per the Spark documentation:

'MEMORY_AND_DISK: It stores partitions that do not fit in memory on disk and keeps the rest in memory. This can be useful when working with datasets that are larger than the available memory.'

--- Perficient Blogs: Spark - StorageLevel

This behavior ensures that Spark can handle datasets larger than the available memory by spilling excess data to disk, thus preventing job failures due to memory constraints.


Get Premium Databricks Certified Associate Developer for Apache Spark 3.5 Questions as Interactive Web-Based Practice Test or PDF Real Exam Format Detailed Explanations Instant Access Get Your Practice Test
18 comments · newest first
LA
Lauryn 3 days ago
D is interesting, but I feel it complicates things too much.
FR
Freeman 8 days ago
A sounds tempting, but it seems inefficient to duplicate data.
JU
Justine 14 days ago
I agree with Martha. C shows how Spark handles memory efficiently.
MA
Martha 19 days ago
I think C is the right choice. It makes sense to spill to disk when memory is full.
LO
Louisa 24 days ago
B is just wrong; Spark doesn’t split evenly like that.
CH
Chuck 29 days ago
Definitely not A, that would waste resources.
DE
Destiny 1 month ago
Wait, so it doesn’t duplicate the DataFrame? That’s surprising!
MI
Mitsue 1 month ago
I think D sounds reasonable too, but it’s not how Spark works.
MA
Magda 1 month ago
C is the right answer! Spark spills to disk when memory is full.
DA
Dacia 2 months ago
Definitely not A or B, those don't make sense in this context.
EL
Elvera 2 months ago
Wait, so it just spills over? That’s surprising!
HO
Honey 2 months ago
D sounds plausible, but I don't think that's how Spark handles it.
VA
Valda 2 months ago
I thought it would duplicate the DataFrame, but that's not how it works.
NE
Nettie 2 months ago
C is the right answer! It spills to disk when memory is full.
LE
Lennie 3 months ago
I remember that Spark tries to optimize performance, so C seems right. It makes sense that it would spill to disk when memory runs out.
SH
Sherman 3 months ago
I’m leaning towards A, but it doesn’t quite match what I recall about how Spark handles memory. I need to double-check the caching behavior.
BE
Beckie 3 months ago
I feel like I’ve seen a question like this before, and I think it was about how Spark manages memory. C sounds familiar, but D also seems plausible.
PA
Pa 3 months ago
I think the answer might be C, but I'm not entirely sure. I remember something about spilling data to disk when memory is full.
0/2000 characters

Save Cancel