Databricks Certified Data Engineer Professional Exam - Topic 6 Question 15 Discussion

Actual exam question for Databricks's Databricks Certified Data Engineer Professional exam

Question #: 15
Topic #: 6

[All Databricks Certified Data Engineer Professional Questions]

The data engineer is using Spark's MEMORY_ONLY storage level.

Which indicators should the data engineer look for in the spark UI's Storage tab to signal that a cached table is not performing optimally?

ASize on Disk is> 0

BThe number of Cached Partitions> the number of Spark Partitions

CThe RDD Block Name included the '' annotation signaling failure to cache

DOn Heap Memory Usage is within 75% of off Heap Memory usage

Show Suggested Answer

Suggested Answer: C

In the Spark UI's Storage tab, an indicator that a cached table is not performing optimally would be the presence of the _disk annotation in the RDD Block Name. This annotation indicates that some partitions of the cached data have been spilled to disk because there wasn't enough memory to hold them. This is suboptimal because accessing data from disk is much slower than from memory. The goal of caching is to keep data in memory for fast access, and a spill to disk means that this goal is not fully achieved.

by Lazaro at Jul 06, 2024, 01:14 PM

Limited Time Offer

25%

Off

Get Premium Databricks Certified Data Engineer Professional Questions as Interactive Web-Based Practice Test or PDF

Contribute your Thoughts:

Submit Cancel

Fatima

3 months ago

I think the cached partitions being greater is a good sign sometimes.

upvoted 0 times

...

Antione

3 months ago

On Heap Memory Usage being high is a red flag for sure.

upvoted 0 times

...

Henriette

3 months ago

Wait, is it normal for RDD Block Names to show cache failures?

upvoted 0 times

...

Iluminada

4 months ago

Totally agree, cached partitions should match Spark partitions.

upvoted 0 times

...

Tandra

4 months ago

Size on Disk should be 0 for MEMORY_ONLY!

upvoted 0 times

...

Raylene

4 months ago

I practiced a question similar to this, and I think the On Heap Memory Usage being too close to Off Heap Memory could signal potential issues with caching performance.

upvoted 0 times

...

Joaquin

4 months ago

I vaguely recall something about RDD Block Names showing failures to cache, so that might be a key indicator to look for in the Spark UI.

upvoted 0 times

...

Flo

4 months ago

I'm not entirely sure, but I feel like having more Cached Partitions than Spark Partitions could mean inefficiencies in how the data is being stored.

upvoted 0 times

...

Kendra

5 months ago

I think I remember that if the Size on Disk is greater than 0, it might indicate that the data isn't fully cached in memory, which could be a problem.

upvoted 0 times

...

Laquita

5 months ago

I've got this! The answer is B - the number of Cached Partitions should be less than or equal to the number of Spark Partitions.

upvoted 0 times

...

Tiffiny

5 months ago

I'm a bit confused by the options. I'll need to double-check the Spark documentation to make sure I understand the correct indicators.

upvoted 0 times

...

Merri

5 months ago

Okay, let's think this through step-by-step. The key is to identify the signs that the cached table is not performing optimally.

upvoted 0 times

...

Kati

5 months ago

Hmm, I'm not sure about this one. I'll need to carefully review the Spark UI storage tab to look for the right indicators.

upvoted 0 times

...

Frank

5 months ago

This question seems pretty straightforward. I think I can handle it.

upvoted 0 times

...

Norah

5 months ago

This seems like a straightforward question about LUN migration. I'll need to remember the key details about LUN migration and how it's used to move LUNs without disrupting service.

upvoted 0 times

...

Mabel

5 months ago

This looks like a straightforward question on social engineering attack vectors. I'm confident I can identify the three major ones.

upvoted 0 times

...

Anissa

2 years ago

I'm no data engineer, but I hear caching is like trying to remember where you left your car keys. If the Spark UI is confused, you know you've got a problem!

upvoted 0 times

Phillip

2 years ago

B) The number of Cached Partitions> the number of Spark Partitions

upvoted 0 times

...

Jettie

2 years ago

A) Size on Disk is> 0

upvoted 0 times

...

Raul

2 years ago

I feel like the answer is B. Anything that doesn't match the actual Spark partitions is probably not a good sign.

upvoted 0 times

Lizbeth

2 years ago

Yeah, it's important to keep an eye on those indicators in the Spark UI's Storage tab.

upvoted 0 times

...

Jerry

2 years ago

I agree, that would definitely be a sign that the cached table is not performing optimally.

upvoted 0 times

...

Frederica

2 years ago

I think the answer is B. If the number of Cached Partitions is greater than the number of Spark Partitions, it's not good.

upvoted 0 times

...

Lea

2 years ago

I think C) The RDD Block Name included the '' annotation signaling failure to cache is also a valid indicator of poor performance.

upvoted 0 times

...

Enola

2 years ago

But if the number of Cached Partitions is greater than the number of Spark Partitions, wouldn't that indicate a performance issue?

upvoted 0 times

...

Lemuel

2 years ago

I disagree, I believe the correct answer is A) Size on Disk is> 0.

upvoted 0 times

...

Georgeanna

2 years ago

This is a tough one, but I'm going to go with C. The '' annotation in the RDD Block Name is a clear indication of caching failure.

upvoted 0 times

Valentin

1 year ago

I agree with User1, C) The RDD Block Name included the '' annotation signaling failure to cache seems like the right choice

upvoted 0 times

...

Lachelle

1 year ago

I'm leaning towards B) The number of Cached Partitions> the number of Spark Partitions

upvoted 0 times

...

Jenelle

1 year ago

I think it's A) Size on Disk is> 0

upvoted 0 times

...

Latanya

2 years ago

I'm not sure, but I think it might be B) The number of Cached Partitions> the number of Spark Partitions

upvoted 0 times

...

Helaine

2 years ago

I disagree, I believe it's C) The RDD Block Name included the '' annotation signaling failure to cache

upvoted 0 times

...

Nell

2 years ago

I think it's A) Size on Disk is> 0

upvoted 0 times

...

Thaddeus

2 years ago

Hmm, I'm going with B. If the number of cached partitions is greater than the actual Spark partitions, that's a red flag.

upvoted 0 times

Yolando

2 years ago

Yeah, if the number of cached partitions is more than the Spark partitions, it's not performing optimally.

upvoted 0 times

...

Darci

2 years ago

I think B is the right indicator to look for.

upvoted 0 times

...

Enola

2 years ago

I think the answer is B) The number of Cached Partitions> the number of Spark Partitions.

upvoted 0 times

...

Christoper

2 years ago

D sounds like the right choice to me. If the on-heap and off-heap memory usage are out of balance, that's a sign of suboptimal caching.

upvoted 0 times

...

Marylin

2 years ago

I think the correct answer is C. The RDD Block Name with the '' annotation signals that the caching was unsuccessful.

upvoted 0 times

Scarlet

2 years ago

D) On Heap Memory Usage is within 75% of off Heap Memory usage

upvoted 0 times

...

Gwenn

2 years ago

C) The RDD Block Name included the '' annotation signaling failure to cache

upvoted 0 times

...

Mariko

2 years ago

B) The number of Cached Partitions> the number of Spark Partitions

upvoted 0 times

...

Clare

2 years ago

A) Size on Disk is> 0

upvoted 0 times

...

Vivienne

2 years ago

User2

upvoted 0 times

...

Layla

2 years ago

User1

upvoted 0 times

...