Discuss Databricks Databricks Certified Data Engineer Professional Exam Topic 6 Question 46

Which statement describes Delta Lake Auto Compaction?

AAn asynchronous job runs after the write completes to detect if files could be further compacted; if yes, an optimize job is executed toward a default of 1 GB.

BBefore a Jobs cluster terminates, optimize is executed on all tables modified during the most recent job.

COptimized writes use logical partitions instead of directory partitions; because partition boundaries are only represented in metadata, fewer small files are written.

DData is queued in a messaging bus instead of committing data directly to memory; all data is committed from the messaging bus in one batch once the job is complete.

EAn asynchronous job runs after the write completes to detect if files could be further compacted; if yes, an optimize job is executed toward a default of 128 MB.

Suggested Answer: E

This is the correct answer because it describes the behavior of Delta Lake Auto Compaction, which is a feature that automatically optimizes the layout of Delta Lake tables by coalescing small files into larger ones. Auto Compaction runs as an asynchronous job after a write to a table has succeeded and checks if files within a partition can be further compacted. If yes, it runs an optimize job with a default target file size of 128 MB. Auto Compaction only compacts files that have not been compacted previously. Verified Reference: [Databricks Certified Data Engineer Professional], under ''Delta Lake'' section; Databricks Documentation, under ''Auto Compaction for Delta Lake on Databricks'' section.

'Auto compaction occurs after a write to a table has succeeded and runs synchronously on the cluster that has performed the write. Auto compaction only compacts files that haven't been compacted previously.'

https://learn.microsoft.com/en-us/azure/databricks/delta/tune-file-size

Get Premium Databricks Certified Data Engineer Professional Questions as Interactive Web-Based Practice Test or PDF Real Exam Format Detailed Explanations Instant Access Get Your Practice Test

12 comments · newest first

Dexter 3 months ago

I lean towards B. Executing optimize before cluster termination is practical.

Noel 3 months ago

I feel option E is more accurate. 128 MB seems more efficient for compaction.

Mi 3 months ago

I think option A is correct. It makes sense for optimization after writing.

Becky 3 months ago

E is misleading, it should be 1 GB, not 128 MB!

Malcolm 3 months ago

D sounds off, I doubt data is queued in a messaging bus like that.

Renato 4 months ago

C is interesting, I didn't know logical partitions worked that way!

Amie 5 months ago

I think B is more accurate, optimize should run before cluster termination.

Jacquelyne 5 months ago

A is correct, it mentions the 1 GB default for optimization.

Lai 6 months ago

I don't think option D is correct because it mentions a messaging bus, which doesn't seem to fit with how Delta Lake handles data commits.

Jordan 6 months ago

I feel like option C is related to how optimized writes work, but I’m confused about the difference between logical and directory partitions.

Kirk 6 months ago

I remember practicing a question about how optimize works, but I can't recall if it runs before the cluster terminates or after the write completes.

Truman 7 months ago

I think Delta Lake Auto Compaction involves some kind of asynchronous job, but I'm not sure if it's 1 GB or 128 MB for the optimize job.

Databricks Certified Data Engineer Professional Exam - Topic 6 Question 46 Discussion