Discuss Databricks Databricks Certified Data Engineer Professional Exam Topic 2 Question 29

In order to prevent accidental commits to production data, a senior data engineer has instituted a policy that all development work will reference clones of Delta Lake tables. After testing both deep and shallow clone, development tables are created using shallow clone.

A few weeks after initial table creation, the cloned versions of several tables implemented as Type 1 Slowly Changing Dimension (SCD) stop working. The transaction logs for the source tables show that vacuum was run the day before.

Why are the cloned tables no longer working?

AThe data files compacted by vacuum are not tracked by the cloned metadata; running refresh on the cloned table will pull in recent changes.

BBecause Type 1 changes overwrite existing records, Delta Lake cannot guarantee data consistency for cloned tables.

CThe metadata created by the clone operation is referencing data files that were purged as invalid by the vacuum command

DRunning vacuum automatically invalidates any shallow clones of a table; deep clone should always be used when a cloned table will be repeatedly queried.

Suggested Answer: C

In Delta Lake, a shallow clone creates a new table by copying the metadata of the source table without duplicating the data files. When the vacuum command is run on the source table, it removes old data files that are no longer needed to maintain the transactional log's integrity, potentially including files referenced by the shallow clone's metadata. If these files are purged, the shallow cloned tables will reference non-existent data files, causing them to stop working properly. This highlights the dependency of shallow clones on the source table's data files and the impact of data management operations like vacuum on these clones. Reference: Databricks documentation on Delta Lake, particularly the sections on cloning tables (shallow and deep cloning) and data retention with the vacuum command (https://docs.databricks.com/delta/index.html).

Get Premium Databricks Certified Data Engineer Professional Questions as Interactive Web-Based Practice Test or PDF Real Exam Format Detailed Explanations Instant Access Get Your Practice Test

29 comments · newest first

Francesco 10 months ago

Wow, I didn't know vacuum could mess up clones like that!

Glendora 11 months ago

B sounds plausible too, but I lean towards C.

Annice 11 months ago

Wait, I thought shallow clones were safe?

Augustine 11 months ago

Totally agree, C is the right answer!

Chaya 11 months ago

Cloned tables lose track of data files after vacuum.

Rodolfo 12 months ago

I recall that shallow clones don't track changes like deep clones do, but I'm not confident if vacuuming affects them as much as D suggests.

Lili 12 months ago

I thought that Type 1 SCDs just overwrite records, so I'm not sure how that relates to the cloning issue. Could it be B?

Haydee 12 months ago

This seems similar to a practice question we did about metadata and how it interacts with data files after vacuuming. I think C might be the right answer.

Shannan 1 year ago

I remember discussing how vacuuming affects Delta Lake tables, but I'm not entirely sure if it invalidates shallow clones directly.

Rana 1 year ago

I think the answer is C - the metadata created by the clone operation is referencing data files that were purged as invalid by the vacuum command. The shallow clone doesn't fully capture the table state, so when vacuum runs, it breaks the cloned table.

Dottie 1 year ago

Okay, let's break this down step-by-step. The key seems to be understanding how shallow cloning works and how that can be impacted by vacuum. I'll focus on that angle.

Gregoria 1 year ago

Hmm, I'm a bit confused on the details of how Delta Lake handles cloning and vacuum operations. I'll need to review the documentation to make sure I understand the key concepts.

Marylou 1 year ago

This seems like a tricky one. I'll need to think through the implications of shallow cloning and how that interacts with the vacuum process.

Eden 2 years ago

Or maybe we should consider using deep clone instead of shallow clone to prevent this issue in the future.

Breana 2 years ago

That's a good point, Isidra. Maybe the solution is to run refresh on the cloned tables after vacuum is executed on the source tables.

Isidra 2 years ago

But shouldn't running refresh on the cloned table pull in recent changes and fix the issue?

Eden 2 years ago

I agree with Breana. The vacuum command probably caused the issue with the cloned tables.

Breana 2 years ago

I think the cloned tables are no longer working because the metadata created by the clone operation is referencing data files that were purged as invalid by the vacuum command.

Paulina 2 years ago

B is a good try, but C is the correct answer. Type 1 SCD changes shouldn't affect the cloned tables, the real issue is that the vacuum command wiped out the data files that the cloned metadata was pointing to. Cloning is like a game of hide and seek, you gotta make sure your data doesn't get lost in the vacuum.

Devon 2 years ago

↳ replying to Paulina

C) The metadata created by the clone operation is referencing data files that were purged as invalid by the vacuum command

Eden 2 years ago

↳ replying to Paulina

A) The data files compacted by vacuum are not tracked by the cloned metadata; running refresh on the cloned table will pull in recent changes.

Pamella 2 years ago

Option D is the funniest one, but it's not the right answer. Shallow clones are like a house of cards - one gust of vacuum and they come tumbling down. Deep cloning is the way to go, it's like building a fortress to protect your data.

Hyun 2 years ago

I agree with C. Vacuum is like a cosmic vacuum cleaner, sucking up all the old data files and leaving the clones high and dry. Shallow cloning is like running with scissors, it's all fun and games until someone loses an eye (or a functioning table).

Temeka 2 years ago

↳ replying to Hyun

C) The metadata created by the clone operation is referencing data files that were purged as invalid by the vacuum command

Patti 2 years ago

↳ replying to Hyun

A) The data files compacted by vacuum are not tracked by the cloned metadata; running refresh on the cloned table will pull in recent changes.

Carol 2 years ago

The correct answer is C. The vacuum command purged the data files that the cloned metadata was referencing, causing the cloned tables to stop working. Shallow clones don't track changes to the source tables like deep clones do.

Chantell 2 years ago

↳ replying to Carol

A) Oh, that makes sense. The vacuum command caused the issue by purging the data files the cloned tables were referencing.

Evangelina 2 years ago

↳ replying to Carol

C) The metadata created by the clone operation is referencing data files that were purged as invalid by the vacuum command

Kattie 2 years ago

↳ replying to Carol

A) The data files compacted by vacuum are not tracked by the cloned metadata; running refresh on the cloned table will pull in recent changes.

Databricks Certified Data Engineer Professional Exam - Topic 2 Question 29 Discussion