MENU
A team of data engineer are adding tables to a DLT pipeline that contain repetitive expectations for many of the same data quality checks.
One member of the team suggests reusing these data quality rules across all tables defined for this pipeline.
What approach would allow them to do this?
Suggested Answer: A
Get Premium Databricks Certified Data Engineer Professional Questions
as Interactive Web-Based Practice Test or PDF
Real Exam Format
Detailed Explanations
Instant Access
Get Your Practice Test
Maintaining data quality rules in a centralized Delta table allows for the reuse of these rules across multiple DLT (Delta Live Tables) pipelines. By storing these rules outside the pipeline's target schema and referencing the schema name as a pipeline parameter, the team can apply the same set of data quality checks to different tables within the pipeline. This approach ensures consistency in data quality validations and reduces redundancy in code by not having to replicate the same rules in each DLT notebook or file.
Databricks Documentation on Delta Live Tables: Delta Live Tables Guide
36 comments · newest first
DO
Dolores
10 months ago
I agree with A, it keeps things organized and efficient.
KA
Kanisha
11 months ago
Wait, can we really maintain rules in a separate notebook? That’s new to me!
DA
Daisy
11 months ago
C seems a bit complicated for just reusing rules.
SE
Selene
11 months ago
I think B could work too, but not sure if it's the best choice.
SH
Shala
11 months ago
Option A sounds like a solid plan for reusability!
LA
Lavonna
12 months ago
Option D seems like a good idea since maintaining rules in a separate notebook could keep things organized, but I wonder if it complicates the workflow too much.
GR
Graciela
12 months ago
I feel like option C could work, but I’m struggling to recall if external jobs can really access the pipeline configuration files as needed.
ED
Edmond
12 months ago
I remember practicing a question about using global variables, so option B might be the way to go, but I’m not confident about the scope of those variables.
BE
Becky
1 year ago
I think option A sounds familiar, but I'm not entirely sure how to implement it with the pipeline parameters.
PA
Pamella
1 year ago
Option C seems interesting, but I'm not sure how practical it would be to add constraints through an external job. That could add unnecessary complexity to the pipeline. I'm leaning more towards option A or D.
LE
Leatha
1 year ago
Hmm, I'm a bit confused by this question. I'm not sure if using global Python variables (option B) is the best approach, as that might make the code harder to maintain. I'll need to think this through a bit more.
MA
Margarett
1 year ago
This seems like a straightforward question about reusing data quality rules across a DLT pipeline. I think option A is the best approach, as it allows us to maintain the rules in a separate Delta table that can be referenced by the pipeline.
LO
Loreen
1 year ago
I feel pretty confident about this one. Maintaining the data quality rules in a separate Databricks notebook (option D) seems like a clean and modular approach. That way, we can easily update the rules without having to modify the pipeline code.
EL
Elenore
1 year ago
I'm leaning towards the Database.queryLocator method. It seems like the best way to preserve the entire result set while improving performance.
LU
Luke
2 years ago
I'm just picturing the team arguing over which option is best, like a bunch of data ninjas fighting over the perfect data quality kata.
FE
Fernanda
2 years ago
D) Maintain data quality rules in a separate Databricks notebook that each DLT notebook of file.
AN
Anglea
2 years ago
A) Maintain data quality rules in a Delta table outside of this pipeline's target schema, providing the schema name as a pipeline parameter.
AL
Alba
2 years ago
Option D is the one for me! Keeping the data quality rules in a separate notebook is like a data engineer's version of 'Keep Calm and Carry On.'
SH
Sharita
2 years ago
I agree, having a separate notebook for data quality rules makes it easier to manage.
NI
Nieves
2 years ago
Option D is a good choice. It helps keep things organized.
ZA
Zachary
2 years ago
Using global Python variables (option B) feels a bit hacky. I'd prefer a more structured approach like option A or D.
LA
Lashawnda
2 years ago
D) Maintain data quality rules in a separate Databricks notebook that each DLT notebook of file.
LE
Leanora
2 years ago
A) Maintain data quality rules in a Delta table outside of this pipeline's target schema, providing the schema name as a pipeline parameter.
JO
Jose
2 years ago
I agree with Val, option A seems like the most practical solution.
DO
Doug
2 years ago
I'm feeling option C. Adding constraints through an external job with access to the pipeline config seems like a robust solution.
KA
Kandis
2 years ago
Let's go with option C then, it seems like the most practical approach.
BE
Becky
2 years ago
It would definitely streamline the process and make it easier to manage.
DO
Dominque
2 years ago
I agree, having an external job handle the constraints seems efficient.
DE
Delbert
2 years ago
Option C sounds like a good idea. It would centralize the data quality rules.
VA
Val
2 years ago
But with option A, we can easily maintain and update the data quality rules.
MI
Mike
2 years ago
I disagree, I believe option D would be more efficient.
VA
Val
2 years ago
I think option A is the best approach.
AN
Antonio
2 years ago
Option A is the way to go! Maintaining data quality rules in a separate Delta table is a clean and organized approach.
LE
Lennie
2 years ago
That sounds like a smart solution to ensure consistency and efficiency in the data quality checks.
DE
Detra
2 years ago
I agree, it would make it easier to manage and update the data quality rules for all tables in the pipeline.
TE
Tequila
2 years ago
Option A is the way to go! Maintaining data quality rules in a separate Delta table is a clean and organized approach.