Google Exam Professional-Data-Engineer Topic 3 Question 80 Discussion

Actual exam question for Google's Google Cloud Certified Professional Data Engineer exam

Question #: 80
Topic #: 3

[All Google Cloud Certified Professional Data Engineer Questions]

You are loading CSV files from Cloud Storage to BigQuery. The files have known data quality issues, including mismatched data types, such as STRINGS and INT64s in the same column, and inconsistent formatting of values such as phone numbers or addresses. You need to create the data pipeline to maintain data quality and perform the required cleansing and transformation. What should you do?

AUse Data Fusion to transform the data before loading it into BigQuery.

BLoad the CSV files into a staging table with the desired schema, perform the transformations with SQL. and then write the results to the final destination table.

CCreate a table with the desired schema, toad the CSV files into the table, and perform the transformations in place using SQL.

DUse Data Fusion to convert the CSV files lo a self-describing data formal, such as AVRO. before loading the data to BigOuery.

Show Suggested Answer

Suggested Answer: A

Data Fusion's advantages:

Visual interface: Offers a user-friendly interface for designing data pipelines without extensive coding, making it accessible to a wider range of users.

Built-in transformations: Includes a wide range of pre-built transformations to handle common data quality issues, such as:

Data type conversions

Data cleansing (e.g., removing invalid characters, correcting formatting)

Data validation (e.g., checking for missing values, enforcing constraints)

Data enrichment (e.g., adding derived fields, joining with other datasets)

Custom transformations: Allows for custom transformations using SQL or Java code for more complex cleaning tasks.

Scalability: Can handle large datasets efficiently, making it suitable for processing CSV files with potential data quality issues.

Integration with BigQuery: Integrates seamlessly with BigQuery, allowing for direct loading of transformed data.

by Mollie at Apr 08, 2024, 02:05 PM

Limited Time Offer

25%

Off

Get Premium Professional-Data-Engineer Questions as Interactive Web-Based Practice Test or PDF

Contribute your Thoughts:

Submit Cancel

Tawny

17 hours ago

Hey, guys, I've got a crazy idea. What if we just load the files as-is and let BigQuery handle the data type and formatting issues? That way, we can skip the whole transformation process and save a ton of time. *winks*

upvoted 0 times

...

Elza

18 hours ago

Haha, 'load the CSV files into a table and perform the transformations in place'? That sounds like a recipe for disaster! I can just imagine the table getting super messy and hard to manage. Hard pass on option C.

upvoted 0 times

...

Narcisa

2 days ago

Option A with Data Fusion sounds interesting, but I'm not sure how well it would handle the data quality issues mentioned in the question. I'd be a bit worried about potential performance or scalability problems.

upvoted 0 times

...

Juan

3 days ago

Hmm, this is a tricky one. I think I'm leaning towards option B. Loading the data into a staging table and then using SQL to perform the transformations seems like a pretty robust and flexible approach.

upvoted 0 times

...

Winfred

3 days ago

I don't know, Gearldine. Relying on a third-party tool like Data Fusion seems a bit risky to me. What if it doesn't play nice with our existing infrastructure? I think I'm leaning more towards option B as well.

upvoted 0 times

...

Elly

4 days ago

I'm not a big fan of this question. It seems to be testing very specific knowledge about data pipelines and data transformation tools, which isn't really my strong suit. I'll have to think carefully about this one.

upvoted 0 times

...

Gearldine

4 days ago

Hmm, I'm not so sure. Option D with Data Fusion might be worth considering. It could save us a lot of time and effort in the long run, especially if we have to deal with this kind of data quality issue regularly.

upvoted 0 times

...

Stephaine

5 days ago

I'm with Emogene on this one. Option B is the way to go. Who wants to deal with manually converting the files to a self-describing format? That sounds like a headache waiting to happen.

upvoted 0 times

...

Emogene

7 days ago

Option B sounds like the way to go. Staging the data first and then transforming it with SQL gives you more control and flexibility. Plus, you can easily track the changes and audit the process.

upvoted 0 times

...

Cherry

8 days ago

Ugh, this question is a real doozy! I've dealt with data quality issues before, and it's definitely not a walk in the park. I'm leaning towards option B - it seems like the most comprehensive approach to handling the data cleansing and transformation.

upvoted 0 times

...