MENU
You are designing a pipeline to process data files that arrive in Cloud Storage by 3:00 am each day. Data processing is performed in stages, where the output of one stage becomes the input of the next. Each stage takes a long time to run. Occasionally a stage fails, and you have to address
the problem. You need to ensure that the final output is generated as quickly as possible. What should you do?
Suggested Answer: D
Get Premium Associate Data Practitioner Questions
as Interactive Web-Based Practice Test or PDF
Real Exam Format
Detailed Explanations
Instant Access
Get Your Practice Test
Using Cloud Composer to design the processing pipeline as a Directed Acyclic Graph (DAG) is the most suitable approach because:
Fault tolerance: Cloud Composer (based on Apache Airflow) allows for handling failures at specific stages. You can clear the state of a failed task and rerun it without reprocessing the entire pipeline.
Stage-based processing: DAGs are ideal for workflows with interdependent stages where the output of one stage serves as input to the next.
Efficiency: This approach minimizes downtime and ensures that only failed stages are rerun, leading to faster final output generation.
34 comments · newest first
MA
Mammie
10 months ago
I agree, B is definitely the way to go for efficiency!
TA
Tamesha
10 months ago
D seems like the best choice for managing dependencies.
JE
Jerrod
10 months ago
Option B sounds solid for handling errors.
MA
Magdalene
11 months ago
I think A is a bit slow with the user input part.
ET
Ettie
11 months ago
Wait, can you really jump stages in C? That seems risky.
BE
Beula
11 months ago
I recall that using Cloud Workflows allows for more flexibility in rerunning stages, but I’m not entirely clear on how to implement the input parameter logic.
MI
Millie
11 months ago
I practiced a similar question about error handling in Spark, but I feel like waiting for user input might slow things down too much.
ER
Erick
11 months ago
I think using Cloud Composer could be a solid option since it manages workflows well, but I’m a bit confused about how to clear the state of a failed task.
MI
Mignon
12 months ago
I remember we discussed how Dataflow is great for handling failures and can restart from specific points, but I'm not sure if it’s the best choice here.
LA
Layla
12 months ago
Option C with Cloud Workflow could be a good solution. Being able to jump to a specific stage based on an input parameter could make it easier to address errors.
DE
Deonna
1 year ago
I'm leaning towards Option D with Cloud Composer. The ability to clear the state of a failed task and rerun the workflow seems really useful.
KE
Keneth
1 year ago
I think the key here is to design a pipeline that can handle failures and restart efficiently. Option B with Dataflow seems like a good choice.
JO
Jose
1 year ago
Hmm, I'm a bit confused by the question. I'm not sure which approach would be the best for this scenario.
AD
Adelaide
1 year ago
This looks like a tricky question. I'll need to think carefully about the different options and their pros and cons.
TA
Tasia
1 year ago
I'd go with Option B. Dataflow's 'retry' feature is like the undo button for your data pipeline. Brilliant!
CR
Cristal
9 months ago
Option A seems too manual. I prefer automation!
MA
Mariann
9 months ago
I wonder if Option D would be better for complex workflows.
TR
Trina
10 months ago
Agreed! The retry feature really saves time.
DI
Dick
10 months ago
I like Option B too! Dataflow makes it so easy to manage failures.
LA
Layla
1 year ago
Option A? Really? Waiting for user input? That's so 1990s. Let's keep this cloud-native, folks!
ME
Melvin
1 year ago
D) Design the processing as a directed acyclic graph (DAG) in Cloud Composer. Clear the state of the failed task after correcting any stage output data errors.
RI
Rima
1 year ago
B) Design the pipeline as a set of PTransforms in Dataflow. Restart the pipeline after correcting any stage output data errors.
BI
Billy
1 year ago
I think option B sounds like a good approach.
MI
Mila
1 year ago
Option D seems like the most elegant solution. A DAG in Cloud Composer would give me better visibility and control over the entire process.
MI
Mica
1 year ago
That's a good point too. Using Dataflow for the pipeline design can also be a reliable option for processing the data files.
BO
Bo
1 year ago
I think I would go with option B. Designing the pipeline as a set of PTransforms in Dataflow seems like a straightforward approach.
GA
Garry
1 year ago
I agree, having a directed acyclic graph in Cloud Composer would definitely help in managing the data processing stages efficiently.
JO
Josphine
1 year ago
Option D seems like the most elegant solution. A DAG in Cloud Composer would give me better visibility and control over the entire process.
NA
Nakita
1 year ago
I'm leaning towards Option C. The flexibility of being able to jump to a specific stage in the workflow could really come in handy when troubleshooting issues.
RO
Rolland
1 year ago
I agree, Option C seems like the most efficient way to handle errors in the pipeline.
LE
Levi
1 year ago
Option C sounds like a good choice. Being able to jump to a specific stage in the workflow can save a lot of time.
KR
Kristofer
1 year ago
Option B sounds like the way to go. Dataflow's ability to restart the pipeline after fixing errors seems like the most efficient approach.
AV
Avery
1 year ago
I agree, using PTransforms in Dataflow allows for easy correction of errors and quick restart of the pipeline.
LI
Lisandra
1 year ago
Option B sounds like the way to go. Dataflow's ability to restart the pipeline after fixing errors seems like the most efficient approach.