Discuss Google Professional Data Engineer Exam Topic 4 Question 123

You are designing a data processing pipeline. The pipeline must be able to scale automatically as load increases. Messages must be processed at least once, and must be ordered within windows of 1 hour. How should you design the solution?

AUse Apache Kafka for message ingestion and use Cloud Dataproc for streaming analysis.

BUse Apache Kafka for message ingestion and use Cloud Dataflow for streaming analysis.

CUse Cloud Pub/Sub for message ingestion and Cloud Dataproc for streaming analysis.

DUse Cloud Pub/Sub for message ingestion and Cloud Dataflow for streaming analysis.

Suggested Answer: D

Get Premium Professional Data Engineer Questions as Interactive Web-Based Practice Test or PDF Real Exam Format Detailed Explanations Instant Access Get Your Practice Test

17 comments · newest first

Sheron 1 month ago

Exactly! Plus, Dataflow's integration is seamless.

Pete 1 month ago

B ensures ordered processing within the hour.

Maile 1 month ago

I feel like C might not handle the load as well as B.

Geraldo 2 months ago

A is interesting, but I worry about Dataproc's scaling.

Jose 2 months ago

D is good too, but I prefer B for streaming.

Melissa 2 months ago

I’m leaning towards D. Pub/Sub is great for ingestion.

Miss 2 months ago

I agree, B seems solid. Dataflow scales automatically.

Tyra 2 months ago

I think option B is the best choice. Kafka handles the load well.

Tiffiny 2 months ago

B is the best choice for scalability and processing guarantees!

Trinidad 3 months ago

Wait, can Pub/Sub really guarantee message ordering? Sounds sketchy.

Bethanie 3 months ago

A is interesting, but I prefer Dataflow for its auto-scaling features.

Krissy 3 months ago

I think D could work too, but not sure about Pub/Sub for ordering.

Martina 3 months ago

Definitely B! Kafka and Dataflow are a solid combo for streaming.

Arthur 3 months ago

I lean towards option D because Cloud Pub/Sub is known for its scalability, but I'm a bit uncertain about how it handles message ordering compared to Kafka.

Youlanda 4 months ago

I feel like we practiced a similar question where we had to choose between Pub/Sub and Kafka, but I can't recall the exact advantages of each.

Mauricio 4 months ago

I think Cloud Dataflow is designed for stream processing and can scale automatically, which might make option B or D the best choices.

Karon 4 months ago

I remember we discussed Kafka and its ability to handle high throughput, but I'm not sure if it guarantees message ordering within the specified windows.

Google Professional Data Engineer Exam - Topic 4 Question 123 Discussion