Discuss Google Professional Data Engineer Exam Topic 1 Question 101

Your car factory is pushing machine measurements as messages into a Pub/Sub topic in your Google Cloud project. A Dataflow streaming job. that you wrote with the Apache Beam SDK, reads these messages, sends acknowledgment lo Pub/Sub. applies some custom business logic in a Doffs instance, and writes the result to BigQuery. You want to ensure that if your business logic fails on a message, the message will be sent to a Pub/Sub topic that you want to monitor for alerting purposes. What should you do?

AUse an exception handling block in your Data Flow's Doffs code to push the messages that failed to be transformed through a side output
and to a new Pub/Sub topic. Use Cloud Monitoring to monitor the topic/num_jnacked_messages_by_region metric on this new topic.

BEnable retaining of acknowledged messages in your Pub/Sub pull subscription. Use Cloud Monitoring to monitor the
subscription/num_retained_acked_messages metric on this subscription.

CEnable dead lettering in your Pub/Sub pull subscription, and specify a new Pub/Sub topic as the dead letter topic. Use Cloud Monitoring to
monitor the subscription/dead_letter_message_count metric on your pull subscription.

DCreate a snapshot of your Pub/Sub pull subscription. Use Cloud Monitoring to monitor the snapshot/numessages metric on this
snapshot.

Suggested Answer: C

To ensure that messages failing to process in your Dataflow job are sent to a Pub/Sub topic for monitoring and alerting, the best approach is to use Pub/Sub's dead-letter topic feature. Here's why option C is the best choice:

Dead-Letter Topic:

Pub/Sub's dead-letter topic feature allows messages that fail to be processed successfully to be redirected to a specified topic. This ensures that these messages are not lost and can be reviewed for debugging and alerting purposes.

Monitoring and Alerting:

By specifying a new Pub/Sub topic as the dead-letter topic, you can use Cloud Monitoring to track metrics such as subscription/dead_letter_message_count, providing visibility into the number of failed messages.

This allows you to set up alerts based on these metrics to notify the appropriate teams when failures occur.

Steps to Implement:

Enable Dead-Letter Topic:

Configure your Pub/Sub pull subscription to enable dead lettering and specify the new Pub/Sub topic for dead-letter messages.

Set Up Monitoring:

Use Cloud Monitoring to monitor the subscription/dead_letter_message_count metric on your pull subscription.

Configure alerts based on this metric to notify the team of any processing failures.

Pub/Sub Dead Letter Policy

Cloud Monitoring with Pub/Sub

Get Premium Professional Data Engineer Questions as Interactive Web-Based Practice Test or PDF Real Exam Format Detailed Explanations Instant Access Get Your Practice Test

34 comments · newest first

Wilbert 11 months ago

I’m leaning towards C too, seems like the safest bet!

Dahlia 11 months ago

Acknowledged messages retention seems unnecessary, B is a no-go.

Tommy 11 months ago

Wait, can you really monitor dead letter counts with Cloud Monitoring?

Odelia 11 months ago

I think C is the better choice for dead lettering.

Ilona 12 months ago

Option A sounds solid for handling failures.

Milly 12 months ago

Option B seems off to me; retaining acknowledged messages doesn't really address the failure scenario we need to handle.

Stefan 1 year ago

I feel like we practiced something similar with monitoring metrics in Cloud Monitoring, but I can't recall the specifics for each option.

Junita 1 year ago

I'm not entirely sure about the dead lettering concept, but I think option C might be the right approach for tracking failures.

Mila 1 year ago

I remember we discussed using side outputs in Dataflow for handling failed messages, so option A sounds familiar.

Kristofer 1 year ago

This seems like a straightforward problem, but I want to make sure I understand the details correctly. I think I'll go with option A and use an exception handling block to push the failed messages to a new Pub/Sub topic. That way, I can monitor the topic and get alerts if there are any issues.

Tamra 1 year ago

Okay, I've got it! The key is to use a side output in the Dataflow job to send the failed messages to a new Pub/Sub topic, and then monitor that topic using the num_jnacked_messages_by_region metric. That way, we can easily identify and address any issues with the business logic.

Daniel 1 year ago

Hmm, I'm a bit confused about the different options here. I'm not sure if enabling dead lettering or using a snapshot would be the best approach for this use case. I'll need to think through the pros and cons of each option.

Rickie 1 year ago

This question seems straightforward. I think the best approach is to use an exception handling block in the Dataflow job to push failed messages to a new Pub/Sub topic, and then monitor that topic using Cloud Monitoring.

Malcom 2 years ago

Haha, I bet the folks who wrote this question thought they were being clever. But I'm going with C - dead lettering is the way to go, in my opinion.

Christene 2 years ago

↳ replying to Malcom

Dead lettering seems like the most reliable option to ensure that no messages are lost in case of failures.

Leatha 2 years ago

↳ replying to Malcom

I would go with C as well. It's always better to have a designated place for failed messages.

Doyle 2 years ago

↳ replying to Malcom

Yeah, I agree. It's important to have a mechanism in place to deal with messages that couldn't be processed.

Cordelia 2 years ago

↳ replying to Malcom

I think C is the best option too. Dead lettering is a good way to handle failed messages.

Ligia 2 years ago

I'm torn between A and C. Both seem viable, but I think C might be a bit more straightforward and easier to implement.

Reena 2 years ago

↳ replying to Ligia

True, A could be a good solution as well. It's a tough decision.

Alease 2 years ago

↳ replying to Ligia

But A also seems like a good choice, with the side output feature.

Temeka 2 years ago

↳ replying to Ligia

I agree, C does seem like a straightforward option.

Janessa 2 years ago

↳ replying to Ligia

I think C might be the way to go. It sounds simpler.

Danica 2 years ago

C looks good to me. Dead lettering is a common approach for handling failed messages, and monitoring the dead letter topic is a smart move.

Cecil 2 years ago

↳ replying to Danica

I agree with A. It's important to handle failed messages at the code level to ensure proper processing and monitoring.

Deeanna 2 years ago

↳ replying to Danica

I think A might be a better option. Using exception handling in the Data Flow's DoFn code seems like a more direct way to handle failed messages.

Rodrigo 2 years ago

↳ replying to Danica

C looks good to me. Dead lettering is a common approach for handling failed messages, and monitoring the dead letter topic is a smart move.

Deandrea 2 years ago

I'm not sure, but maybe enabling dead lettering in the Pub/Sub pull subscription could also be a good option for monitoring failed messages.

Ria 2 years ago

I agree with Tequila. It's important to monitor the messages that fail to be transformed and handle them accordingly.

Letha 2 years ago

Option A seems like the way to go. I like the idea of using a side output to handle failed messages and monitor them separately.

Tu 2 years ago

↳ replying to Letha

User 3: I agree. Monitoring and handling failed messages in a separate topic can help with troubleshooting and alerting.

Ezekiel 2 years ago

↳ replying to Letha

User 2: Yes, that sounds like a good approach. It's important to have a way to track and handle failed messages effectively.

Shay 2 years ago

↳ replying to Letha

User 1: Option A seems like the way to go. I like the idea of using a side output to handle failed messages and monitor them separately.

Tequila 2 years ago

I think we should use an exception handling block in the Data Flow's DoFn code to push failed messages to a new Pub/Sub topic for monitoring.

Google Professional Data Engineer Exam - Topic 1 Question 101 Discussion