Discuss Amazon MLA-C01 Exam Topic 4 Question 19

A company wants to improve its customer retention ML model. The current model has 85% accuracy and a new model shows 87% accuracy in testing. The company wants to validate the new model's performance in production.

Which solution will meet these requirements?

ADeploy the new model for 4 weeks across all production traffic. Monitor performance metrics and validate improvements.

BRun A/B testing on both models for 4 weeks. Route 20% of traffic to the new model. Monitor customer retention rates across both variants.

CRun both models in parallel for 4 weeks. Analyze offline predictions weekly by using historical customer data analysis.

DImplement alternating deployments for 4 weeks between the current model and the new model. Track performance metrics for comparison.

Suggested Answer: B

AWS ML best practices recommend A/B testing to validate model improvements in production while minimizing risk. By routing a controlled portion of live traffic (for example, 20%) to the new model and keeping the majority of traffic on the existing model, the company can directly compare real-world performance using the same data distribution.

This approach allows statistically meaningful comparison of business metrics such as customer retention, rather than relying solely on offline accuracy. It also limits potential negative impact if the new model underperforms in production.

Deploying the new model to 100% of traffic (Option A) introduces unnecessary risk. Offline analysis (Option C) does not reflect live user behavior. Alternating deployments (Option D) introduces confounding factors such as time-based effects.

Therefore, A/B testing is the correct solution.

Get Premium MLA-C01 Questions as Interactive Web-Based Practice Test or PDF Real Exam Format Detailed Explanations Instant Access Get Your Practice Test

28 comments · newest first

Manish Malhotra 11 days ago

B makes the most sense here because option B uses A/B testing with 20% traffic to measure retention changes, highlighting tricky part of statistical significance.

Ruqayyah Malik 3 days ago

↳ replying to Manish Malhotra

I had the same understanding because it measures real user retention differences, not just offline accuracy.

Corazon

B balances risk and data collection well. Customer feedback is key!

Evette 5 days ago

A is too risky. We need to monitor before full deployment.

Loren 10 days ago

D could work too. Alternating might show performance shifts.

Mable 15 days ago

C seems safer. Analyzing offline predictions can reduce risks.

Sunshine 21 days ago

I agree, B allows for direct comparison. More reliable results!

Denny 26 days ago

I think B is the best option. A/B testing gives clear insights.

Marti 1 month ago

B balances risk and validation. Customer feedback is crucial!

Juan 1 month ago

I feel A is too risky. Full deployment without testing isn't wise.

Maira 1 month ago

D could work too. Alternating might highlight differences effectively.

Shayne 2 months ago

C seems safer. Analyzing offline predictions can reduce risk.

Gilma 2 months ago

I agree, B allows for direct comparison. More reliable results!

Jesusita 2 months ago

I think B is the best option. A/B testing gives clear insights.

Lelia 2 months ago

Alternating deployments could lead to inconsistent results.

Lamar 2 months ago

Not sure if 4 weeks is enough to see real differences.

Chi 2 months ago

Surprised the new model only has a 2% improvement!

Carmen 3 months ago

I think running both models in parallel is a solid approach too.

Raylene 3 months ago

A/B testing is the best way to validate performance!

Twanna 3 months ago

Deploying the new model directly seems risky.

Adelina 3 months ago

Not sure if 4 weeks is enough to validate performance.

Annabelle 3 months ago

Surprised the new model only has a 2% improvement!

Celestina 3 months ago

I think running both models in parallel makes more sense.

Joni 4 months ago

A/B testing is the way to go!

Natalie 4 months ago

Alternating deployments seem risky to me; I think option D could lead to confusion in results. I’d prefer a more controlled approach like A/B testing.

Willow 4 months ago

I feel like option C could give us a good understanding of how both models perform over time, but I wonder if analyzing offline predictions is enough for real-time validation.

Lewis 4 months ago

I remember discussing the importance of monitoring performance metrics, but I’m a bit uncertain about whether deploying the new model for all traffic is a good idea without testing first.

Farrah 6 months ago

I think option B sounds familiar; A/B testing is a common method to compare models, right? But I'm not sure if 20% traffic is enough to see a significant difference.

Amazon MLA-C01 Exam - Topic 4 Question 19 Discussion