Amazon MLA-C01 Exam - Topic 4 Question 19 Discussion

Actual exam question for Amazon's MLA-C01 exam

Question #: 19
Topic #: 4

A company wants to improve its customer retention ML model. The current model has 85% accuracy and a new model shows 87% accuracy in testing. The company wants to validate the new model's performance in production.

Which solution will meet these requirements?

ADeploy the new model for 4 weeks across all production traffic. Monitor performance metrics and validate improvements.

BRun A/B testing on both models for 4 weeks. Route 20% of traffic to the new model. Monitor customer retention rates across both variants.

CRun both models in parallel for 4 weeks. Analyze offline predictions weekly by using historical customer data analysis.

DImplement alternating deployments for 4 weeks between the current model and the new model. Track performance metrics for comparison.

Show Suggested Answer

Suggested Answer: B

AWS ML best practices recommend A/B testing to validate model improvements in production while minimizing risk. By routing a controlled portion of live traffic (for example, 20%) to the new model and keeping the majority of traffic on the existing model, the company can directly compare real-world performance using the same data distribution.

This approach allows statistically meaningful comparison of business metrics such as customer retention, rather than relying solely on offline accuracy. It also limits potential negative impact if the new model underperforms in production.

Deploying the new model to 100% of traffic (Option A) introduces unnecessary risk. Offline analysis (Option C) does not reflect live user behavior. Alternating deployments (Option D) introduces confounding factors such as time-based effects.

Therefore, A/B testing is the correct solution.

by Vallie at May 11, 2026, 07:17 AM

Limited Time Offer

25%

Off

Get Premium MLA-C01 Questions as Interactive Web-Based Practice Test or PDF

Contribute your Thoughts:

Submit Cancel

Currently there are no comments in this discussion, be the first to comment!