Deal of The Day! Hurry Up, Grab the Special Discount - Save 25% - Ends In 00:00:00 Coupon code: SAVE25
Welcome to Pass4Success

- Free Preparation Discussions
Mail Us support@pass4success.com
Location US
Mob_nav_barsMENU

Amazon MLA-C01 Exam - Topic 4 Question 19 Discussion

A company wants to improve its customer retention ML model. The current model has 85% accuracy and a new model shows 87% accuracy in testing. The company wants to validate the new model's performance in production.Which solution will meet these requirements?
B) Run A/B testing on both models for 4 weeks. Route 20% of traffic to the new model. Monitor customer retention rates across both variants.
A) Deploy the new model for 4 weeks across all production traffic. Monitor performance metrics and validate improvements.
C) Run both models in parallel for 4 weeks. Analyze offline predictions weekly by using historical customer data analysis.
D) Implement alternating deployments for 4 weeks between the current model and the new model. Track performance metrics for comparison.

Amazon MLA-C01 Exam - Topic 4 Question 19 Discussion

Actual exam question for Amazon's MLA-C01 exam
Question #: 19
Topic #: 4
[All MLA-C01 Questions]

A company wants to improve its customer retention ML model. The current model has 85% accuracy and a new model shows 87% accuracy in testing. The company wants to validate the new model's performance in production.

Which solution will meet these requirements?

Show Suggested Answer Hide Answer
Suggested Answer: B

AWS ML best practices recommend A/B testing to validate model improvements in production while minimizing risk. By routing a controlled portion of live traffic (for example, 20%) to the new model and keeping the majority of traffic on the existing model, the company can directly compare real-world performance using the same data distribution.

This approach allows statistically meaningful comparison of business metrics such as customer retention, rather than relying solely on offline accuracy. It also limits potential negative impact if the new model underperforms in production.

Deploying the new model to 100% of traffic (Option A) introduces unnecessary risk. Offline analysis (Option C) does not reflect live user behavior. Alternating deployments (Option D) introduces confounding factors such as time-based effects.

Therefore, A/B testing is the correct solution.


by Vallie at May 11, 2026, 07:17 AM

Limited Time Offer

25%

Off

Get Premium MLA-C01 Questions as Interactive Web-Based Practice Test or PDF

Contribute your Thoughts:

0/2000 characters
Submit Cancel
Jesusita
1 day ago
I think B is the best option. A/B testing gives clear insights.
upvoted 0 times
...
Lelia
7 days ago
Alternating deployments could lead to inconsistent results.
upvoted 0 times
...
Lamar
12 days ago
Not sure if 4 weeks is enough to see real differences.
upvoted 0 times
...
Chi
17 days ago
Surprised the new model only has a 2% improvement!
upvoted 0 times
...
Carmen
22 days ago
I think running both models in parallel is a solid approach too.
upvoted 0 times
...
Raylene
27 days ago
A/B testing is the best way to validate performance!
upvoted 0 times
...
Twanna
1 month ago
Deploying the new model directly seems risky.
upvoted 0 times
...
Adelina
1 month ago
Not sure if 4 weeks is enough to validate performance.
upvoted 0 times
...
Annabelle
1 month ago
Surprised the new model only has a 2% improvement!
upvoted 0 times
...
Celestina
2 months ago
I think running both models in parallel makes more sense.
upvoted 0 times
...
Joni
2 months ago
A/B testing is the way to go!
upvoted 0 times
...
Natalie
2 months ago
Alternating deployments seem risky to me; I think option D could lead to confusion in results. I’d prefer a more controlled approach like A/B testing.
upvoted 0 times
...
Willow
2 months ago
I feel like option C could give us a good understanding of how both models perform over time, but I wonder if analyzing offline predictions is enough for real-time validation.
upvoted 0 times
...
Lewis
2 months ago
I remember discussing the importance of monitoring performance metrics, but I’m a bit uncertain about whether deploying the new model for all traffic is a good idea without testing first.
upvoted 0 times
...
Farrah
4 months ago
I think option B sounds familiar; A/B testing is a common method to compare models, right? But I'm not sure if 20% traffic is enough to see a significant difference.
upvoted 0 times
...

Save Cancel