Databricks Certified Generative AI Engineer Associate Exam - Topic 6 Question 15 Discussion

Actual exam question for Databricks's Databricks Certified Generative AI Engineer Associate exam

Question #: 15
Topic #: 6

[All Databricks Certified Generative AI Engineer Associate Questions]

A Generative AI Engineer just deployed an LLM application at a digital marketing company that assists with answering customer service inquiries.

Which metric should they monitor for their customer service LLM application in production?

ANumber of customer inquiries processed per unit of time

BEnergy usage per query

CFinal perplexity scores for the training of the model

DHuggingFace Leaderboard values for the base LLM

Show Suggested Answer

Suggested Answer: A

When deploying an LLM application for customer service inquiries, the primary focus is on measuring the operational efficiency and quality of the responses. Here's why A is the correct metric:

Number of customer inquiries processed per unit of time: This metric tracks the throughput of the customer service system, reflecting how many customer inquiries the LLM application can handle in a given time period (e.g., per minute or hour). High throughput is crucial in customer service applications where quick response times are essential to user satisfaction and business efficiency.

Real-time performance monitoring: Monitoring the number of queries processed is an important part of ensuring that the model is performing well under load, especially during peak traffic times. It also helps ensure the system scales properly to meet demand.

Why other options are not ideal:

B . Energy usage per query: While energy efficiency is a consideration, it is not the primary concern for a customer-facing application where user experience (i.e., fast and accurate responses) is critical.

C . Final perplexity scores for the training of the model: Perplexity is a metric for model training, but it doesn't reflect the real-time operational performance of an LLM in production.

D . HuggingFace Leaderboard values for the base LLM: The HuggingFace Leaderboard is more relevant during model selection and benchmarking. However, it is not a direct measure of the model's performance in a specific customer service application in production.

Focusing on throughput (inquiries processed per unit time) ensures that the LLM application is meeting business needs for fast and efficient customer service responses.

by Brinda at Apr 30, 2025, 09:52 PM

Limited Time Offer

25%

Off

Get Premium Databricks Certified Generative AI Engineer Associate Questions as Interactive Web-Based Practice Test or PDF

Contribute your Thoughts:

Submit Cancel

Latanya

5 months ago

Final perplexity scores? Seems a bit too technical for production.

upvoted 0 times

...

Bette

5 months ago

I agree, that's the most relevant metric for customer service.

upvoted 0 times

...

Coral

5 months ago

Definitely should track the number of inquiries processed!

upvoted 0 times

...

Dominic

6 months ago

But what about energy usage? It could impact costs!

upvoted 0 times

...

Gertude

6 months ago

Wait, are we really looking at HuggingFace Leaderboard values? That’s surprising!

upvoted 0 times

...

Katina

6 months ago

The HuggingFace Leaderboard values seem more relevant for comparing models rather than monitoring a deployed application. I think we need something more practical for customer service.

upvoted 0 times

...

Danilo

6 months ago

I feel like final perplexity scores are more about training performance rather than how the model performs in production. So, I’m leaning away from that option.

upvoted 0 times

...

Marti

7 months ago

I'm not entirely sure, but I remember something about monitoring energy usage per query being important for sustainability. Could that be relevant here?

upvoted 0 times

...

Ardella

7 months ago

I think we should focus on the number of customer inquiries processed per unit of time. It seems like a direct measure of the LLM's effectiveness in a customer service role.

upvoted 0 times

...

Audria

7 months ago

For this type of customer service LLM application, I'd say the number of inquiries processed per unit of time (A) is the best metric to focus on. That will give us a clear sense of how efficiently the model is handling the workload.

upvoted 0 times

...

Celeste

7 months ago

I'm a little confused on this one. Is the final perplexity score (C) or the HuggingFace Leaderboard values (D) something we should be looking at? I'm not sure how those metrics relate to the customer service performance.

upvoted 0 times

...

Shannon

7 months ago

I think the answer is A. The key thing we want to measure is how well the LLM is handling the customer service inquiries, so the number of inquiries processed is the most relevant metric.

upvoted 0 times

...

Wynell

8 months ago

Hmm, I'm a bit unsure about this one. I'm wondering if energy usage per query (B) might be more relevant since we want to monitor the efficiency and resource usage of the LLM application. I'll have to think this through a bit more.

upvoted 0 times

...

Ernestine

8 months ago

This seems like a pretty straightforward question. I'd go with A - the number of customer inquiries processed per unit of time is a key metric to monitor for a customer service LLM application.

upvoted 0 times

...

Blondell

12 months ago

I think we should consider both A) and C) to get a comprehensive view of the performance of the LLM application.

upvoted 0 times

...

Alline

12 months ago

I believe monitoring C) Final perplexity scores for the training of the model is also important to ensure the accuracy of the responses.

upvoted 0 times

...

Matthew

1 year ago

I agree with Alfred. That metric will show us how efficient the LLM application is in handling customer inquiries.

upvoted 0 times

...

Tricia

1 year ago

The correct answer is clearly A - number of customer inquiries processed. Unless they're running this thing on a potato, the energy usage is probably not a concern. And who cares about the leaderboard when you've got customers to serve?

upvoted 0 times

Sunny

11 months ago

Customer satisfaction should be the top priority when it comes to customer service applications.

upvoted 0 times

...

Renea

11 months ago

Final perplexity scores and HuggingFace Leaderboard values are more for model evaluation rather than production monitoring.

upvoted 0 times

...

Mable

11 months ago

Energy usage per query is not as important as ensuring efficient customer service.

upvoted 0 times

...

Laurel

11 months ago

I agree, monitoring the number of customer inquiries processed is crucial for the success of the application.

upvoted 0 times

...

Aracelis

1 year ago

Haha, energy usage per query? What is this, a green AI challenge? I think the Generative AI Engineer needs to focus on the actual business metrics, not how much electricity the model is chugging.

upvoted 0 times

...

Paola

1 year ago

I'm going with option A. Gotta keep those customers happy and make sure the LLM is keeping up with the demand. Energy usage and leaderboard scores don't matter if the users aren't satisfied.

upvoted 0 times

Cassi

11 months ago

User 3: Monitoring the number of inquiries processed per unit of time is essential for efficiency.

upvoted 0 times

...

Roslyn

11 months ago

User 2: Definitely, keeping up with the demand is crucial for success.

upvoted 0 times

...

Lachelle

12 months ago

User 1: I agree, customer satisfaction is key. Option A is the way to go.

upvoted 0 times

...

Alfred

1 year ago

I think we should monitor A) Number of customer inquiries processed per unit of time.

upvoted 0 times

...

Latrice

1 year ago

Definitely the number of customer inquiries processed per unit of time. That's the key metric to track for a customer service LLM application. Anything else is just a distraction.

upvoted 0 times

Cristina

11 months ago

C: Final perplexity scores for the training of the model could also give us insights into the performance and accuracy of the LLM.

upvoted 0 times

...

Julie

11 months ago

B: Energy usage per query might be important too, we need to ensure efficiency in our operations.

upvoted 0 times

...

Lili

11 months ago

A: I agree, monitoring the number of customer inquiries processed per unit of time is crucial for the success of the LLM application.

upvoted 0 times

...

Paulina

12 months ago

C: Final perplexity scores for the training of the model could provide insights into the overall effectiveness and accuracy of the LLM application.

upvoted 0 times

...

Pearline

1 year ago

B: Energy usage per query might also be important to consider to ensure efficiency and cost-effectiveness.

upvoted 0 times

...

Brynn

1 year ago

A: I agree, tracking the number of customer inquiries processed per unit of time is crucial for monitoring the performance of the LLM application.

upvoted 0 times

...