iSQI CT-AI Exam - Topic 7 Question 30 Discussion

The problem described in the question is a classic case of concept drift. Concept drift occurs when the relationship between input variables and the output variable changes over time, leading to a decline in model accuracy.

In this scenario, the average passenger and baggage weights used in the model changed due to seasonal variations, but the model was not updated accordingly. This resulted in inaccurate predictions for fuel requirements in the winter season. This is an example of seasonal drift, where model behavior changes periodically due to recurring trends (e.g., higher luggage weights in winter compared to summer).

To prevent such problems:

The model should be regularly tested for concept drift against agreed ML functional performance criteria.

Exploratory Data Analysis (EDA) should be performed periodically to detect gradual changes in input distributions.

Retraining of the model with updated training data should be done to maintain accuracy.

If drift is detected, mitigation techniques such as incremental learning, retraining with new data, or adjusting model parameters should be employed.

Why Other Options Are Incorrect:

Option B (Easing the performance standard instead of addressing drift): Lowering the performance standard is not a solution; it only masks the problem without fixing it. Instead, regular testing and retraining should be used to handle drift properly.

Option C (Corruption and reloading the model): Model corruption is unrelated to this issue. Corruption refers to accidental or malicious damage to the model or data, whereas this case is due to a changing data environment.

Option D (Lack of transparency): Transparency refers to how understandable the model's decisions are, but the problem here is a change in data distributions, making drift the primary concern.

Supporting Reference from ISTQB Certified Tester AI Testing Study Guide:

ISTQB CT-AI Syllabus (Section 7.6: Testing for Concept Drift)

'The operational environment can change over time without the trained model changing correspondingly. This phenomenon is known as concept drift and typically causes the outputs of the model to become increasingly less accurate and less useful.'

'Systems that may be prone to concept drift should be regularly tested against their agreed ML functional performance criteria to ensure that any occurrences of concept drift are detected soon enough for the problem to be mitigated.'

ISTQB CT-AI Syllabus (Section 7.7: Selecting a Test Approach for an ML System)

'If concept drift is detected, it may be mitigated by retraining the system with up-to-date training data followed by confirmation testing, regression testing, and possibly A/B testing where the updated system must outperform the original system.'

Conclusion:

Since the question describes a situation where seasonal variations affected input data distributions, the correct answer is A: The model suffers from drift and therefore should be regularly tested to ensure that any occurrences of drift are detected soon enough for the problem to be mitigated.