An analyst uses an AI platform to help correlate events. The AI output contains events that did not happen. This results in inaccurate correlations.
Which of the following best describes what has occurred?
The scenario describes an AI hallucination, commonly termed confabulation in formal AI risk-management literature. The defining characteristic is that the model produces information that appears plausible but is factually incorrect or unsupported. Here, the AI system introduces events that never occurred, contaminating the event-correlation process and potentially causing analysts to reach incorrect conclusions.
NIST's Generative AI Profile identifies confabulation as the production of confidently stated but erroneous or false content and treats it as an AI risk that requires verification and monitoring. NIST cybersecurity guidance also recognizes hallucination and confabulation as risks to information accuracy when AI is incorporated into cybersecurity workflows.
Data exposure would involve unauthorized disclosure of confidential or sensitive information. A malicious prompt involves intentionally crafted input designed to influence model behavior or bypass restrictions. Model poisoning occurs when an adversary manipulates training or model-related data to corrupt the system's behavior. None of these conditions is required in the scenario; the critical evidence is fabrication of nonexistent events.
Security analysts therefore must treat AI-generated correlation as analytical assistance rather than unquestioned evidence and validate important conclusions against authoritative logs and telemetry.
Study Guide Reference: Security Operations Artificial Intelligence AI Risks Hallucinations Data Exposure Malicious Prompts Model Poisoning Human Validation.
Currently there are no comments in this discussion, be the first to comment!