ISACA Advanced in AI Audit AAIA Practice Test 2
Question 10
An LLM assistant is presented as a deterministic source of policy truth without any retrieval citation or uncertainty handling. From an AI audit perspective, which corrective action is the MOST direct response to the weakness described, rather than a useful control for some different AI risk?
- Constrain answers to approved retrieved sources and route ungrounded responses to human review
- Compare simpler baselines and ensembles under the same limited-label and explainability criteria
- Test speech models on actual acoustic environments, accents and supported devices
- Require rolling-origin validation with strict time cutoffs and a final unseen period
Explanation
Correct Answer
A:Correct. Constrain answers to approved retrieved sources and route ungrounded responses to human review is aimed at the actual failure: generative probabilistic output is being mistaken for verified authoritative advice. The audit should subsequently obtain grounding evaluation logs with citations, abstention cases and reviewer overrides to test execution, not merely the existence of a policy.
Incorrect Answers
B:“Compare simpler baselines and ensembles under the same limited-label and explainability criteria” is a control for algorithm choice is unsubstantiated against sample size, interpretability and operating cost, but the audit finding here is generative probabilistic output is being mistaken for verified authoritative advice. The relevant measure is “Constrain answers to approved retrieved sources and route ungrounded responses to human review”.
C:“Test speech models on actual acoustic environments, accents and supported devices” is a control for deployment conditions differ from the representative validation environment, but the audit finding here is generative probabilistic output is being mistaken for verified authoritative advice. The relevant measure is “Constrain answers to approved retrieved sources and route ungrounded responses to human review”.
D:“Require rolling-origin validation with strict time cutoffs and a final unseen period” is a control for temporal leakage from future periods contaminates evaluation, but the audit finding here is generative probabilistic output is being mistaken for verified authoritative advice. The relevant measure is “Constrain answers to approved retrieved sources and route ungrounded responses to human review”.
Question 11
In an audit of AI Models, Considerations, and Requirements, managers say this measure is live: Constrain answers to approved retrieved sources and route ungrounded responses to human review. They claim it corrected a material audit gap. What direct operating evidence would allow the auditor to substantiate that particular claim across a defensible sample?
- Reproducible benchmark notebook with baseline metrics, error analysis and deployment-cost estimates
- Field-stratified word error reports tied to approved operational service thresholds
- Dated split manifests and out-of-time backtests showing no future records in features
- Grounding evaluation logs with citations, abstention cases and reviewer overrides
Explanation
Correct Answer
D:Correct. The proposed records—grounding evaluation logs with citations, abstention cases and reviewer overrides—are directly tied to the asserted safeguard and allow its actual operation to be corroborated. The gap being tested is generative probabilistic output is being mistaken for verified authoritative advice.
Incorrect Answers
A:“Reproducible benchmark notebook with baseline metrics, error analysis and deployment-cost estimates” would test algorithm choice is unsubstantiated against sample size, interpretability and operating cost, not the present safeguard against generative probabilistic output is being mistaken for verified authoritative advice. The expected audit evidence is “Grounding evaluation logs with citations, abstention cases and reviewer overrides”.
B:“Field-stratified word error reports tied to approved operational service thresholds” would test deployment conditions differ from the representative validation environment, not the present safeguard against generative probabilistic output is being mistaken for verified authoritative advice. The expected audit evidence is “Grounding evaluation logs with citations, abstention cases and reviewer overrides”.
C:“Dated split manifests and out-of-time backtests showing no future records in features” would test temporal leakage from future periods contaminates evaluation, not the present safeguard against generative probabilistic output is being mistaken for verified authoritative advice. The expected audit evidence is “Grounding evaluation logs with citations, abstention cases and reviewer overrides”.
Question 12
In a review of AI Models, Considerations, and Requirements, investigators cannot establish the expected control proof: grounding evaluation logs with citations, abstention cases and reviewer overrides. The underlying event was that an LLM assistant is presented as a deterministic source of policy truth without any retrieval citation or uncertainty handling. Which control weakness is the MOST likely cause requiring focused audit work?
- Algorithm choice is unsubstantiated against sample size, interpretability and operating cost
- Deployment conditions differ from the representative validation environment
- Generative probabilistic output is being mistaken for verified authoritative advice
- Temporal leakage from future periods contaminates evaluation
Explanation
Correct Answer
C:Correct. The observed condition specifically indicates generative probabilistic output is being mistaken for verified authoritative advice. A meaningful audit response is to verify this diagnosis and then recommend constrain answers to approved retrieved sources and route ungrounded responses to human review.
Incorrect Answers
A:“Algorithm choice is unsubstantiated against sample size, interpretability and operating cost” concerns a separate audit weakness, not the missing assurance described by generative probabilistic output is being mistaken for verified authoritative advice. The correct cause is “Generative probabilistic output is being mistaken for verified authoritative advice”.
B:“Deployment conditions differ from the representative validation environment” concerns a separate audit weakness, not the missing assurance described by generative probabilistic output is being mistaken for verified authoritative advice. The correct cause is “Generative probabilistic output is being mistaken for verified authoritative advice”.
D:“Temporal leakage from future periods contaminates evaluation” concerns a separate audit weakness, not the missing assurance described by generative probabilistic output is being mistaken for verified authoritative advice. The correct cause is “Generative probabilistic output is being mistaken for verified authoritative advice”.
Question 13
A medical triage classifier receives unstructured images but its initial requirements list only tabular columns. From an AI audit perspective, which corrective action is the MOST direct response to the weakness described, rather than a useful control for some different AI risk?
- Reassess purpose, affected users, risk tier and mandatory safeguards before new use
- Define image quality, preprocessing, provenance and clinical operating boundaries in system requirements
- Define tail-latency, fail-safe response and reliability service objectives in selection gates
- Perform per-jurisdiction applicability assessment and validate stratified outcomes before rollout
Explanation
Correct Answer
B:Correct. Define image quality, preprocessing, provenance and clinical operating boundaries in system requirements is aimed at the actual failure: input modality, preprocessing and clinical context requirements were omitted. The audit should subsequently obtain signed multimodal interface specification and testing for unreadable or out-of-scope images to test execution, not merely the existence of a policy.
Incorrect Answers
A:“Reassess purpose, affected users, risk tier and mandatory safeguards before new use” is a control for material scope expansion bypassed intended-use and risk reapproval, but the audit finding here is input modality, preprocessing and clinical context requirements were omitted. The relevant measure is “Define image quality, preprocessing, provenance and clinical operating boundaries in system requirements”.
C:“Define tail-latency, fail-safe response and reliability service objectives in selection gates” is a control for nonfunctional latency and availability requirements were excluded from model suitability, but the audit finding here is input modality, preprocessing and clinical context requirements were omitted. The relevant measure is “Define image quality, preprocessing, provenance and clinical operating boundaries in system requirements”.
D:“Perform per-jurisdiction applicability assessment and validate stratified outcomes before rollout” is a control for model suitability was not evaluated for each intended deployment population, but the audit finding here is input modality, preprocessing and clinical context requirements were omitted. The relevant measure is “Define image quality, preprocessing, provenance and clinical operating boundaries in system requirements”.
Question 14
In an audit of AI Models, Considerations, and Requirements, managers say this measure is live: Define image quality, preprocessing, provenance and clinical operating boundaries in system requirements. They claim it corrected a material audit gap. What direct operating evidence would allow the auditor to substantiate that particular claim across a defensible sample?
- Signed multimodal interface specification and testing for unreadable or out-of-scope images
- Change ticket, updated intended-use record and signed risk-owner reauthorization
- p95/p99 traces, failover drills and completed acceptance results against approved SLAs
- Approved population inventory and disaggregated performance tests by subsidiary
Explanation
Correct Answer
A:Correct. The proposed records—signed multimodal interface specification and testing for unreadable or out-of-scope images—are directly tied to the asserted safeguard and allow its actual operation to be corroborated. The gap being tested is input modality, preprocessing and clinical context requirements were omitted.
Incorrect Answers
B:“Change ticket, updated intended-use record and signed risk-owner reauthorization” would test material scope expansion bypassed intended-use and risk reapproval, not the present safeguard against input modality, preprocessing and clinical context requirements were omitted. The expected audit evidence is “Signed multimodal interface specification and testing for unreadable or out-of-scope images”.
C:“p95/p99 traces, failover drills and completed acceptance results against approved SLAs” would test nonfunctional latency and availability requirements were excluded from model suitability, not the present safeguard against input modality, preprocessing and clinical context requirements were omitted. The expected audit evidence is “Signed multimodal interface specification and testing for unreadable or out-of-scope images”.
D:“Approved population inventory and disaggregated performance tests by subsidiary” would test model suitability was not evaluated for each intended deployment population, not the present safeguard against input modality, preprocessing and clinical context requirements were omitted. The expected audit evidence is “Signed multimodal interface specification and testing for unreadable or out-of-scope images”.
Question 15
In a review of AI Models, Considerations, and Requirements, investigators cannot establish the expected control proof: signed multimodal interface specification and testing for unreadable or out-of-scope images. The underlying event was that a medical triage classifier receives unstructured images but its initial requirements list only tabular columns. Which control weakness is the MOST likely cause requiring focused audit work?
- Material scope expansion bypassed intended-use and risk reapproval
- Nonfunctional latency and availability requirements were excluded from model suitability
- Model suitability was not evaluated for each intended deployment population
- Input modality, preprocessing and clinical context requirements were omitted
Explanation
Correct Answer
D:Correct. The observed condition specifically indicates input modality, preprocessing and clinical context requirements were omitted. A meaningful audit response is to verify this diagnosis and then recommend define image quality, preprocessing, provenance and clinical operating boundaries in system requirements.
Incorrect Answers
A:“Material scope expansion bypassed intended-use and risk reapproval” concerns a separate audit weakness, not the missing assurance described by input modality, preprocessing and clinical context requirements were omitted. The correct cause is “Input modality, preprocessing and clinical context requirements were omitted”.
B:“Nonfunctional latency and availability requirements were excluded from model suitability” concerns a separate audit weakness, not the missing assurance described by input modality, preprocessing and clinical context requirements were omitted. The correct cause is “Input modality, preprocessing and clinical context requirements were omitted”.
C:“Model suitability was not evaluated for each intended deployment population” concerns a separate audit weakness, not the missing assurance described by input modality, preprocessing and clinical context requirements were omitted. The correct cause is “Input modality, preprocessing and clinical context requirements were omitted”.
Question 16
A vendor describes a complex ensemble as inherently superior despite extremely sparse labeled failure cases. From an AI audit perspective, which corrective action is the MOST direct response to the weakness described, rather than a useful control for some different AI risk?
- Establish context-specific evaluation cohorts and acceptable alert burden before threshold approval
- Specify tool permissions, spend limits, approvals and rollback semantics for every autonomous action
- Compare simpler baselines and ensembles under the same limited-label and explainability criteria
- Constrain answers to approved retrieved sources and route ungrounded responses to human review
Explanation
Correct Answer
C:Correct. Compare simpler baselines and ensembles under the same limited-label and explainability criteria is aimed at the actual failure: algorithm choice is unsubstantiated against sample size, interpretability and operating cost. The audit should subsequently obtain reproducible benchmark notebook with baseline metrics, error analysis and deployment-cost estimates to test execution, not merely the existence of a policy.
Incorrect Answers
A:“Establish context-specific evaluation cohorts and acceptable alert burden before threshold approval” is a control for normal operational context and false-positive tolerance are absent from design assumptions, but the audit finding here is algorithm choice is unsubstantiated against sample size, interpretability and operating cost. The relevant measure is “Compare simpler baselines and ensembles under the same limited-label and explainability criteria”.
B:“Specify tool permissions, spend limits, approvals and rollback semantics for every autonomous action” is a control for delegated action authority and transaction boundaries were omitted, but the audit finding here is algorithm choice is unsubstantiated against sample size, interpretability and operating cost. The relevant measure is “Compare simpler baselines and ensembles under the same limited-label and explainability criteria”.
D:“Constrain answers to approved retrieved sources and route ungrounded responses to human review” is a control for generative probabilistic output is being mistaken for verified authoritative advice, but the audit finding here is algorithm choice is unsubstantiated against sample size, interpretability and operating cost. The relevant measure is “Compare simpler baselines and ensembles under the same limited-label and explainability criteria”.
Question 17
In an audit of AI Models, Considerations, and Requirements, managers say this measure is live: Compare simpler baselines and ensembles under the same limited-label and explainability criteria. They claim it corrected a material audit gap. What direct operating evidence would allow the auditor to substantiate that particular claim across a defensible sample?
- Season-tagged test results and operations signoff on alert volume and missed anomalies
- Reproducible benchmark notebook with baseline metrics, error analysis and deployment-cost estimates
- Approved tool-use matrix and negative tests for unauthorized purchase requests
- Grounding evaluation logs with citations, abstention cases and reviewer overrides
Explanation
Correct Answer
B:Correct. The proposed records—reproducible benchmark notebook with baseline metrics, error analysis and deployment-cost estimates—are directly tied to the asserted safeguard and allow its actual operation to be corroborated. The gap being tested is algorithm choice is unsubstantiated against sample size, interpretability and operating cost.
Incorrect Answers
A:“Season-tagged test results and operations signoff on alert volume and missed anomalies” would test normal operational context and false-positive tolerance are absent from design assumptions, not the present safeguard against algorithm choice is unsubstantiated against sample size, interpretability and operating cost. The expected audit evidence is “Reproducible benchmark notebook with baseline metrics, error analysis and deployment-cost estimates”.
C:“Approved tool-use matrix and negative tests for unauthorized purchase requests” would test delegated action authority and transaction boundaries were omitted, not the present safeguard against algorithm choice is unsubstantiated against sample size, interpretability and operating cost. The expected audit evidence is “Reproducible benchmark notebook with baseline metrics, error analysis and deployment-cost estimates”.
D:“Grounding evaluation logs with citations, abstention cases and reviewer overrides” would test generative probabilistic output is being mistaken for verified authoritative advice, not the present safeguard against algorithm choice is unsubstantiated against sample size, interpretability and operating cost. The expected audit evidence is “Reproducible benchmark notebook with baseline metrics, error analysis and deployment-cost estimates”.
Question 18
In a review of AI Models, Considerations, and Requirements, investigators cannot establish the expected control proof: reproducible benchmark notebook with baseline metrics, error analysis and deployment-cost estimates. The underlying event was that a vendor describes a complex ensemble as inherently superior despite extremely sparse labeled failure cases. Which control weakness is the MOST likely cause requiring focused audit work?
- Algorithm choice is unsubstantiated against sample size, interpretability and operating cost
- Normal operational context and false-positive tolerance are absent from design assumptions
- Delegated action authority and transaction boundaries were omitted
- Generative probabilistic output is being mistaken for verified authoritative advice
Explanation
Correct Answer
A:Correct. The observed condition specifically indicates algorithm choice is unsubstantiated against sample size, interpretability and operating cost. A meaningful audit response is to verify this diagnosis and then recommend compare simpler baselines and ensembles under the same limited-label and explainability criteria.
Incorrect Answers
B:“Normal operational context and false-positive tolerance are absent from design assumptions” concerns a separate audit weakness, not the missing assurance described by algorithm choice is unsubstantiated against sample size, interpretability and operating cost. The correct cause is “Algorithm choice is unsubstantiated against sample size, interpretability and operating cost”.
C:“Delegated action authority and transaction boundaries were omitted” concerns a separate audit weakness, not the missing assurance described by algorithm choice is unsubstantiated against sample size, interpretability and operating cost. The correct cause is “Algorithm choice is unsubstantiated against sample size, interpretability and operating cost”.
D:“Generative probabilistic output is being mistaken for verified authoritative advice” concerns a separate audit weakness, not the missing assurance described by algorithm choice is unsubstantiated against sample size, interpretability and operating cost. The correct cause is “Algorithm choice is unsubstantiated against sample size, interpretability and operating cost”.
Question 19
A customer chatbot is promoted from product FAQ to financial advice without revisiting its risk classification. From an AI audit perspective, which corrective action is the MOST direct response to the weakness described, rather than a useful control for some different AI risk?
- Test speech models on actual acoustic environments, accents and supported devices
- Approve class-specific recall, precision and cost-weighted thresholds with business risk owners
- Define image quality, preprocessing, provenance and clinical operating boundaries in system requirements
- Reassess purpose, affected users, risk tier and mandatory safeguards before new use
Explanation
Correct Answer
D:Correct. Reassess purpose, affected users, risk tier and mandatory safeguards before new use is aimed at the actual failure: material scope expansion bypassed intended-use and risk reapproval. The audit should subsequently obtain change ticket, updated intended-use record and signed risk-owner reauthorization to test execution, not merely the existence of a policy.
Incorrect Answers
A:“Test speech models on actual acoustic environments, accents and supported devices” is a control for deployment conditions differ from the representative validation environment, but the audit finding here is material scope expansion bypassed intended-use and risk reapproval. The relevant measure is “Reassess purpose, affected users, risk tier and mandatory safeguards before new use”.
B:“Approve class-specific recall, precision and cost-weighted thresholds with business risk owners” is a control for business loss asymmetry is missing from model acceptance criteria, but the audit finding here is material scope expansion bypassed intended-use and risk reapproval. The relevant measure is “Reassess purpose, affected users, risk tier and mandatory safeguards before new use”.
C:“Define image quality, preprocessing, provenance and clinical operating boundaries in system requirements” is a control for input modality, preprocessing and clinical context requirements were omitted, but the audit finding here is material scope expansion bypassed intended-use and risk reapproval. The relevant measure is “Reassess purpose, affected users, risk tier and mandatory safeguards before new use”.
Question 20
In an audit of AI Models, Considerations, and Requirements, managers say this measure is live: Reassess purpose, affected users, risk tier and mandatory safeguards before new use. They claim it corrected a material audit gap. What direct operating evidence would allow the auditor to substantiate that particular claim across a defensible sample?
- Field-stratified word error reports tied to approved operational service thresholds
- Signed acceptance thresholds, stratified confusion matrices and loss simulations from the held-out release set
- Change ticket, updated intended-use record and signed risk-owner reauthorization
- Signed multimodal interface specification and testing for unreadable or out-of-scope images
Explanation
Correct Answer
C:Correct. The proposed records—change ticket, updated intended-use record and signed risk-owner reauthorization—are directly tied to the asserted safeguard and allow its actual operation to be corroborated. The gap being tested is material scope expansion bypassed intended-use and risk reapproval.
Incorrect Answers
A:“Field-stratified word error reports tied to approved operational service thresholds” would test deployment conditions differ from the representative validation environment, not the present safeguard against material scope expansion bypassed intended-use and risk reapproval. The expected audit evidence is “Change ticket, updated intended-use record and signed risk-owner reauthorization”.
B:“Signed acceptance thresholds, stratified confusion matrices and loss simulations from the held-out release set” would test business loss asymmetry is missing from model acceptance criteria, not the present safeguard against material scope expansion bypassed intended-use and risk reapproval. The expected audit evidence is “Change ticket, updated intended-use record and signed risk-owner reauthorization”.
D:“Signed multimodal interface specification and testing for unreadable or out-of-scope images” would test input modality, preprocessing and clinical context requirements were omitted, not the present safeguard against material scope expansion bypassed intended-use and risk reapproval. The expected audit evidence is “Change ticket, updated intended-use record and signed risk-owner reauthorization”.