ISACA Advanced in AI Audit AAIA Practice Test 1
Question 1
A fraud-screening pilot is evaluated only for global accuracy although missed fraud has ten times the cost of a false alarm. From an AI audit perspective, which corrective action is the MOST direct response to the weakness described, rather than a useful control for some different AI risk?
- Perform per-jurisdiction applicability assessment and validate stratified outcomes before rollout
- Approve class-specific recall, precision and cost-weighted thresholds with business risk owners
- Compare simpler baselines and ensembles under the same limited-label and explainability criteria
- Define tail-latency, fail-safe response and reliability service objectives in selection gates
Explanation
Correct Answer
B:Correct. Approve class-specific recall, precision and cost-weighted thresholds with business risk owners is aimed at the actual failure: business loss asymmetry is missing from model acceptance criteria. The audit should subsequently obtain signed acceptance thresholds, stratified confusion matrices and loss simulations from the held-out release set to test execution, not merely the existence of a policy.
Incorrect Answers
A:“Perform per-jurisdiction applicability assessment and validate stratified outcomes before rollout” is a control for model suitability was not evaluated for each intended deployment population, but the audit finding here is business loss asymmetry is missing from model acceptance criteria. The relevant measure is “Approve class-specific recall, precision and cost-weighted thresholds with business risk owners”.
C:“Compare simpler baselines and ensembles under the same limited-label and explainability criteria” is a control for algorithm choice is unsubstantiated against sample size, interpretability and operating cost, but the audit finding here is business loss asymmetry is missing from model acceptance criteria. The relevant measure is “Approve class-specific recall, precision and cost-weighted thresholds with business risk owners”.
D:“Define tail-latency, fail-safe response and reliability service objectives in selection gates” is a control for nonfunctional latency and availability requirements were excluded from model suitability, but the audit finding here is business loss asymmetry is missing from model acceptance criteria. The relevant measure is “Approve class-specific recall, precision and cost-weighted thresholds with business risk owners”.
Question 2
In an audit of AI Models, Considerations, and Requirements, managers say this measure is live: Approve class-specific recall, precision and cost-weighted thresholds with business risk owners. They claim it corrected a material audit gap. What direct operating evidence would allow the auditor to substantiate that particular claim across a defensible sample?
- Signed acceptance thresholds, stratified confusion matrices and loss simulations from the held-out release set
- Approved population inventory and disaggregated performance tests by subsidiary
- Reproducible benchmark notebook with baseline metrics, error analysis and deployment-cost estimates
- p95/p99 traces, failover drills and completed acceptance results against approved SLAs
Explanation
Correct Answer
A:Correct. The proposed records—signed acceptance thresholds, stratified confusion matrices and loss simulations from the held-out release set—are directly tied to the asserted safeguard and allow its actual operation to be corroborated. The gap being tested is business loss asymmetry is missing from model acceptance criteria.
Incorrect Answers
B:“Approved population inventory and disaggregated performance tests by subsidiary” would test model suitability was not evaluated for each intended deployment population, not the present safeguard against business loss asymmetry is missing from model acceptance criteria. The expected audit evidence is “Signed acceptance thresholds, stratified confusion matrices and loss simulations from the held-out release set”.
C:“Reproducible benchmark notebook with baseline metrics, error analysis and deployment-cost estimates” would test algorithm choice is unsubstantiated against sample size, interpretability and operating cost, not the present safeguard against business loss asymmetry is missing from model acceptance criteria. The expected audit evidence is “Signed acceptance thresholds, stratified confusion matrices and loss simulations from the held-out release set”.
D:“p95/p99 traces, failover drills and completed acceptance results against approved SLAs” would test nonfunctional latency and availability requirements were excluded from model suitability, not the present safeguard against business loss asymmetry is missing from model acceptance criteria. The expected audit evidence is “Signed acceptance thresholds, stratified confusion matrices and loss simulations from the held-out release set”.
Question 3
In a review of AI Models, Considerations, and Requirements, investigators cannot establish the expected control proof: signed acceptance thresholds, stratified confusion matrices and loss simulations from the held-out release set. The underlying event was that a fraud-screening pilot is evaluated only for global accuracy although missed fraud has ten times the cost of a false alarm. Which control weakness is the MOST likely cause requiring focused audit work?
- Model suitability was not evaluated for each intended deployment population
- Algorithm choice is unsubstantiated against sample size, interpretability and operating cost
- Nonfunctional latency and availability requirements were excluded from model suitability
- Business loss asymmetry is missing from model acceptance criteria
Explanation
Correct Answer
D:Correct. The observed condition specifically indicates business loss asymmetry is missing from model acceptance criteria. A meaningful audit response is to verify this diagnosis and then recommend approve class-specific recall, precision and cost-weighted thresholds with business risk owners.
Incorrect Answers
A:“Model suitability was not evaluated for each intended deployment population” concerns a separate audit weakness, not the missing assurance described by business loss asymmetry is missing from model acceptance criteria. The correct cause is “Business loss asymmetry is missing from model acceptance criteria”.
B:“Algorithm choice is unsubstantiated against sample size, interpretability and operating cost” concerns a separate audit weakness, not the missing assurance described by business loss asymmetry is missing from model acceptance criteria. The correct cause is “Business loss asymmetry is missing from model acceptance criteria”.
C:“Nonfunctional latency and availability requirements were excluded from model suitability” concerns a separate audit weakness, not the missing assurance described by business loss asymmetry is missing from model acceptance criteria. The correct cause is “Business loss asymmetry is missing from model acceptance criteria”.
Question 4
A forecasting model was approved on random row splits even though sales patterns shift by quarter. From an AI audit perspective, which corrective action is the MOST direct response to the weakness described, rather than a useful control for some different AI risk?
- Constrain answers to approved retrieved sources and route ungrounded responses to human review
- Reassess purpose, affected users, risk tier and mandatory safeguards before new use
- Require rolling-origin validation with strict time cutoffs and a final unseen period
- Specify tool permissions, spend limits, approvals and rollback semantics for every autonomous action
Explanation
Correct Answer
C:Correct. Require rolling-origin validation with strict time cutoffs and a final unseen period is aimed at the actual failure: temporal leakage from future periods contaminates evaluation. The audit should subsequently obtain dated split manifests and out-of-time backtests showing no future records in features to test execution, not merely the existence of a policy.
Incorrect Answers
A:“Constrain answers to approved retrieved sources and route ungrounded responses to human review” is a control for generative probabilistic output is being mistaken for verified authoritative advice, but the audit finding here is temporal leakage from future periods contaminates evaluation. The relevant measure is “Require rolling-origin validation with strict time cutoffs and a final unseen period”.
B:“Reassess purpose, affected users, risk tier and mandatory safeguards before new use” is a control for material scope expansion bypassed intended-use and risk reapproval, but the audit finding here is temporal leakage from future periods contaminates evaluation. The relevant measure is “Require rolling-origin validation with strict time cutoffs and a final unseen period”.
D:“Specify tool permissions, spend limits, approvals and rollback semantics for every autonomous action” is a control for delegated action authority and transaction boundaries were omitted, but the audit finding here is temporal leakage from future periods contaminates evaluation. The relevant measure is “Require rolling-origin validation with strict time cutoffs and a final unseen period”.
Question 5
In an audit of AI Models, Considerations, and Requirements, managers say this measure is live: Require rolling-origin validation with strict time cutoffs and a final unseen period. They claim it corrected a material audit gap. What direct operating evidence would allow the auditor to substantiate that particular claim across a defensible sample?
- Grounding evaluation logs with citations, abstention cases and reviewer overrides
- Dated split manifests and out-of-time backtests showing no future records in features
- Change ticket, updated intended-use record and signed risk-owner reauthorization
- Approved tool-use matrix and negative tests for unauthorized purchase requests
Explanation
Correct Answer
B:Correct. The proposed records—dated split manifests and out-of-time backtests showing no future records in features—are directly tied to the asserted safeguard and allow its actual operation to be corroborated. The gap being tested is temporal leakage from future periods contaminates evaluation.
Incorrect Answers
A:“Grounding evaluation logs with citations, abstention cases and reviewer overrides” would test generative probabilistic output is being mistaken for verified authoritative advice, not the present safeguard against temporal leakage from future periods contaminates evaluation. The expected audit evidence is “Dated split manifests and out-of-time backtests showing no future records in features”.
C:“Change ticket, updated intended-use record and signed risk-owner reauthorization” would test material scope expansion bypassed intended-use and risk reapproval, not the present safeguard against temporal leakage from future periods contaminates evaluation. The expected audit evidence is “Dated split manifests and out-of-time backtests showing no future records in features”.
D:“Approved tool-use matrix and negative tests for unauthorized purchase requests” would test delegated action authority and transaction boundaries were omitted, not the present safeguard against temporal leakage from future periods contaminates evaluation. The expected audit evidence is “Dated split manifests and out-of-time backtests showing no future records in features”.
Question 6
In a review of AI Models, Considerations, and Requirements, investigators cannot establish the expected control proof: dated split manifests and out-of-time backtests showing no future records in features. The underlying event was that a forecasting model was approved on random row splits even though sales patterns shift by quarter. Which control weakness is the MOST likely cause requiring focused audit work?
- Temporal leakage from future periods contaminates evaluation
- Generative probabilistic output is being mistaken for verified authoritative advice
- Material scope expansion bypassed intended-use and risk reapproval
- Delegated action authority and transaction boundaries were omitted
Explanation
Correct Answer
A:Correct. The observed condition specifically indicates temporal leakage from future periods contaminates evaluation. A meaningful audit response is to verify this diagnosis and then recommend require rolling-origin validation with strict time cutoffs and a final unseen period.
Incorrect Answers
B:“Generative probabilistic output is being mistaken for verified authoritative advice” concerns a separate audit weakness, not the missing assurance described by temporal leakage from future periods contaminates evaluation. The correct cause is “Temporal leakage from future periods contaminates evaluation”.
C:“Material scope expansion bypassed intended-use and risk reapproval” concerns a separate audit weakness, not the missing assurance described by temporal leakage from future periods contaminates evaluation. The correct cause is “Temporal leakage from future periods contaminates evaluation”.
D:“Delegated action authority and transaction boundaries were omitted” concerns a separate audit weakness, not the missing assurance described by temporal leakage from future periods contaminates evaluation. The correct cause is “Temporal leakage from future periods contaminates evaluation”.
Question 7
A claims model is deployed across subsidiaries whose eligibility rules are materially different. From an AI audit perspective, which corrective action is the MOST direct response to the weakness described, rather than a useful control for some different AI risk?
- Define image quality, preprocessing, provenance and clinical operating boundaries in system requirements
- Establish context-specific evaluation cohorts and acceptable alert burden before threshold approval
- Approve class-specific recall, precision and cost-weighted thresholds with business risk owners
- Perform per-jurisdiction applicability assessment and validate stratified outcomes before rollout
Explanation
Correct Answer
D:Correct. Perform per-jurisdiction applicability assessment and validate stratified outcomes before rollout is aimed at the actual failure: model suitability was not evaluated for each intended deployment population. The audit should subsequently obtain approved population inventory and disaggregated performance tests by subsidiary to test execution, not merely the existence of a policy.
Incorrect Answers
A:“Define image quality, preprocessing, provenance and clinical operating boundaries in system requirements” is a control for input modality, preprocessing and clinical context requirements were omitted, but the audit finding here is model suitability was not evaluated for each intended deployment population. The relevant measure is “Perform per-jurisdiction applicability assessment and validate stratified outcomes before rollout”.
B:“Establish context-specific evaluation cohorts and acceptable alert burden before threshold approval” is a control for normal operational context and false-positive tolerance are absent from design assumptions, but the audit finding here is model suitability was not evaluated for each intended deployment population. The relevant measure is “Perform per-jurisdiction applicability assessment and validate stratified outcomes before rollout”.
C:“Approve class-specific recall, precision and cost-weighted thresholds with business risk owners” is a control for business loss asymmetry is missing from model acceptance criteria, but the audit finding here is model suitability was not evaluated for each intended deployment population. The relevant measure is “Perform per-jurisdiction applicability assessment and validate stratified outcomes before rollout”.
Question 8
In an audit of AI Models, Considerations, and Requirements, managers say this measure is live: Perform per-jurisdiction applicability assessment and validate stratified outcomes before rollout. They claim it corrected a material audit gap. What direct operating evidence would allow the auditor to substantiate that particular claim across a defensible sample?
- Signed multimodal interface specification and testing for unreadable or out-of-scope images
- Season-tagged test results and operations signoff on alert volume and missed anomalies
- Approved population inventory and disaggregated performance tests by subsidiary
- Signed acceptance thresholds, stratified confusion matrices and loss simulations from the held-out release set
Explanation
Correct Answer
C:Correct. The proposed records—approved population inventory and disaggregated performance tests by subsidiary—are directly tied to the asserted safeguard and allow its actual operation to be corroborated. The gap being tested is model suitability was not evaluated for each intended deployment population.
Incorrect Answers
A:“Signed multimodal interface specification and testing for unreadable or out-of-scope images” would test input modality, preprocessing and clinical context requirements were omitted, not the present safeguard against model suitability was not evaluated for each intended deployment population. The expected audit evidence is “Approved population inventory and disaggregated performance tests by subsidiary”.
B:“Season-tagged test results and operations signoff on alert volume and missed anomalies” would test normal operational context and false-positive tolerance are absent from design assumptions, not the present safeguard against model suitability was not evaluated for each intended deployment population. The expected audit evidence is “Approved population inventory and disaggregated performance tests by subsidiary”.
D:“Signed acceptance thresholds, stratified confusion matrices and loss simulations from the held-out release set” would test business loss asymmetry is missing from model acceptance criteria, not the present safeguard against model suitability was not evaluated for each intended deployment population. The expected audit evidence is “Approved population inventory and disaggregated performance tests by subsidiary”.
Question 9
In a review of AI Models, Considerations, and Requirements, investigators cannot establish the expected control proof: approved population inventory and disaggregated performance tests by subsidiary. The underlying event was that a claims model is deployed across subsidiaries whose eligibility rules are materially different. Which control weakness is the MOST likely cause requiring focused audit work?
- Input modality, preprocessing and clinical context requirements were omitted
- Model suitability was not evaluated for each intended deployment population
- Normal operational context and false-positive tolerance are absent from design assumptions
- Business loss asymmetry is missing from model acceptance criteria
Explanation
Correct Answer
B:Correct. The observed condition specifically indicates model suitability was not evaluated for each intended deployment population. A meaningful audit response is to verify this diagnosis and then recommend perform per-jurisdiction applicability assessment and validate stratified outcomes before rollout.
Incorrect Answers
A:“Input modality, preprocessing and clinical context requirements were omitted” concerns a separate audit weakness, not the missing assurance described by model suitability was not evaluated for each intended deployment population. The correct cause is “Model suitability was not evaluated for each intended deployment population”.
C:“Normal operational context and false-positive tolerance are absent from design assumptions” concerns a separate audit weakness, not the missing assurance described by model suitability was not evaluated for each intended deployment population. The correct cause is “Model suitability was not evaluated for each intended deployment population”.
D:“Business loss asymmetry is missing from model acceptance criteria” concerns a separate audit weakness, not the missing assurance described by model suitability was not evaluated for each intended deployment population. The correct cause is “Model suitability was not evaluated for each intended deployment population”.
Question 10
An LLM assistant is presented as a deterministic source of policy truth without any retrieval citation or uncertainty handling. From an AI audit perspective, which corrective action is the MOST direct response to the weakness described, rather than a useful control for some different AI risk?
- Constrain answers to approved retrieved sources and route ungrounded responses to human review
- Compare simpler baselines and ensembles under the same limited-label and explainability criteria
- Test speech models on actual acoustic environments, accents and supported devices
- Require rolling-origin validation with strict time cutoffs and a final unseen period
Explanation
Correct Answer
A:Correct. Constrain answers to approved retrieved sources and route ungrounded responses to human review is aimed at the actual failure: generative probabilistic output is being mistaken for verified authoritative advice. The audit should subsequently obtain grounding evaluation logs with citations, abstention cases and reviewer overrides to test execution, not merely the existence of a policy.
Incorrect Answers
B:“Compare simpler baselines and ensembles under the same limited-label and explainability criteria” is a control for algorithm choice is unsubstantiated against sample size, interpretability and operating cost, but the audit finding here is generative probabilistic output is being mistaken for verified authoritative advice. The relevant measure is “Constrain answers to approved retrieved sources and route ungrounded responses to human review”.
C:“Test speech models on actual acoustic environments, accents and supported devices” is a control for deployment conditions differ from the representative validation environment, but the audit finding here is generative probabilistic output is being mistaken for verified authoritative advice. The relevant measure is “Constrain answers to approved retrieved sources and route ungrounded responses to human review”.
D:“Require rolling-origin validation with strict time cutoffs and a final unseen period” is a control for temporal leakage from future periods contaminates evaluation, but the audit finding here is generative probabilistic output is being mistaken for verified authoritative advice. The relevant measure is “Constrain answers to approved retrieved sources and route ungrounded responses to human review”.