ISACA Advanced in AI Audit AAIA Practice Test 3
Question 20
In an audit of AI Models, Considerations, and Requirements, managers say this measure is live: Reassess purpose, affected users, risk tier and mandatory safeguards before new use. They claim it corrected a material audit gap. What direct operating evidence would allow the auditor to substantiate that particular claim across a defensible sample?
- Field-stratified word error reports tied to approved operational service thresholds
- Signed acceptance thresholds, stratified confusion matrices and loss simulations from the held-out release set
- Change ticket, updated intended-use record and signed risk-owner reauthorization
- Signed multimodal interface specification and testing for unreadable or out-of-scope images
Explanation
Correct Answer
C:Correct. The proposed records—change ticket, updated intended-use record and signed risk-owner reauthorization—are directly tied to the asserted safeguard and allow its actual operation to be corroborated. The gap being tested is material scope expansion bypassed intended-use and risk reapproval.
Incorrect Answers
A:“Field-stratified word error reports tied to approved operational service thresholds” would test deployment conditions differ from the representative validation environment, not the present safeguard against material scope expansion bypassed intended-use and risk reapproval. The expected audit evidence is “Change ticket, updated intended-use record and signed risk-owner reauthorization”.
B:“Signed acceptance thresholds, stratified confusion matrices and loss simulations from the held-out release set” would test business loss asymmetry is missing from model acceptance criteria, not the present safeguard against material scope expansion bypassed intended-use and risk reapproval. The expected audit evidence is “Change ticket, updated intended-use record and signed risk-owner reauthorization”.
D:“Signed multimodal interface specification and testing for unreadable or out-of-scope images” would test input modality, preprocessing and clinical context requirements were omitted, not the present safeguard against material scope expansion bypassed intended-use and risk reapproval. The expected audit evidence is “Change ticket, updated intended-use record and signed risk-owner reauthorization”.
Question 21
In a review of AI Models, Considerations, and Requirements, investigators cannot establish the expected control proof: change ticket, updated intended-use record and signed risk-owner reauthorization. The underlying event was that a customer chatbot is promoted from product FAQ to financial advice without revisiting its risk classification. Which control weakness is the MOST likely cause requiring focused audit work?
- Deployment conditions differ from the representative validation environment
- Material scope expansion bypassed intended-use and risk reapproval
- Business loss asymmetry is missing from model acceptance criteria
- Input modality, preprocessing and clinical context requirements were omitted
Explanation
Correct Answer
B:Correct. The observed condition specifically indicates material scope expansion bypassed intended-use and risk reapproval. A meaningful audit response is to verify this diagnosis and then recommend reassess purpose, affected users, risk tier and mandatory safeguards before new use.
Incorrect Answers
A:“Deployment conditions differ from the representative validation environment” concerns a separate audit weakness, not the missing assurance described by material scope expansion bypassed intended-use and risk reapproval. The correct cause is “Material scope expansion bypassed intended-use and risk reapproval”.
C:“Business loss asymmetry is missing from model acceptance criteria” concerns a separate audit weakness, not the missing assurance described by material scope expansion bypassed intended-use and risk reapproval. The correct cause is “Material scope expansion bypassed intended-use and risk reapproval”.
D:“Input modality, preprocessing and clinical context requirements were omitted” concerns a separate audit weakness, not the missing assurance described by material scope expansion bypassed intended-use and risk reapproval. The correct cause is “Material scope expansion bypassed intended-use and risk reapproval”.
Question 22
An anomaly detector treats every extreme reading as a threat despite known seasonal maintenance windows. From an AI audit perspective, which corrective action is the MOST direct response to the weakness described, rather than a useful control for some different AI risk?
- Establish context-specific evaluation cohorts and acceptable alert burden before threshold approval
- Define tail-latency, fail-safe response and reliability service objectives in selection gates
- Require rolling-origin validation with strict time cutoffs and a final unseen period
- Compare simpler baselines and ensembles under the same limited-label and explainability criteria
Explanation
Correct Answer
A:Correct. Establish context-specific evaluation cohorts and acceptable alert burden before threshold approval is aimed at the actual failure: normal operational context and false-positive tolerance are absent from design assumptions. The audit should subsequently obtain season-tagged test results and operations signoff on alert volume and missed anomalies to test execution, not merely the existence of a policy.
Incorrect Answers
B:“Define tail-latency, fail-safe response and reliability service objectives in selection gates” is a control for nonfunctional latency and availability requirements were excluded from model suitability, but the audit finding here is normal operational context and false-positive tolerance are absent from design assumptions. The relevant measure is “Establish context-specific evaluation cohorts and acceptable alert burden before threshold approval”.
C:“Require rolling-origin validation with strict time cutoffs and a final unseen period” is a control for temporal leakage from future periods contaminates evaluation, but the audit finding here is normal operational context and false-positive tolerance are absent from design assumptions. The relevant measure is “Establish context-specific evaluation cohorts and acceptable alert burden before threshold approval”.
D:“Compare simpler baselines and ensembles under the same limited-label and explainability criteria” is a control for algorithm choice is unsubstantiated against sample size, interpretability and operating cost, but the audit finding here is normal operational context and false-positive tolerance are absent from design assumptions. The relevant measure is “Establish context-specific evaluation cohorts and acceptable alert burden before threshold approval”.
Question 23
In an audit of AI Models, Considerations, and Requirements, managers say this measure is live: Establish context-specific evaluation cohorts and acceptable alert burden before threshold approval. They claim it corrected a material audit gap. What direct operating evidence would allow the auditor to substantiate that particular claim across a defensible sample?
- p95/p99 traces, failover drills and completed acceptance results against approved SLAs
- Dated split manifests and out-of-time backtests showing no future records in features
- Reproducible benchmark notebook with baseline metrics, error analysis and deployment-cost estimates
- Season-tagged test results and operations signoff on alert volume and missed anomalies
Explanation
Correct Answer
D:Correct. The proposed records—season-tagged test results and operations signoff on alert volume and missed anomalies—are directly tied to the asserted safeguard and allow its actual operation to be corroborated. The gap being tested is normal operational context and false-positive tolerance are absent from design assumptions.
Incorrect Answers
A:“p95/p99 traces, failover drills and completed acceptance results against approved SLAs” would test nonfunctional latency and availability requirements were excluded from model suitability, not the present safeguard against normal operational context and false-positive tolerance are absent from design assumptions. The expected audit evidence is “Season-tagged test results and operations signoff on alert volume and missed anomalies”.
B:“Dated split manifests and out-of-time backtests showing no future records in features” would test temporal leakage from future periods contaminates evaluation, not the present safeguard against normal operational context and false-positive tolerance are absent from design assumptions. The expected audit evidence is “Season-tagged test results and operations signoff on alert volume and missed anomalies”.
C:“Reproducible benchmark notebook with baseline metrics, error analysis and deployment-cost estimates” would test algorithm choice is unsubstantiated against sample size, interpretability and operating cost, not the present safeguard against normal operational context and false-positive tolerance are absent from design assumptions. The expected audit evidence is “Season-tagged test results and operations signoff on alert volume and missed anomalies”.
Question 24
In a review of AI Models, Considerations, and Requirements, investigators cannot establish the expected control proof: season-tagged test results and operations signoff on alert volume and missed anomalies. The underlying event was that an anomaly detector treats every extreme reading as a threat despite known seasonal maintenance windows. Which control weakness is the MOST likely cause requiring focused audit work?
- Nonfunctional latency and availability requirements were excluded from model suitability
- Temporal leakage from future periods contaminates evaluation
- Normal operational context and false-positive tolerance are absent from design assumptions
- Algorithm choice is unsubstantiated against sample size, interpretability and operating cost
Explanation
Correct Answer
C:Correct. The observed condition specifically indicates normal operational context and false-positive tolerance are absent from design assumptions. A meaningful audit response is to verify this diagnosis and then recommend establish context-specific evaluation cohorts and acceptable alert burden before threshold approval.
Incorrect Answers
A:“Nonfunctional latency and availability requirements were excluded from model suitability” concerns a separate audit weakness, not the missing assurance described by normal operational context and false-positive tolerance are absent from design assumptions. The correct cause is “Normal operational context and false-positive tolerance are absent from design assumptions”.
B:“Temporal leakage from future periods contaminates evaluation” concerns a separate audit weakness, not the missing assurance described by normal operational context and false-positive tolerance are absent from design assumptions. The correct cause is “Normal operational context and false-positive tolerance are absent from design assumptions”.
D:“Algorithm choice is unsubstantiated against sample size, interpretability and operating cost” concerns a separate audit weakness, not the missing assurance described by normal operational context and false-positive tolerance are absent from design assumptions. The correct cause is “Normal operational context and false-positive tolerance are absent from design assumptions”.
Question 25
A team evaluates speech recognition only on quiet office recordings although it will serve noisy field technicians. From an AI audit perspective, which corrective action is the MOST direct response to the weakness described, rather than a useful control for some different AI risk?
- Specify tool permissions, spend limits, approvals and rollback semantics for every autonomous action
- Test speech models on actual acoustic environments, accents and supported devices
- Perform per-jurisdiction applicability assessment and validate stratified outcomes before rollout
- Reassess purpose, affected users, risk tier and mandatory safeguards before new use
Explanation
Correct Answer
B:Correct. Test speech models on actual acoustic environments, accents and supported devices is aimed at the actual failure: deployment conditions differ from the representative validation environment. The audit should subsequently obtain field-stratified word error reports tied to approved operational service thresholds to test execution, not merely the existence of a policy.
Incorrect Answers
A:“Specify tool permissions, spend limits, approvals and rollback semantics for every autonomous action” is a control for delegated action authority and transaction boundaries were omitted, but the audit finding here is deployment conditions differ from the representative validation environment. The relevant measure is “Test speech models on actual acoustic environments, accents and supported devices”.
C:“Perform per-jurisdiction applicability assessment and validate stratified outcomes before rollout” is a control for model suitability was not evaluated for each intended deployment population, but the audit finding here is deployment conditions differ from the representative validation environment. The relevant measure is “Test speech models on actual acoustic environments, accents and supported devices”.
D:“Reassess purpose, affected users, risk tier and mandatory safeguards before new use” is a control for material scope expansion bypassed intended-use and risk reapproval, but the audit finding here is deployment conditions differ from the representative validation environment. The relevant measure is “Test speech models on actual acoustic environments, accents and supported devices”.
Question 26
In an audit of AI Models, Considerations, and Requirements, managers say this measure is live: Test speech models on actual acoustic environments, accents and supported devices. They claim it corrected a material audit gap. What direct operating evidence would allow the auditor to substantiate that particular claim across a defensible sample?
- Field-stratified word error reports tied to approved operational service thresholds
- Approved tool-use matrix and negative tests for unauthorized purchase requests
- Approved population inventory and disaggregated performance tests by subsidiary
- Change ticket, updated intended-use record and signed risk-owner reauthorization
Explanation
Correct Answer
A:Correct. The proposed records—field-stratified word error reports tied to approved operational service thresholds—are directly tied to the asserted safeguard and allow its actual operation to be corroborated. The gap being tested is deployment conditions differ from the representative validation environment.
Incorrect Answers
B:“Approved tool-use matrix and negative tests for unauthorized purchase requests” would test delegated action authority and transaction boundaries were omitted, not the present safeguard against deployment conditions differ from the representative validation environment. The expected audit evidence is “Field-stratified word error reports tied to approved operational service thresholds”.
C:“Approved population inventory and disaggregated performance tests by subsidiary” would test model suitability was not evaluated for each intended deployment population, not the present safeguard against deployment conditions differ from the representative validation environment. The expected audit evidence is “Field-stratified word error reports tied to approved operational service thresholds”.
D:“Change ticket, updated intended-use record and signed risk-owner reauthorization” would test material scope expansion bypassed intended-use and risk reapproval, not the present safeguard against deployment conditions differ from the representative validation environment. The expected audit evidence is “Field-stratified word error reports tied to approved operational service thresholds”.
Question 27
In a review of AI Models, Considerations, and Requirements, investigators cannot establish the expected control proof: field-stratified word error reports tied to approved operational service thresholds. The underlying event was that a team evaluates speech recognition only on quiet office recordings although it will serve noisy field technicians. Which control weakness is the MOST likely cause requiring focused audit work?
- Delegated action authority and transaction boundaries were omitted
- Model suitability was not evaluated for each intended deployment population
- Material scope expansion bypassed intended-use and risk reapproval
- Deployment conditions differ from the representative validation environment
Explanation
Correct Answer
D:Correct. The observed condition specifically indicates deployment conditions differ from the representative validation environment. A meaningful audit response is to verify this diagnosis and then recommend test speech models on actual acoustic environments, accents and supported devices.
Incorrect Answers
A:“Delegated action authority and transaction boundaries were omitted” concerns a separate audit weakness, not the missing assurance described by deployment conditions differ from the representative validation environment. The correct cause is “Deployment conditions differ from the representative validation environment”.
B:“Model suitability was not evaluated for each intended deployment population” concerns a separate audit weakness, not the missing assurance described by deployment conditions differ from the representative validation environment. The correct cause is “Deployment conditions differ from the representative validation environment”.
C:“Material scope expansion bypassed intended-use and risk reapproval” concerns a separate audit weakness, not the missing assurance described by deployment conditions differ from the representative validation environment. The correct cause is “Deployment conditions differ from the representative validation environment”.
Question 28
A real-time scoring API occasionally exceeds a response deadline but has excellent average accuracy. From an AI audit perspective, which corrective action is the MOST direct response to the weakness described, rather than a useful control for some different AI risk?
- Approve class-specific recall, precision and cost-weighted thresholds with business risk owners
- Constrain answers to approved retrieved sources and route ungrounded responses to human review
- Define tail-latency, fail-safe response and reliability service objectives in selection gates
- Establish context-specific evaluation cohorts and acceptable alert burden before threshold approval
Explanation
Correct Answer
C:Correct. Define tail-latency, fail-safe response and reliability service objectives in selection gates is aimed at the actual failure: nonfunctional latency and availability requirements were excluded from model suitability. The audit should subsequently obtain p95/p99 traces, failover drills and completed acceptance results against approved slas to test execution, not merely the existence of a policy.
Incorrect Answers
A:“Approve class-specific recall, precision and cost-weighted thresholds with business risk owners” is a control for business loss asymmetry is missing from model acceptance criteria, but the audit finding here is nonfunctional latency and availability requirements were excluded from model suitability. The relevant measure is “Define tail-latency, fail-safe response and reliability service objectives in selection gates”.
B:“Constrain answers to approved retrieved sources and route ungrounded responses to human review” is a control for generative probabilistic output is being mistaken for verified authoritative advice, but the audit finding here is nonfunctional latency and availability requirements were excluded from model suitability. The relevant measure is “Define tail-latency, fail-safe response and reliability service objectives in selection gates”.
D:“Establish context-specific evaluation cohorts and acceptable alert burden before threshold approval” is a control for normal operational context and false-positive tolerance are absent from design assumptions, but the audit finding here is nonfunctional latency and availability requirements were excluded from model suitability. The relevant measure is “Define tail-latency, fail-safe response and reliability service objectives in selection gates”.
Question 29
In an audit of AI Models, Considerations, and Requirements, managers say this measure is live: Define tail-latency, fail-safe response and reliability service objectives in selection gates. They claim it corrected a material audit gap. What direct operating evidence would allow the auditor to substantiate that particular claim across a defensible sample?
- Signed acceptance thresholds, stratified confusion matrices and loss simulations from the held-out release set
- p95/p99 traces, failover drills and completed acceptance results against approved SLAs
- Grounding evaluation logs with citations, abstention cases and reviewer overrides
- Season-tagged test results and operations signoff on alert volume and missed anomalies
Explanation
Correct Answer
B:Correct. The proposed records—p95/p99 traces, failover drills and completed acceptance results against approved slas—are directly tied to the asserted safeguard and allow its actual operation to be corroborated. The gap being tested is nonfunctional latency and availability requirements were excluded from model suitability.
Incorrect Answers
A:“Signed acceptance thresholds, stratified confusion matrices and loss simulations from the held-out release set” would test business loss asymmetry is missing from model acceptance criteria, not the present safeguard against nonfunctional latency and availability requirements were excluded from model suitability. The expected audit evidence is “p95/p99 traces, failover drills and completed acceptance results against approved SLAs”.
C:“Grounding evaluation logs with citations, abstention cases and reviewer overrides” would test generative probabilistic output is being mistaken for verified authoritative advice, not the present safeguard against nonfunctional latency and availability requirements were excluded from model suitability. The expected audit evidence is “p95/p99 traces, failover drills and completed acceptance results against approved SLAs”.
D:“Season-tagged test results and operations signoff on alert volume and missed anomalies” would test normal operational context and false-positive tolerance are absent from design assumptions, not the present safeguard against nonfunctional latency and availability requirements were excluded from model suitability. The expected audit evidence is “p95/p99 traces, failover drills and completed acceptance results against approved SLAs”.
Question 30
In a review of AI Models, Considerations, and Requirements, investigators cannot establish the expected control proof: p95/p99 traces, failover drills and completed acceptance results against approved slas. The underlying event was that a real-time scoring API occasionally exceeds a response deadline but has excellent average accuracy. Which control weakness is the MOST likely cause requiring focused audit work?
- Nonfunctional latency and availability requirements were excluded from model suitability
- Business loss asymmetry is missing from model acceptance criteria
- Generative probabilistic output is being mistaken for verified authoritative advice
- Normal operational context and false-positive tolerance are absent from design assumptions
Explanation
Correct Answer
A:Correct. The observed condition specifically indicates nonfunctional latency and availability requirements were excluded from model suitability. A meaningful audit response is to verify this diagnosis and then recommend define tail-latency, fail-safe response and reliability service objectives in selection gates.
Incorrect Answers
B:“Business loss asymmetry is missing from model acceptance criteria” concerns a separate audit weakness, not the missing assurance described by nonfunctional latency and availability requirements were excluded from model suitability. The correct cause is “Nonfunctional latency and availability requirements were excluded from model suitability”.
C:“Generative probabilistic output is being mistaken for verified authoritative advice” concerns a separate audit weakness, not the missing assurance described by nonfunctional latency and availability requirements were excluded from model suitability. The correct cause is “Nonfunctional latency and availability requirements were excluded from model suitability”.
D:“Normal operational context and false-positive tolerance are absent from design assumptions” concerns a separate audit weakness, not the missing assurance described by nonfunctional latency and availability requirements were excluded from model suitability. The correct cause is “Nonfunctional latency and availability requirements were excluded from model suitability”.