ISACA Advanced in AI Audit AAIA Practice Test 33
Question 320
In an audit of Testing Techniques for AI Solutions, managers say this measure is live: Calibrate judge assessments against blinded human rubrics and counterexamples. They claim it corrected a material audit gap. What direct operating evidence would allow the auditor to substantiate that particular claim across a defensible sample?
- Production image stratification and condition-specific error rates
- Confusion matrix, PR curve and threshold cost analysis
- Inter-rater comparison, blinded sample and judge-bias analysis
- Failing schema test records and explicit rejection responses
Explanation
Correct Answer
C:Correct. The proposed records—inter-rater comparison, blinded sample and judge-bias analysis—are directly tied to the asserted safeguard and allow its actual operation to be corroborated. The gap being tested is evaluation is biased by the automated judge's stylistic preference.
Incorrect Answers
A:“Production image stratification and condition-specific error rates” would test test dataset lacks representative camera and environmental conditions, not the present safeguard against evaluation is biased by the automated judge's stylistic preference. The expected audit evidence is “Inter-rater comparison, blinded sample and judge-bias analysis”.
B:“Confusion matrix, PR curve and threshold cost analysis” would test class imbalance makes accuracy alone a misleading test metric, not the present safeguard against evaluation is biased by the automated judge's stylistic preference. The expected audit evidence is “Inter-rater comparison, blinded sample and judge-bias analysis”.
D:“Failing schema test records and explicit rejection responses” would test negative testing did not exercise invalid or incomplete feature contracts, not the present safeguard against evaluation is biased by the automated judge's stylistic preference. The expected audit evidence is “Inter-rater comparison, blinded sample and judge-bias analysis”.
Question 321
In a review of Testing Techniques for AI Solutions, investigators cannot establish the expected control proof: inter-rater comparison, blinded sample and judge-bias analysis. The underlying event was that an LLM output evaluation uses a single model judge that consistently favors verbose answers. Which control weakness is the MOST likely cause requiring focused audit work?
- Test dataset lacks representative camera and environmental conditions
- Evaluation is biased by the automated judge's stylistic preference
- Class imbalance makes accuracy alone a misleading test metric
- Negative testing did not exercise invalid or incomplete feature contracts
Explanation
Correct Answer
B:Correct. The observed condition specifically indicates evaluation is biased by the automated judge's stylistic preference. A meaningful audit response is to verify this diagnosis and then recommend calibrate judge assessments against blinded human rubrics and counterexamples.
Incorrect Answers
A:“Test dataset lacks representative camera and environmental conditions” concerns a separate audit weakness, not the missing assurance described by evaluation is biased by the automated judge's stylistic preference. The correct cause is “Evaluation is biased by the automated judge's stylistic preference”.
C:“Class imbalance makes accuracy alone a misleading test metric” concerns a separate audit weakness, not the missing assurance described by evaluation is biased by the automated judge's stylistic preference. The correct cause is “Evaluation is biased by the automated judge's stylistic preference”.
D:“Negative testing did not exercise invalid or incomplete feature contracts” concerns a separate audit weakness, not the missing assurance described by evaluation is biased by the automated judge's stylistic preference. The correct cause is “Evaluation is biased by the automated judge's stylistic preference”.
Question 322
A model performs well with complete inputs but fails when a cloud data feed is delayed. From an AI audit perspective, which corrective action is the MOST direct response to the weakness described, rather than a useful control for some different AI risk?
- Inject late, stale and missing feature feeds and verify safe fallback
- Execute denied tool-call, excessive-permission and approval-bypass test cases
- Report uncertainty and obtain representative subgroup samples before fairness signoff
- Red-team untrusted retrieved content and verify trust-boundary enforcement
Explanation
Correct Answer
A:Correct. Inject late, stale and missing feature feeds and verify safe fallback is aimed at the actual failure: resilience tests excluded degraded dependency states. The audit should subsequently obtain failure-injection reports and observed fallback decisions to test execution, not merely the existence of a policy.
Incorrect Answers
B:“Execute denied tool-call, excessive-permission and approval-bypass test cases” is a control for security negative testing omitted privilege escalation and action abuse, but the audit finding here is resilience tests excluded degraded dependency states. The relevant measure is “Inject late, stale and missing feature feeds and verify safe fallback”.
C:“Report uncertainty and obtain representative subgroup samples before fairness signoff” is a control for subgroup sample size is inadequate to support equity conclusions, but the audit finding here is resilience tests excluded degraded dependency states. The relevant measure is “Inject late, stale and missing feature feeds and verify safe fallback”.
D:“Red-team untrusted retrieved content and verify trust-boundary enforcement” is a control for indirect prompt-injection resistance was not assessed before release, but the audit finding here is resilience tests excluded degraded dependency states. The relevant measure is “Inject late, stale and missing feature feeds and verify safe fallback”.
Question 323
In an audit of Testing Techniques for AI Solutions, managers say this measure is live: Inject late, stale and missing feature feeds and verify safe fallback. They claim it corrected a material audit gap. What direct operating evidence would allow the auditor to substantiate that particular claim across a defensible sample?
- Blocked tool-call traces and least-privilege test coverage report
- Confidence intervals, power assessment and subgroup sample-augmentation plan
- Injection corpus, tool-call denials and reviewed attack replay logs
- Failure-injection reports and observed fallback decisions
Explanation
Correct Answer
D:Correct. The proposed records—failure-injection reports and observed fallback decisions—are directly tied to the asserted safeguard and allow its actual operation to be corroborated. The gap being tested is resilience tests excluded degraded dependency states.
Incorrect Answers
A:“Blocked tool-call traces and least-privilege test coverage report” would test security negative testing omitted privilege escalation and action abuse, not the present safeguard against resilience tests excluded degraded dependency states. The expected audit evidence is “Failure-injection reports and observed fallback decisions”.
B:“Confidence intervals, power assessment and subgroup sample-augmentation plan” would test subgroup sample size is inadequate to support equity conclusions, not the present safeguard against resilience tests excluded degraded dependency states. The expected audit evidence is “Failure-injection reports and observed fallback decisions”.
C:“Injection corpus, tool-call denials and reviewed attack replay logs” would test indirect prompt-injection resistance was not assessed before release, not the present safeguard against resilience tests excluded degraded dependency states. The expected audit evidence is “Failure-injection reports and observed fallback decisions”.
Question 324
In a review of Testing Techniques for AI Solutions, investigators cannot establish the expected control proof: failure-injection reports and observed fallback decisions. The underlying event was that a model performs well with complete inputs but fails when a cloud data feed is delayed. Which control weakness is the MOST likely cause requiring focused audit work?
- Security negative testing omitted privilege escalation and action abuse
- Subgroup sample size is inadequate to support equity conclusions
- Resilience tests excluded degraded dependency states
- Indirect prompt-injection resistance was not assessed before release
Explanation
Correct Answer
C:Correct. The observed condition specifically indicates resilience tests excluded degraded dependency states. A meaningful audit response is to verify this diagnosis and then recommend inject late, stale and missing feature feeds and verify safe fallback.
Incorrect Answers
A:“Security negative testing omitted privilege escalation and action abuse” concerns a separate audit weakness, not the missing assurance described by resilience tests excluded degraded dependency states. The correct cause is “Resilience tests excluded degraded dependency states”.
B:“Subgroup sample size is inadequate to support equity conclusions” concerns a separate audit weakness, not the missing assurance described by resilience tests excluded degraded dependency states. The correct cause is “Resilience tests excluded degraded dependency states”.
D:“Indirect prompt-injection resistance was not assessed before release” concerns a separate audit weakness, not the missing assurance described by resilience tests excluded degraded dependency states. The correct cause is “Resilience tests excluded degraded dependency states”.
Question 325
A vision model labels defects well on lab photographs but fails under production lighting. From an AI audit perspective, which corrective action is the MOST direct response to the weakness described, rather than a useful control for some different AI risk?
- Evaluate calibration across score bands and changing populations
- Build field-condition holdouts and test illumination, occlusion and device shifts
- Use out-of-time validation and compare performance across business cycles
- Calibrate judge assessments against blinded human rubrics and counterexamples
Explanation
Correct Answer
B:Correct. Build field-condition holdouts and test illumination, occlusion and device shifts is aimed at the actual failure: test dataset lacks representative camera and environmental conditions. The audit should subsequently obtain production image stratification and condition-specific error rates to test execution, not merely the existence of a policy.
Incorrect Answers
A:“Evaluate calibration across score bands and changing populations” is a control for calibration and confidence reliability were excluded from acceptance testing, but the audit finding here is test dataset lacks representative camera and environmental conditions. The relevant measure is “Build field-condition holdouts and test illumination, occlusion and device shifts”.
C:“Use out-of-time validation and compare performance across business cycles” is a control for temporal overfitting was not detected by training-only evaluation, but the audit finding here is test dataset lacks representative camera and environmental conditions. The relevant measure is “Build field-condition holdouts and test illumination, occlusion and device shifts”.
D:“Calibrate judge assessments against blinded human rubrics and counterexamples” is a control for evaluation is biased by the automated judge's stylistic preference, but the audit finding here is test dataset lacks representative camera and environmental conditions. The relevant measure is “Build field-condition holdouts and test illumination, occlusion and device shifts”.
Question 326
In an audit of Testing Techniques for AI Solutions, managers say this measure is live: Build field-condition holdouts and test illumination, occlusion and device shifts. They claim it corrected a material audit gap. What direct operating evidence would allow the auditor to substantiate that particular claim across a defensible sample?
- Production image stratification and condition-specific error rates
- Reliability diagrams, Brier scores and out-of-sample calibration tables
- Rolling-window backtest with held-out date boundaries
- Inter-rater comparison, blinded sample and judge-bias analysis
Explanation
Correct Answer
A:Correct. The proposed records—production image stratification and condition-specific error rates—are directly tied to the asserted safeguard and allow its actual operation to be corroborated. The gap being tested is test dataset lacks representative camera and environmental conditions.
Incorrect Answers
B:“Reliability diagrams, Brier scores and out-of-sample calibration tables” would test calibration and confidence reliability were excluded from acceptance testing, not the present safeguard against test dataset lacks representative camera and environmental conditions. The expected audit evidence is “Production image stratification and condition-specific error rates”.
C:“Rolling-window backtest with held-out date boundaries” would test temporal overfitting was not detected by training-only evaluation, not the present safeguard against test dataset lacks representative camera and environmental conditions. The expected audit evidence is “Production image stratification and condition-specific error rates”.
D:“Inter-rater comparison, blinded sample and judge-bias analysis” would test evaluation is biased by the automated judge's stylistic preference, not the present safeguard against test dataset lacks representative camera and environmental conditions. The expected audit evidence is “Production image stratification and condition-specific error rates”.
Question 327
In a review of Testing Techniques for AI Solutions, investigators cannot establish the expected control proof: production image stratification and condition-specific error rates. The underlying event was that a vision model labels defects well on lab photographs but fails under production lighting. Which control weakness is the MOST likely cause requiring focused audit work?
- Calibration and confidence reliability were excluded from acceptance testing
- Temporal overfitting was not detected by training-only evaluation
- Evaluation is biased by the automated judge's stylistic preference
- Test dataset lacks representative camera and environmental conditions
Explanation
Correct Answer
D:Correct. The observed condition specifically indicates test dataset lacks representative camera and environmental conditions. A meaningful audit response is to verify this diagnosis and then recommend build field-condition holdouts and test illumination, occlusion and device shifts.
Incorrect Answers
A:“Calibration and confidence reliability were excluded from acceptance testing” concerns a separate audit weakness, not the missing assurance described by test dataset lacks representative camera and environmental conditions. The correct cause is “Test dataset lacks representative camera and environmental conditions”.
B:“Temporal overfitting was not detected by training-only evaluation” concerns a separate audit weakness, not the missing assurance described by test dataset lacks representative camera and environmental conditions. The correct cause is “Test dataset lacks representative camera and environmental conditions”.
C:“Evaluation is biased by the automated judge's stylistic preference” concerns a separate audit weakness, not the missing assurance described by test dataset lacks representative camera and environmental conditions. The correct cause is “Test dataset lacks representative camera and environmental conditions”.
Question 328
An AI agent's tool-use tests verify intended actions but never attempt unauthorized actions. From an AI audit perspective, which corrective action is the MOST direct response to the weakness described, rather than a useful control for some different AI risk?
- Measure fraud recall, precision and cost-sensitive error tradeoffs
- Test typographic, paraphrase and input corruption sensitivity
- Execute denied tool-call, excessive-permission and approval-bypass test cases
- Inject late, stale and missing feature feeds and verify safe fallback
Explanation
Correct Answer
C:Correct. Execute denied tool-call, excessive-permission and approval-bypass test cases is aimed at the actual failure: security negative testing omitted privilege escalation and action abuse. The audit should subsequently obtain blocked tool-call traces and least-privilege test coverage report to test execution, not merely the existence of a policy.
Incorrect Answers
A:“Measure fraud recall, precision and cost-sensitive error tradeoffs” is a control for class imbalance makes accuracy alone a misleading test metric, but the audit finding here is security negative testing omitted privilege escalation and action abuse. The relevant measure is “Execute denied tool-call, excessive-permission and approval-bypass test cases”.
B:“Test typographic, paraphrase and input corruption sensitivity” is a control for robustness tests omitted realistic noisy and adversarial perturbations, but the audit finding here is security negative testing omitted privilege escalation and action abuse. The relevant measure is “Execute denied tool-call, excessive-permission and approval-bypass test cases”.
D:“Inject late, stale and missing feature feeds and verify safe fallback” is a control for resilience tests excluded degraded dependency states, but the audit finding here is security negative testing omitted privilege escalation and action abuse. The relevant measure is “Execute denied tool-call, excessive-permission and approval-bypass test cases”.
Question 329
In an audit of Testing Techniques for AI Solutions, managers say this measure is live: Execute denied tool-call, excessive-permission and approval-bypass test cases. They claim it corrected a material audit gap. What direct operating evidence would allow the auditor to substantiate that particular claim across a defensible sample?
- Confusion matrix, PR curve and threshold cost analysis
- Blocked tool-call traces and least-privilege test coverage report
- Perturbation test results with categorized failures and remediation
- Failure-injection reports and observed fallback decisions
Explanation
Correct Answer
B:Correct. The proposed records—blocked tool-call traces and least-privilege test coverage report—are directly tied to the asserted safeguard and allow its actual operation to be corroborated. The gap being tested is security negative testing omitted privilege escalation and action abuse.
Incorrect Answers
A:“Confusion matrix, PR curve and threshold cost analysis” would test class imbalance makes accuracy alone a misleading test metric, not the present safeguard against security negative testing omitted privilege escalation and action abuse. The expected audit evidence is “Blocked tool-call traces and least-privilege test coverage report”.
C:“Perturbation test results with categorized failures and remediation” would test robustness tests omitted realistic noisy and adversarial perturbations, not the present safeguard against security negative testing omitted privilege escalation and action abuse. The expected audit evidence is “Blocked tool-call traces and least-privilege test coverage report”.
D:“Failure-injection reports and observed fallback decisions” would test resilience tests excluded degraded dependency states, not the present safeguard against security negative testing omitted privilege escalation and action abuse. The expected audit evidence is “Blocked tool-call traces and least-privilege test coverage report”.
Question 330
In a review of Testing Techniques for AI Solutions, investigators cannot establish the expected control proof: blocked tool-call traces and least-privilege test coverage report. The underlying event was that an AI agent's tool-use tests verify intended actions but never attempt unauthorized actions. Which control weakness is the MOST likely cause requiring focused audit work?
- Security negative testing omitted privilege escalation and action abuse
- Class imbalance makes accuracy alone a misleading test metric
- Robustness tests omitted realistic noisy and adversarial perturbations
- Resilience tests excluded degraded dependency states
Explanation
Correct Answer
A:Correct. The observed condition specifically indicates security negative testing omitted privilege escalation and action abuse. A meaningful audit response is to verify this diagnosis and then recommend execute denied tool-call, excessive-permission and approval-bypass test cases.
Incorrect Answers
B:“Class imbalance makes accuracy alone a misleading test metric” concerns a separate audit weakness, not the missing assurance described by security negative testing omitted privilege escalation and action abuse. The correct cause is “Security negative testing omitted privilege escalation and action abuse”.
C:“Robustness tests omitted realistic noisy and adversarial perturbations” concerns a separate audit weakness, not the missing assurance described by security negative testing omitted privilege escalation and action abuse. The correct cause is “Security negative testing omitted privilege escalation and action abuse”.
D:“Resilience tests excluded degraded dependency states” concerns a separate audit weakness, not the missing assurance described by security negative testing omitted privilege escalation and action abuse. The correct cause is “Security negative testing omitted privilege escalation and action abuse”.