ISACA Advanced in AI Audit AAIA Practice Test 26
Question 250
A security patch alters a vector database's retrieval ranking but the AI team considers it unrelated infrastructure. From an AI audit perspective, which corrective action is the MOST direct response to the weakness described, rather than a useful control for some different AI risk?
- Require retrieval and answer-quality regression after dependent infrastructure changes
- Assess output-policy changes with workflow owners and revalidate impacted processes
- Version and roll back model, data contract, prompt and dependent services together
- Use controlled canaries with guardrail metrics and automatic pause thresholds
Explanation
Correct Answer
A:Correct. Require retrieval and answer-quality regression after dependent infrastructure changes is aimed at the actual failure: supporting-component changes were excluded from end-to-end behavioral testing. The audit should subsequently obtain before/after retrieval rankings, grounding metrics and release tickets to test execution, not merely the existence of a policy.
Incorrect Answers
B:“Assess output-policy changes with workflow owners and revalidate impacted processes” is a control for vendor policy revisions bypass business-impact communication and acceptance, but the audit finding here is supporting-component changes were excluded from end-to-end behavioral testing. The relevant measure is “Require retrieval and answer-quality regression after dependent infrastructure changes”.
C:“Version and roll back model, data contract, prompt and dependent services together” is a control for recovery plan is incomplete across the full inference dependency bundle, but the audit finding here is supporting-component changes were excluded from end-to-end behavioral testing. The relevant measure is “Require retrieval and answer-quality regression after dependent infrastructure changes”.
D:“Use controlled canaries with guardrail metrics and automatic pause thresholds” is a control for progressive delivery and stop criteria were not included in rollout governance, but the audit finding here is supporting-component changes were excluded from end-to-end behavioral testing. The relevant measure is “Require retrieval and answer-quality regression after dependent infrastructure changes”.
Question 251
In an audit of Change Management Specific to AI, managers say this measure is live: Require retrieval and answer-quality regression after dependent infrastructure changes. They claim it corrected a material audit gap. What direct operating evidence would allow the auditor to substantiate that particular claim across a defensible sample?
- Vendor policy diff, business test results and signed readiness decision
- End-to-end rollback rehearsal with matching artifact manifest
- Canary dashboard, pause trigger history and deployment decision log
- Before/after retrieval rankings, grounding metrics and release tickets
Explanation
Correct Answer
D:Correct. The proposed records—before/after retrieval rankings, grounding metrics and release tickets—are directly tied to the asserted safeguard and allow its actual operation to be corroborated. The gap being tested is supporting-component changes were excluded from end-to-end behavioral testing.
Incorrect Answers
A:“Vendor policy diff, business test results and signed readiness decision” would test vendor policy revisions bypass business-impact communication and acceptance, not the present safeguard against supporting-component changes were excluded from end-to-end behavioral testing. The expected audit evidence is “Before/after retrieval rankings, grounding metrics and release tickets”.
B:“End-to-end rollback rehearsal with matching artifact manifest” would test recovery plan is incomplete across the full inference dependency bundle, not the present safeguard against supporting-component changes were excluded from end-to-end behavioral testing. The expected audit evidence is “Before/after retrieval rankings, grounding metrics and release tickets”.
C:“Canary dashboard, pause trigger history and deployment decision log” would test progressive delivery and stop criteria were not included in rollout governance, not the present safeguard against supporting-component changes were excluded from end-to-end behavioral testing. The expected audit evidence is “Before/after retrieval rankings, grounding metrics and release tickets”.
Question 252
In a review of Change Management Specific to AI, investigators cannot establish the expected control proof: before/after retrieval rankings, grounding metrics and release tickets. The underlying event was that a security patch alters a vector database's retrieval ranking but the AI team considers it unrelated infrastructure. Which control weakness is the MOST likely cause requiring focused audit work?
- Vendor policy revisions bypass business-impact communication and acceptance
- Recovery plan is incomplete across the full inference dependency bundle
- Supporting-component changes were excluded from end-to-end behavioral testing
- Progressive delivery and stop criteria were not included in rollout governance
Explanation
Correct Answer
C:Correct. The observed condition specifically indicates supporting-component changes were excluded from end-to-end behavioral testing. A meaningful audit response is to verify this diagnosis and then recommend require retrieval and answer-quality regression after dependent infrastructure changes.
Incorrect Answers
A:“Vendor policy revisions bypass business-impact communication and acceptance” concerns a separate audit weakness, not the missing assurance described by supporting-component changes were excluded from end-to-end behavioral testing. The correct cause is “Supporting-component changes were excluded from end-to-end behavioral testing”.
B:“Recovery plan is incomplete across the full inference dependency bundle” concerns a separate audit weakness, not the missing assurance described by supporting-component changes were excluded from end-to-end behavioral testing. The correct cause is “Supporting-component changes were excluded from end-to-end behavioral testing”.
D:“Progressive delivery and stop criteria were not included in rollout governance” concerns a separate audit weakness, not the missing assurance described by supporting-component changes were excluded from end-to-end behavioral testing. The correct cause is “Supporting-component changes were excluded from end-to-end behavioral testing”.
Question 253
Managers manually bypass model alerts during peak season without logging overrides. From an AI audit perspective, which corrective action is the MOST direct response to the weakness described, rather than a useful control for some different AI risk?
- Evaluate tail latency, throughput, fault behavior and recovery under new workload
- Record emergency overrides, time limits and retrospective approval
- Maintain supplier change notification and test significant version updates before production
- Treat preprocessing/schema modifications as model-affecting changes requiring retest
Explanation
Correct Answer
B:Correct. Record emergency overrides, time limits and retrospective approval is aimed at the actual failure: emergency change procedure lacks audit trails and postimplementation review. The audit should subsequently obtain exception log, authorization evidence and completed postchange review to test execution, not merely the existence of a policy.
Incorrect Answers
A:“Evaluate tail latency, throughput, fault behavior and recovery under new workload” is a control for architecture change did not trigger performance and availability reassessment, but the audit finding here is emergency change procedure lacks audit trails and postimplementation review. The relevant measure is “Record emergency overrides, time limits and retrospective approval”.
C:“Maintain supplier change notification and test significant version updates before production” is a control for vendor-driven model updates are not captured as material changes, but the audit finding here is emergency change procedure lacks audit trails and postimplementation review. The relevant measure is “Record emergency overrides, time limits and retrospective approval”.
D:“Treat preprocessing/schema modifications as model-affecting changes requiring retest” is a control for change impact assessment excludes feature engineering dependencies, but the audit finding here is emergency change procedure lacks audit trails and postimplementation review. The relevant measure is “Record emergency overrides, time limits and retrospective approval”.
Question 254
In an audit of Change Management Specific to AI, managers say this measure is live: Record emergency overrides, time limits and retrospective approval. They claim it corrected a material audit gap. What direct operating evidence would allow the auditor to substantiate that particular claim across a defensible sample?
- Exception log, authorization evidence and completed postchange review
- Load test profiles, SLO approval and failure-injection observations
- Supplier release notices linked to impact assessments and regression approvals
- Feature contract diff, replayed cases and approved impact analysis
Explanation
Correct Answer
A:Correct. The proposed records—exception log, authorization evidence and completed postchange review—are directly tied to the asserted safeguard and allow its actual operation to be corroborated. The gap being tested is emergency change procedure lacks audit trails and postimplementation review.
Incorrect Answers
B:“Load test profiles, SLO approval and failure-injection observations” would test architecture change did not trigger performance and availability reassessment, not the present safeguard against emergency change procedure lacks audit trails and postimplementation review. The expected audit evidence is “Exception log, authorization evidence and completed postchange review”.
C:“Supplier release notices linked to impact assessments and regression approvals” would test vendor-driven model updates are not captured as material changes, not the present safeguard against emergency change procedure lacks audit trails and postimplementation review. The expected audit evidence is “Exception log, authorization evidence and completed postchange review”.
D:“Feature contract diff, replayed cases and approved impact analysis” would test change impact assessment excludes feature engineering dependencies, not the present safeguard against emergency change procedure lacks audit trails and postimplementation review. The expected audit evidence is “Exception log, authorization evidence and completed postchange review”.
Question 255
In a review of Change Management Specific to AI, investigators cannot establish the expected control proof: exception log, authorization evidence and completed postchange review. The underlying event was that managers manually bypass model alerts during peak season without logging overrides. Which control weakness is the MOST likely cause requiring focused audit work?
- Architecture change did not trigger performance and availability reassessment
- Vendor-driven model updates are not captured as material changes
- Change impact assessment excludes feature engineering dependencies
- Emergency change procedure lacks audit trails and postimplementation review
Explanation
Correct Answer
D:Correct. The observed condition specifically indicates emergency change procedure lacks audit trails and postimplementation review. A meaningful audit response is to verify this diagnosis and then recommend record emergency overrides, time limits and retrospective approval.
Incorrect Answers
A:“Architecture change did not trigger performance and availability reassessment” concerns a separate audit weakness, not the missing assurance described by emergency change procedure lacks audit trails and postimplementation review. The correct cause is “Emergency change procedure lacks audit trails and postimplementation review”.
B:“Vendor-driven model updates are not captured as material changes” concerns a separate audit weakness, not the missing assurance described by emergency change procedure lacks audit trails and postimplementation review. The correct cause is “Emergency change procedure lacks audit trails and postimplementation review”.
C:“Change impact assessment excludes feature engineering dependencies” concerns a separate audit weakness, not the missing assurance described by emergency change procedure lacks audit trails and postimplementation review. The correct cause is “Emergency change procedure lacks audit trails and postimplementation review”.
Question 256
An updated third-party model changes content filter categories but downstream business workflows are unmodified. From an AI audit perspective, which corrective action is the MOST direct response to the weakness described, rather than a useful control for some different AI risk?
- Pin compatible preprocessing artifacts and enforce training-serving parity tests
- Gate promotion on disaggregated thresholds and risk-owner review
- Assess output-policy changes with workflow owners and revalidate impacted processes
- Require retrieval and answer-quality regression after dependent infrastructure changes
Explanation
Correct Answer
C:Correct. Assess output-policy changes with workflow owners and revalidate impacted processes is aimed at the actual failure: vendor policy revisions bypass business-impact communication and acceptance. The audit should subsequently obtain vendor policy diff, business test results and signed readiness decision to test execution, not merely the existence of a policy.
Incorrect Answers
A:“Pin compatible preprocessing artifacts and enforce training-serving parity tests” is a control for configuration drift between training and production was not detected, but the audit finding here is vendor policy revisions bypass business-impact communication and acceptance. The relevant measure is “Assess output-policy changes with workflow owners and revalidate impacted processes”.
B:“Gate promotion on disaggregated thresholds and risk-owner review” is a control for aggregate improvement is masking high-impact regression during model change, but the audit finding here is vendor policy revisions bypass business-impact communication and acceptance. The relevant measure is “Assess output-policy changes with workflow owners and revalidate impacted processes”.
D:“Require retrieval and answer-quality regression after dependent infrastructure changes” is a control for supporting-component changes were excluded from end-to-end behavioral testing, but the audit finding here is vendor policy revisions bypass business-impact communication and acceptance. The relevant measure is “Assess output-policy changes with workflow owners and revalidate impacted processes”.
Question 257
In an audit of Change Management Specific to AI, managers say this measure is live: Assess output-policy changes with workflow owners and revalidate impacted processes. They claim it corrected a material audit gap. What direct operating evidence would allow the auditor to substantiate that particular claim across a defensible sample?
- Registry hash comparison and parity gate in the release pipeline
- Vendor policy diff, business test results and signed readiness decision
- Candidate/champion cohort comparison and blocked release record
- Before/after retrieval rankings, grounding metrics and release tickets
Explanation
Correct Answer
B:Correct. The proposed records—vendor policy diff, business test results and signed readiness decision—are directly tied to the asserted safeguard and allow its actual operation to be corroborated. The gap being tested is vendor policy revisions bypass business-impact communication and acceptance.
Incorrect Answers
A:“Registry hash comparison and parity gate in the release pipeline” would test configuration drift between training and production was not detected, not the present safeguard against vendor policy revisions bypass business-impact communication and acceptance. The expected audit evidence is “Vendor policy diff, business test results and signed readiness decision”.
C:“Candidate/champion cohort comparison and blocked release record” would test aggregate improvement is masking high-impact regression during model change, not the present safeguard against vendor policy revisions bypass business-impact communication and acceptance. The expected audit evidence is “Vendor policy diff, business test results and signed readiness decision”.
D:“Before/after retrieval rankings, grounding metrics and release tickets” would test supporting-component changes were excluded from end-to-end behavioral testing, not the present safeguard against vendor policy revisions bypass business-impact communication and acceptance. The expected audit evidence is “Vendor policy diff, business test results and signed readiness decision”.
Question 258
In a review of Change Management Specific to AI, investigators cannot establish the expected control proof: vendor policy diff, business test results and signed readiness decision. The underlying event was that an updated third-party model changes content filter categories but downstream business workflows are unmodified. Which control weakness is the MOST likely cause requiring focused audit work?
- Vendor policy revisions bypass business-impact communication and acceptance
- Configuration drift between training and production was not detected
- Aggregate improvement is masking high-impact regression during model change
- Supporting-component changes were excluded from end-to-end behavioral testing
Explanation
Correct Answer
A:Correct. The observed condition specifically indicates vendor policy revisions bypass business-impact communication and acceptance. A meaningful audit response is to verify this diagnosis and then recommend assess output-policy changes with workflow owners and revalidate impacted processes.
Incorrect Answers
B:“Configuration drift between training and production was not detected” concerns a separate audit weakness, not the missing assurance described by vendor policy revisions bypass business-impact communication and acceptance. The correct cause is “Vendor policy revisions bypass business-impact communication and acceptance”.
C:“Aggregate improvement is masking high-impact regression during model change” concerns a separate audit weakness, not the missing assurance described by vendor policy revisions bypass business-impact communication and acceptance. The correct cause is “Vendor policy revisions bypass business-impact communication and acceptance”.
D:“Supporting-component changes were excluded from end-to-end behavioral testing” concerns a separate audit weakness, not the missing assurance described by vendor policy revisions bypass business-impact communication and acceptance. The correct cause is “Vendor policy revisions bypass business-impact communication and acceptance”.
Question 259
A deployment shifts from batch inference to real-time scoring without recalculating capacity risks. From an AI audit perspective, which corrective action is the MOST direct response to the weakness described, rather than a useful control for some different AI risk?
- Version and roll back model, data contract, prompt and dependent services together
- Version prompts as controlled artifacts with peer review and rollback
- Record emergency overrides, time limits and retrospective approval
- Evaluate tail latency, throughput, fault behavior and recovery under new workload
Explanation
Correct Answer
D:Correct. Evaluate tail latency, throughput, fault behavior and recovery under new workload is aimed at the actual failure: architecture change did not trigger performance and availability reassessment. The audit should subsequently obtain load test profiles, slo approval and failure-injection observations to test execution, not merely the existence of a policy.
Incorrect Answers
A:“Version and roll back model, data contract, prompt and dependent services together” is a control for recovery plan is incomplete across the full inference dependency bundle, but the audit finding here is architecture change did not trigger performance and availability reassessment. The relevant measure is “Evaluate tail latency, throughput, fault behavior and recovery under new workload”.
B:“Version prompts as controlled artifacts with peer review and rollback” is a control for prompt configuration changes bypass change authorization and traceability, but the audit finding here is architecture change did not trigger performance and availability reassessment. The relevant measure is “Evaluate tail latency, throughput, fault behavior and recovery under new workload”.
C:“Record emergency overrides, time limits and retrospective approval” is a control for emergency change procedure lacks audit trails and postimplementation review, but the audit finding here is architecture change did not trigger performance and availability reassessment. The relevant measure is “Evaluate tail latency, throughput, fault behavior and recovery under new workload”.
Question 260
In an audit of Change Management Specific to AI, managers say this measure is live: Evaluate tail latency, throughput, fault behavior and recovery under new workload. They claim it corrected a material audit gap. What direct operating evidence would allow the auditor to substantiate that particular claim across a defensible sample?
- End-to-end rollback rehearsal with matching artifact manifest
- Signed prompt diffs, reviewer approval and deployment traces
- Load test profiles, SLO approval and failure-injection observations
- Exception log, authorization evidence and completed postchange review
Explanation
Correct Answer
C:Correct. The proposed records—load test profiles, slo approval and failure-injection observations—are directly tied to the asserted safeguard and allow its actual operation to be corroborated. The gap being tested is architecture change did not trigger performance and availability reassessment.
Incorrect Answers
A:“End-to-end rollback rehearsal with matching artifact manifest” would test recovery plan is incomplete across the full inference dependency bundle, not the present safeguard against architecture change did not trigger performance and availability reassessment. The expected audit evidence is “Load test profiles, SLO approval and failure-injection observations”.
B:“Signed prompt diffs, reviewer approval and deployment traces” would test prompt configuration changes bypass change authorization and traceability, not the present safeguard against architecture change did not trigger performance and availability reassessment. The expected audit evidence is “Load test profiles, SLO approval and failure-injection observations”.
D:“Exception log, authorization evidence and completed postchange review” would test emergency change procedure lacks audit trails and postimplementation review, not the present safeguard against architecture change did not trigger performance and availability reassessment. The expected audit evidence is “Load test profiles, SLO approval and failure-injection observations”.