ISACA Advanced in AI Audit AAIA Practice Test 25
Question 240
In a review of Change Management Specific to AI, investigators cannot establish the expected control proof: candidate/champion cohort comparison and blocked release record. The underlying event was that a retrained classifier performs better globally but worse for a protected subgroup. Which control weakness is the MOST likely cause requiring focused audit work?
- Progressive delivery and stop criteria were not included in rollout governance
- Emergency change procedure lacks audit trails and postimplementation review
- Aggregate improvement is masking high-impact regression during model change
- Recovery plan is incomplete across the full inference dependency bundle
Explanation
Correct Answer
C:Correct. The observed condition specifically indicates aggregate improvement is masking high-impact regression during model change. A meaningful audit response is to verify this diagnosis and then recommend gate promotion on disaggregated thresholds and risk-owner review.
Incorrect Answers
A:“Progressive delivery and stop criteria were not included in rollout governance” concerns a separate audit weakness, not the missing assurance described by aggregate improvement is masking high-impact regression during model change. The correct cause is “Aggregate improvement is masking high-impact regression during model change”.
B:“Emergency change procedure lacks audit trails and postimplementation review” concerns a separate audit weakness, not the missing assurance described by aggregate improvement is masking high-impact regression during model change. The correct cause is “Aggregate improvement is masking high-impact regression during model change”.
D:“Recovery plan is incomplete across the full inference dependency bundle” concerns a separate audit weakness, not the missing assurance described by aggregate improvement is masking high-impact regression during model change. The correct cause is “Aggregate improvement is masking high-impact regression during model change”.
Question 241
Engineers update inference prompts in production without recording who changed system instructions. From an AI audit perspective, which corrective action is the MOST direct response to the weakness described, rather than a useful control for some different AI risk?
- Treat preprocessing/schema modifications as model-affecting changes requiring retest
- Version prompts as controlled artifacts with peer review and rollback
- Assess output-policy changes with workflow owners and revalidate impacted processes
- Maintain supplier change notification and test significant version updates before production
Explanation
Correct Answer
B:Correct. Version prompts as controlled artifacts with peer review and rollback is aimed at the actual failure: prompt configuration changes bypass change authorization and traceability. The audit should subsequently obtain signed prompt diffs, reviewer approval and deployment traces to test execution, not merely the existence of a policy.
Incorrect Answers
A:“Treat preprocessing/schema modifications as model-affecting changes requiring retest” is a control for change impact assessment excludes feature engineering dependencies, but the audit finding here is prompt configuration changes bypass change authorization and traceability. The relevant measure is “Version prompts as controlled artifacts with peer review and rollback”.
C:“Assess output-policy changes with workflow owners and revalidate impacted processes” is a control for vendor policy revisions bypass business-impact communication and acceptance, but the audit finding here is prompt configuration changes bypass change authorization and traceability. The relevant measure is “Version prompts as controlled artifacts with peer review and rollback”.
D:“Maintain supplier change notification and test significant version updates before production” is a control for vendor-driven model updates are not captured as material changes, but the audit finding here is prompt configuration changes bypass change authorization and traceability. The relevant measure is “Version prompts as controlled artifacts with peer review and rollback”.
Question 242
In an audit of Change Management Specific to AI, managers say this measure is live: Version prompts as controlled artifacts with peer review and rollback. They claim it corrected a material audit gap. What direct operating evidence would allow the auditor to substantiate that particular claim across a defensible sample?
- Signed prompt diffs, reviewer approval and deployment traces
- Feature contract diff, replayed cases and approved impact analysis
- Vendor policy diff, business test results and signed readiness decision
- Supplier release notices linked to impact assessments and regression approvals
Explanation
Correct Answer
A:Correct. The proposed records—signed prompt diffs, reviewer approval and deployment traces—are directly tied to the asserted safeguard and allow its actual operation to be corroborated. The gap being tested is prompt configuration changes bypass change authorization and traceability.
Incorrect Answers
B:“Feature contract diff, replayed cases and approved impact analysis” would test change impact assessment excludes feature engineering dependencies, not the present safeguard against prompt configuration changes bypass change authorization and traceability. The expected audit evidence is “Signed prompt diffs, reviewer approval and deployment traces”.
C:“Vendor policy diff, business test results and signed readiness decision” would test vendor policy revisions bypass business-impact communication and acceptance, not the present safeguard against prompt configuration changes bypass change authorization and traceability. The expected audit evidence is “Signed prompt diffs, reviewer approval and deployment traces”.
D:“Supplier release notices linked to impact assessments and regression approvals” would test vendor-driven model updates are not captured as material changes, not the present safeguard against prompt configuration changes bypass change authorization and traceability. The expected audit evidence is “Signed prompt diffs, reviewer approval and deployment traces”.
Question 243
In a review of Change Management Specific to AI, investigators cannot establish the expected control proof: signed prompt diffs, reviewer approval and deployment traces. The underlying event was that engineers update inference prompts in production without recording who changed system instructions. Which control weakness is the MOST likely cause requiring focused audit work?
- Change impact assessment excludes feature engineering dependencies
- Vendor policy revisions bypass business-impact communication and acceptance
- Vendor-driven model updates are not captured as material changes
- Prompt configuration changes bypass change authorization and traceability
Explanation
Correct Answer
D:Correct. The observed condition specifically indicates prompt configuration changes bypass change authorization and traceability. A meaningful audit response is to verify this diagnosis and then recommend version prompts as controlled artifacts with peer review and rollback.
Incorrect Answers
A:“Change impact assessment excludes feature engineering dependencies” concerns a separate audit weakness, not the missing assurance described by prompt configuration changes bypass change authorization and traceability. The correct cause is “Prompt configuration changes bypass change authorization and traceability”.
B:“Vendor policy revisions bypass business-impact communication and acceptance” concerns a separate audit weakness, not the missing assurance described by prompt configuration changes bypass change authorization and traceability. The correct cause is “Prompt configuration changes bypass change authorization and traceability”.
C:“Vendor-driven model updates are not captured as material changes” concerns a separate audit weakness, not the missing assurance described by prompt configuration changes bypass change authorization and traceability. The correct cause is “Prompt configuration changes bypass change authorization and traceability”.
Question 244
An AI feature is rolled out to all customers at once despite uncertain effects on complaint volume. From an AI audit perspective, which corrective action is the MOST direct response to the weakness described, rather than a useful control for some different AI risk?
- Require retrieval and answer-quality regression after dependent infrastructure changes
- Evaluate tail latency, throughput, fault behavior and recovery under new workload
- Use controlled canaries with guardrail metrics and automatic pause thresholds
- Gate promotion on disaggregated thresholds and risk-owner review
Explanation
Correct Answer
C:Correct. Use controlled canaries with guardrail metrics and automatic pause thresholds is aimed at the actual failure: progressive delivery and stop criteria were not included in rollout governance. The audit should subsequently obtain canary dashboard, pause trigger history and deployment decision log to test execution, not merely the existence of a policy.
Incorrect Answers
A:“Require retrieval and answer-quality regression after dependent infrastructure changes” is a control for supporting-component changes were excluded from end-to-end behavioral testing, but the audit finding here is progressive delivery and stop criteria were not included in rollout governance. The relevant measure is “Use controlled canaries with guardrail metrics and automatic pause thresholds”.
B:“Evaluate tail latency, throughput, fault behavior and recovery under new workload” is a control for architecture change did not trigger performance and availability reassessment, but the audit finding here is progressive delivery and stop criteria were not included in rollout governance. The relevant measure is “Use controlled canaries with guardrail metrics and automatic pause thresholds”.
D:“Gate promotion on disaggregated thresholds and risk-owner review” is a control for aggregate improvement is masking high-impact regression during model change, but the audit finding here is progressive delivery and stop criteria were not included in rollout governance. The relevant measure is “Use controlled canaries with guardrail metrics and automatic pause thresholds”.
Question 245
In an audit of Change Management Specific to AI, managers say this measure is live: Use controlled canaries with guardrail metrics and automatic pause thresholds. They claim it corrected a material audit gap. What direct operating evidence would allow the auditor to substantiate that particular claim across a defensible sample?
- Before/after retrieval rankings, grounding metrics and release tickets
- Canary dashboard, pause trigger history and deployment decision log
- Load test profiles, SLO approval and failure-injection observations
- Candidate/champion cohort comparison and blocked release record
Explanation
Correct Answer
B:Correct. The proposed records—canary dashboard, pause trigger history and deployment decision log—are directly tied to the asserted safeguard and allow its actual operation to be corroborated. The gap being tested is progressive delivery and stop criteria were not included in rollout governance.
Incorrect Answers
A:“Before/after retrieval rankings, grounding metrics and release tickets” would test supporting-component changes were excluded from end-to-end behavioral testing, not the present safeguard against progressive delivery and stop criteria were not included in rollout governance. The expected audit evidence is “Canary dashboard, pause trigger history and deployment decision log”.
C:“Load test profiles, SLO approval and failure-injection observations” would test architecture change did not trigger performance and availability reassessment, not the present safeguard against progressive delivery and stop criteria were not included in rollout governance. The expected audit evidence is “Canary dashboard, pause trigger history and deployment decision log”.
D:“Candidate/champion cohort comparison and blocked release record” would test aggregate improvement is masking high-impact regression during model change, not the present safeguard against progressive delivery and stop criteria were not included in rollout governance. The expected audit evidence is “Canary dashboard, pause trigger history and deployment decision log”.
Question 246
In a review of Change Management Specific to AI, investigators cannot establish the expected control proof: canary dashboard, pause trigger history and deployment decision log. The underlying event was that an AI feature is rolled out to all customers at once despite uncertain effects on complaint volume. Which control weakness is the MOST likely cause requiring focused audit work?
- Progressive delivery and stop criteria were not included in rollout governance
- Supporting-component changes were excluded from end-to-end behavioral testing
- Architecture change did not trigger performance and availability reassessment
- Aggregate improvement is masking high-impact regression during model change
Explanation
Correct Answer
A:Correct. The observed condition specifically indicates progressive delivery and stop criteria were not included in rollout governance. A meaningful audit response is to verify this diagnosis and then recommend use controlled canaries with guardrail metrics and automatic pause thresholds.
Incorrect Answers
B:“Supporting-component changes were excluded from end-to-end behavioral testing” concerns a separate audit weakness, not the missing assurance described by progressive delivery and stop criteria were not included in rollout governance. The correct cause is “Progressive delivery and stop criteria were not included in rollout governance”.
C:“Architecture change did not trigger performance and availability reassessment” concerns a separate audit weakness, not the missing assurance described by progressive delivery and stop criteria were not included in rollout governance. The correct cause is “Progressive delivery and stop criteria were not included in rollout governance”.
D:“Aggregate improvement is masking high-impact regression during model change” concerns a separate audit weakness, not the missing assurance described by progressive delivery and stop criteria were not included in rollout governance. The correct cause is “Progressive delivery and stop criteria were not included in rollout governance”.
Question 247
A scoring service changes input units while its model version stays identical. From an AI audit perspective, which corrective action is the MOST direct response to the weakness described, rather than a useful control for some different AI risk?
- Record emergency overrides, time limits and retrospective approval
- Pin compatible preprocessing artifacts and enforce training-serving parity tests
- Version prompts as controlled artifacts with peer review and rollback
- Treat preprocessing/schema modifications as model-affecting changes requiring retest
Explanation
Correct Answer
D:Correct. Treat preprocessing/schema modifications as model-affecting changes requiring retest is aimed at the actual failure: change impact assessment excludes feature engineering dependencies. The audit should subsequently obtain feature contract diff, replayed cases and approved impact analysis to test execution, not merely the existence of a policy.
Incorrect Answers
A:“Record emergency overrides, time limits and retrospective approval” is a control for emergency change procedure lacks audit trails and postimplementation review, but the audit finding here is change impact assessment excludes feature engineering dependencies. The relevant measure is “Treat preprocessing/schema modifications as model-affecting changes requiring retest”.
B:“Pin compatible preprocessing artifacts and enforce training-serving parity tests” is a control for configuration drift between training and production was not detected, but the audit finding here is change impact assessment excludes feature engineering dependencies. The relevant measure is “Treat preprocessing/schema modifications as model-affecting changes requiring retest”.
C:“Version prompts as controlled artifacts with peer review and rollback” is a control for prompt configuration changes bypass change authorization and traceability, but the audit finding here is change impact assessment excludes feature engineering dependencies. The relevant measure is “Treat preprocessing/schema modifications as model-affecting changes requiring retest”.
Question 248
In an audit of Change Management Specific to AI, managers say this measure is live: Treat preprocessing/schema modifications as model-affecting changes requiring retest. They claim it corrected a material audit gap. What direct operating evidence would allow the auditor to substantiate that particular claim across a defensible sample?
- Exception log, authorization evidence and completed postchange review
- Registry hash comparison and parity gate in the release pipeline
- Feature contract diff, replayed cases and approved impact analysis
- Signed prompt diffs, reviewer approval and deployment traces
Explanation
Correct Answer
C:Correct. The proposed records—feature contract diff, replayed cases and approved impact analysis—are directly tied to the asserted safeguard and allow its actual operation to be corroborated. The gap being tested is change impact assessment excludes feature engineering dependencies.
Incorrect Answers
A:“Exception log, authorization evidence and completed postchange review” would test emergency change procedure lacks audit trails and postimplementation review, not the present safeguard against change impact assessment excludes feature engineering dependencies. The expected audit evidence is “Feature contract diff, replayed cases and approved impact analysis”.
B:“Registry hash comparison and parity gate in the release pipeline” would test configuration drift between training and production was not detected, not the present safeguard against change impact assessment excludes feature engineering dependencies. The expected audit evidence is “Feature contract diff, replayed cases and approved impact analysis”.
D:“Signed prompt diffs, reviewer approval and deployment traces” would test prompt configuration changes bypass change authorization and traceability, not the present safeguard against change impact assessment excludes feature engineering dependencies. The expected audit evidence is “Feature contract diff, replayed cases and approved impact analysis”.
Question 249
In a review of Change Management Specific to AI, investigators cannot establish the expected control proof: feature contract diff, replayed cases and approved impact analysis. The underlying event was that a scoring service changes input units while its model version stays identical. Which control weakness is the MOST likely cause requiring focused audit work?
- Emergency change procedure lacks audit trails and postimplementation review
- Change impact assessment excludes feature engineering dependencies
- Configuration drift between training and production was not detected
- Prompt configuration changes bypass change authorization and traceability
Explanation
Correct Answer
B:Correct. The observed condition specifically indicates change impact assessment excludes feature engineering dependencies. A meaningful audit response is to verify this diagnosis and then recommend treat preprocessing/schema modifications as model-affecting changes requiring retest.
Incorrect Answers
A:“Emergency change procedure lacks audit trails and postimplementation review” concerns a separate audit weakness, not the missing assurance described by change impact assessment excludes feature engineering dependencies. The correct cause is “Change impact assessment excludes feature engineering dependencies”.
C:“Configuration drift between training and production was not detected” concerns a separate audit weakness, not the missing assurance described by change impact assessment excludes feature engineering dependencies. The correct cause is “Change impact assessment excludes feature engineering dependencies”.
D:“Prompt configuration changes bypass change authorization and traceability” concerns a separate audit weakness, not the missing assurance described by change impact assessment excludes feature engineering dependencies. The correct cause is “Change impact assessment excludes feature engineering dependencies”.
Question 250
A security patch alters a vector database's retrieval ranking but the AI team considers it unrelated infrastructure. From an AI audit perspective, which corrective action is the MOST direct response to the weakness described, rather than a useful control for some different AI risk?
- Require retrieval and answer-quality regression after dependent infrastructure changes
- Assess output-policy changes with workflow owners and revalidate impacted processes
- Version and roll back model, data contract, prompt and dependent services together
- Use controlled canaries with guardrail metrics and automatic pause thresholds
Explanation
Correct Answer
A:Correct. Require retrieval and answer-quality regression after dependent infrastructure changes is aimed at the actual failure: supporting-component changes were excluded from end-to-end behavioral testing. The audit should subsequently obtain before/after retrieval rankings, grounding metrics and release tickets to test execution, not merely the existence of a policy.
Incorrect Answers
B:“Assess output-policy changes with workflow owners and revalidate impacted processes” is a control for vendor policy revisions bypass business-impact communication and acceptance, but the audit finding here is supporting-component changes were excluded from end-to-end behavioral testing. The relevant measure is “Require retrieval and answer-quality regression after dependent infrastructure changes”.
C:“Version and roll back model, data contract, prompt and dependent services together” is a control for recovery plan is incomplete across the full inference dependency bundle, but the audit finding here is supporting-component changes were excluded from end-to-end behavioral testing. The relevant measure is “Require retrieval and answer-quality regression after dependent infrastructure changes”.
D:“Use controlled canaries with guardrail metrics and automatic pause thresholds” is a control for progressive delivery and stop criteria were not included in rollout governance, but the audit finding here is supporting-component changes were excluded from end-to-end behavioral testing. The relevant measure is “Require retrieval and answer-quality regression after dependent infrastructure changes”.