ISACA Advanced in AI Audit AAIA Practice Test 24
Question 230
In an audit of AI Solution Development Methodologies and Lifecycle, managers say this measure is live: Rotate service credentials and transfer documented operations ownership at contract exit. They claim it corrected a material audit gap. What direct operating evidence would allow the auditor to substantiate that particular claim across a defensible sample?
- Offboarding checklist, credential revocation evidence and runbook acceptance
- Promotion approvals, immutable build hashes and traced artifact lineage
- Dependency lock, vendor change alert and executed regression suite
- Negative-path test library, severity triage and corrected regression evidence
Explanation
Correct Answer
A:Correct. The proposed records—offboarding checklist, credential revocation evidence and runbook acceptance—are directly tied to the asserted safeguard and allow its actual operation to be corroborated. The gap being tested is knowledge transfer and ownership transition are absent from lifecycle controls.
Incorrect Answers
B:“Promotion approvals, immutable build hashes and traced artifact lineage” would test build and release provenance is bypassed by ad hoc notebook deployment, not the present safeguard against knowledge transfer and ownership transition are absent from lifecycle controls. The expected audit evidence is “Offboarding checklist, credential revocation evidence and runbook acceptance”.
C:“Dependency lock, vendor change alert and executed regression suite” would test upstream dependency versions are not pinned or monitored, not the present safeguard against knowledge transfer and ownership transition are absent from lifecycle controls. The expected audit evidence is “Offboarding checklist, credential revocation evidence and runbook acceptance”.
D:“Negative-path test library, severity triage and corrected regression evidence” would test lifecycle testing omitted realistic failures and adversarial interaction, not the present safeguard against knowledge transfer and ownership transition are absent from lifecycle controls. The expected audit evidence is “Offboarding checklist, credential revocation evidence and runbook acceptance”.
Question 231
In a review of AI Solution Development Methodologies and Lifecycle, investigators cannot establish the expected control proof: offboarding checklist, credential revocation evidence and runbook acceptance. The underlying event was that a data science vendor leaves and all model-serving credentials remain embedded in shared scripts. Which control weakness is the MOST likely cause requiring focused audit work?
- Build and release provenance is bypassed by ad hoc notebook deployment
- Upstream dependency versions are not pinned or monitored
- Lifecycle testing omitted realistic failures and adversarial interaction
- Knowledge transfer and ownership transition are absent from lifecycle controls
Explanation
Correct Answer
D:Correct. The observed condition specifically indicates knowledge transfer and ownership transition are absent from lifecycle controls. A meaningful audit response is to verify this diagnosis and then recommend rotate service credentials and transfer documented operations ownership at contract exit.
Incorrect Answers
A:“Build and release provenance is bypassed by ad hoc notebook deployment” concerns a separate audit weakness, not the missing assurance described by knowledge transfer and ownership transition are absent from lifecycle controls. The correct cause is “Knowledge transfer and ownership transition are absent from lifecycle controls”.
B:“Upstream dependency versions are not pinned or monitored” concerns a separate audit weakness, not the missing assurance described by knowledge transfer and ownership transition are absent from lifecycle controls. The correct cause is “Knowledge transfer and ownership transition are absent from lifecycle controls”.
C:“Lifecycle testing omitted realistic failures and adversarial interaction” concerns a separate audit weakness, not the missing assurance described by knowledge transfer and ownership transition are absent from lifecycle controls. The correct cause is “Knowledge transfer and ownership transition are absent from lifecycle controls”.
Question 232
An organization deploys an LLM agent with tool calls, but architecture documentation omits external actions. From an AI audit perspective, which corrective action is the MOST direct response to the weakness described, rather than a useful control for some different AI risk?
- Define monitoring thresholds, decommission owners and safe replacement routes
- Require staged deployment, operational go/no-go criteria and tested rollback
- Model full prompt-to-tool flows, authorization gates and human-stop mechanisms
- Separate training from deployment and require validation plus appropriate approval
Explanation
Correct Answer
C:Correct. Model full prompt-to-tool flows, authorization gates and human-stop mechanisms is aimed at the actual failure: agentic system interfaces and permissions are outside the documented solution boundary. The audit should subsequently obtain reviewed sequence diagrams, negative tool-call tests and approved authority map to test execution, not merely the existence of a policy.
Incorrect Answers
A:“Define monitoring thresholds, decommission owners and safe replacement routes” is a control for lifecycle documentation lacks explicit retirement and sunset criteria, but the audit finding here is agentic system interfaces and permissions are outside the documented solution boundary. The relevant measure is “Model full prompt-to-tool flows, authorization gates and human-stop mechanisms”.
B:“Require staged deployment, operational go/no-go criteria and tested rollback” is a control for operational acceptance gates and fail-safe recovery are omitted from release design, but the audit finding here is agentic system interfaces and permissions are outside the documented solution boundary. The relevant measure is “Model full prompt-to-tool flows, authorization gates and human-stop mechanisms”.
D:“Separate training from deployment and require validation plus appropriate approval” is a control for autonomous retraining lacks risk-tiered reauthorization of changed behavior, but the audit finding here is agentic system interfaces and permissions are outside the documented solution boundary. The relevant measure is “Model full prompt-to-tool flows, authorization gates and human-stop mechanisms”.
Question 233
In an audit of AI Solution Development Methodologies and Lifecycle, managers say this measure is live: Model full prompt-to-tool flows, authorization gates and human-stop mechanisms. They claim it corrected a material audit gap. What direct operating evidence would allow the auditor to substantiate that particular claim across a defensible sample?
- Approved retirement plan and a tested rollback/decommission exercise
- Reviewed sequence diagrams, negative tool-call tests and approved authority map
- Canary decision log, failed deployment drill and rollback telemetry
- Retrain job trace, candidate-vs-champion tests and release approval
Explanation
Correct Answer
B:Correct. The proposed records—reviewed sequence diagrams, negative tool-call tests and approved authority map—are directly tied to the asserted safeguard and allow its actual operation to be corroborated. The gap being tested is agentic system interfaces and permissions are outside the documented solution boundary.
Incorrect Answers
A:“Approved retirement plan and a tested rollback/decommission exercise” would test lifecycle documentation lacks explicit retirement and sunset criteria, not the present safeguard against agentic system interfaces and permissions are outside the documented solution boundary. The expected audit evidence is “Reviewed sequence diagrams, negative tool-call tests and approved authority map”.
C:“Canary decision log, failed deployment drill and rollback telemetry” would test operational acceptance gates and fail-safe recovery are omitted from release design, not the present safeguard against agentic system interfaces and permissions are outside the documented solution boundary. The expected audit evidence is “Reviewed sequence diagrams, negative tool-call tests and approved authority map”.
D:“Retrain job trace, candidate-vs-champion tests and release approval” would test autonomous retraining lacks risk-tiered reauthorization of changed behavior, not the present safeguard against agentic system interfaces and permissions are outside the documented solution boundary. The expected audit evidence is “Reviewed sequence diagrams, negative tool-call tests and approved authority map”.
Question 234
In a review of AI Solution Development Methodologies and Lifecycle, investigators cannot establish the expected control proof: reviewed sequence diagrams, negative tool-call tests and approved authority map. The underlying event was that an organization deploys an LLM agent with tool calls, but architecture documentation omits external actions. Which control weakness is the MOST likely cause requiring focused audit work?
- Agentic system interfaces and permissions are outside the documented solution boundary
- Lifecycle documentation lacks explicit retirement and sunset criteria
- Operational acceptance gates and fail-safe recovery are omitted from release design
- Autonomous retraining lacks risk-tiered reauthorization of changed behavior
Explanation
Correct Answer
A:Correct. The observed condition specifically indicates agentic system interfaces and permissions are outside the documented solution boundary. A meaningful audit response is to verify this diagnosis and then recommend model full prompt-to-tool flows, authorization gates and human-stop mechanisms.
Incorrect Answers
B:“Lifecycle documentation lacks explicit retirement and sunset criteria” concerns a separate audit weakness, not the missing assurance described by agentic system interfaces and permissions are outside the documented solution boundary. The correct cause is “Agentic system interfaces and permissions are outside the documented solution boundary”.
C:“Operational acceptance gates and fail-safe recovery are omitted from release design” concerns a separate audit weakness, not the missing assurance described by agentic system interfaces and permissions are outside the documented solution boundary. The correct cause is “Agentic system interfaces and permissions are outside the documented solution boundary”.
D:“Autonomous retraining lacks risk-tiered reauthorization of changed behavior” concerns a separate audit weakness, not the missing assurance described by agentic system interfaces and permissions are outside the documented solution boundary. The correct cause is “Agentic system interfaces and permissions are outside the documented solution boundary”.
Question 235
A model's approval assumes fixed behavior even though a provider periodically replaces its underlying model. From an AI audit perspective, which corrective action is the MOST direct response to the weakness described, rather than a useful control for some different AI risk?
- Version prompts as controlled artifacts with peer review and rollback
- Require retrieval and answer-quality regression after dependent infrastructure changes
- Pin compatible preprocessing artifacts and enforce training-serving parity tests
- Maintain supplier change notification and test significant version updates before production
Explanation
Correct Answer
D:Correct. Maintain supplier change notification and test significant version updates before production is aimed at the actual failure: vendor-driven model updates are not captured as material changes. The audit should subsequently obtain supplier release notices linked to impact assessments and regression approvals to test execution, not merely the existence of a policy.
Incorrect Answers
A:“Version prompts as controlled artifacts with peer review and rollback” is a control for prompt configuration changes bypass change authorization and traceability, but the audit finding here is vendor-driven model updates are not captured as material changes. The relevant measure is “Maintain supplier change notification and test significant version updates before production”.
B:“Require retrieval and answer-quality regression after dependent infrastructure changes” is a control for supporting-component changes were excluded from end-to-end behavioral testing, but the audit finding here is vendor-driven model updates are not captured as material changes. The relevant measure is “Maintain supplier change notification and test significant version updates before production”.
C:“Pin compatible preprocessing artifacts and enforce training-serving parity tests” is a control for configuration drift between training and production was not detected, but the audit finding here is vendor-driven model updates are not captured as material changes. The relevant measure is “Maintain supplier change notification and test significant version updates before production”.
Question 236
In an audit of Change Management Specific to AI, managers say this measure is live: Maintain supplier change notification and test significant version updates before production. They claim it corrected a material audit gap. What direct operating evidence would allow the auditor to substantiate that particular claim across a defensible sample?
- Signed prompt diffs, reviewer approval and deployment traces
- Before/after retrieval rankings, grounding metrics and release tickets
- Supplier release notices linked to impact assessments and regression approvals
- Registry hash comparison and parity gate in the release pipeline
Explanation
Correct Answer
C:Correct. The proposed records—supplier release notices linked to impact assessments and regression approvals—are directly tied to the asserted safeguard and allow its actual operation to be corroborated. The gap being tested is vendor-driven model updates are not captured as material changes.
Incorrect Answers
A:“Signed prompt diffs, reviewer approval and deployment traces” would test prompt configuration changes bypass change authorization and traceability, not the present safeguard against vendor-driven model updates are not captured as material changes. The expected audit evidence is “Supplier release notices linked to impact assessments and regression approvals”.
B:“Before/after retrieval rankings, grounding metrics and release tickets” would test supporting-component changes were excluded from end-to-end behavioral testing, not the present safeguard against vendor-driven model updates are not captured as material changes. The expected audit evidence is “Supplier release notices linked to impact assessments and regression approvals”.
D:“Registry hash comparison and parity gate in the release pipeline” would test configuration drift between training and production was not detected, not the present safeguard against vendor-driven model updates are not captured as material changes. The expected audit evidence is “Supplier release notices linked to impact assessments and regression approvals”.
Question 237
In a review of Change Management Specific to AI, investigators cannot establish the expected control proof: supplier release notices linked to impact assessments and regression approvals. The underlying event was that a model's approval assumes fixed behavior even though a provider periodically replaces its underlying model. Which control weakness is the MOST likely cause requiring focused audit work?
- Prompt configuration changes bypass change authorization and traceability
- Vendor-driven model updates are not captured as material changes
- Supporting-component changes were excluded from end-to-end behavioral testing
- Configuration drift between training and production was not detected
Explanation
Correct Answer
B:Correct. The observed condition specifically indicates vendor-driven model updates are not captured as material changes. A meaningful audit response is to verify this diagnosis and then recommend maintain supplier change notification and test significant version updates before production.
Incorrect Answers
A:“Prompt configuration changes bypass change authorization and traceability” concerns a separate audit weakness, not the missing assurance described by vendor-driven model updates are not captured as material changes. The correct cause is “Vendor-driven model updates are not captured as material changes”.
C:“Supporting-component changes were excluded from end-to-end behavioral testing” concerns a separate audit weakness, not the missing assurance described by vendor-driven model updates are not captured as material changes. The correct cause is “Vendor-driven model updates are not captured as material changes”.
D:“Configuration drift between training and production was not detected” concerns a separate audit weakness, not the missing assurance described by vendor-driven model updates are not captured as material changes. The correct cause is “Vendor-driven model updates are not captured as material changes”.
Question 238
A retrained classifier performs better globally but worse for a protected subgroup. From an AI audit perspective, which corrective action is the MOST direct response to the weakness described, rather than a useful control for some different AI risk?
- Gate promotion on disaggregated thresholds and risk-owner review
- Use controlled canaries with guardrail metrics and automatic pause thresholds
- Record emergency overrides, time limits and retrospective approval
- Version and roll back model, data contract, prompt and dependent services together
Explanation
Correct Answer
A:Correct. Gate promotion on disaggregated thresholds and risk-owner review is aimed at the actual failure: aggregate improvement is masking high-impact regression during model change. The audit should subsequently obtain candidate/champion cohort comparison and blocked release record to test execution, not merely the existence of a policy.
Incorrect Answers
B:“Use controlled canaries with guardrail metrics and automatic pause thresholds” is a control for progressive delivery and stop criteria were not included in rollout governance, but the audit finding here is aggregate improvement is masking high-impact regression during model change. The relevant measure is “Gate promotion on disaggregated thresholds and risk-owner review”.
C:“Record emergency overrides, time limits and retrospective approval” is a control for emergency change procedure lacks audit trails and postimplementation review, but the audit finding here is aggregate improvement is masking high-impact regression during model change. The relevant measure is “Gate promotion on disaggregated thresholds and risk-owner review”.
D:“Version and roll back model, data contract, prompt and dependent services together” is a control for recovery plan is incomplete across the full inference dependency bundle, but the audit finding here is aggregate improvement is masking high-impact regression during model change. The relevant measure is “Gate promotion on disaggregated thresholds and risk-owner review”.
Question 239
In an audit of Change Management Specific to AI, managers say this measure is live: Gate promotion on disaggregated thresholds and risk-owner review. They claim it corrected a material audit gap. What direct operating evidence would allow the auditor to substantiate that particular claim across a defensible sample?
- Canary dashboard, pause trigger history and deployment decision log
- Exception log, authorization evidence and completed postchange review
- End-to-end rollback rehearsal with matching artifact manifest
- Candidate/champion cohort comparison and blocked release record
Explanation
Correct Answer
D:Correct. The proposed records—candidate/champion cohort comparison and blocked release record—are directly tied to the asserted safeguard and allow its actual operation to be corroborated. The gap being tested is aggregate improvement is masking high-impact regression during model change.
Incorrect Answers
A:“Canary dashboard, pause trigger history and deployment decision log” would test progressive delivery and stop criteria were not included in rollout governance, not the present safeguard against aggregate improvement is masking high-impact regression during model change. The expected audit evidence is “Candidate/champion cohort comparison and blocked release record”.
B:“Exception log, authorization evidence and completed postchange review” would test emergency change procedure lacks audit trails and postimplementation review, not the present safeguard against aggregate improvement is masking high-impact regression during model change. The expected audit evidence is “Candidate/champion cohort comparison and blocked release record”.
C:“End-to-end rollback rehearsal with matching artifact manifest” would test recovery plan is incomplete across the full inference dependency bundle, not the present safeguard against aggregate improvement is masking high-impact regression during model change. The expected audit evidence is “Candidate/champion cohort comparison and blocked release record”.
Question 240
In a review of Change Management Specific to AI, investigators cannot establish the expected control proof: candidate/champion cohort comparison and blocked release record. The underlying event was that a retrained classifier performs better globally but worse for a protected subgroup. Which control weakness is the MOST likely cause requiring focused audit work?
- Progressive delivery and stop criteria were not included in rollout governance
- Emergency change procedure lacks audit trails and postimplementation review
- Aggregate improvement is masking high-impact regression during model change
- Recovery plan is incomplete across the full inference dependency bundle
Explanation
Correct Answer
C:Correct. The observed condition specifically indicates aggregate improvement is masking high-impact regression during model change. A meaningful audit response is to verify this diagnosis and then recommend gate promotion on disaggregated thresholds and risk-owner review.
Incorrect Answers
A:“Progressive delivery and stop criteria were not included in rollout governance” concerns a separate audit weakness, not the missing assurance described by aggregate improvement is masking high-impact regression during model change. The correct cause is “Aggregate improvement is masking high-impact regression during model change”.
B:“Emergency change procedure lacks audit trails and postimplementation review” concerns a separate audit weakness, not the missing assurance described by aggregate improvement is masking high-impact regression during model change. The correct cause is “Aggregate improvement is masking high-impact regression during model change”.
D:“Recovery plan is incomplete across the full inference dependency bundle” concerns a separate audit weakness, not the missing assurance described by aggregate improvement is masking high-impact regression during model change. The correct cause is “Aggregate improvement is masking high-impact regression during model change”.