ISACA Advanced in AI Audit AAIA Practice Test 34
Question 330
In a review of Testing Techniques for AI Solutions, investigators cannot establish the expected control proof: blocked tool-call traces and least-privilege test coverage report. The underlying event was that an AI agent's tool-use tests verify intended actions but never attempt unauthorized actions. Which control weakness is the MOST likely cause requiring focused audit work?
- Security negative testing omitted privilege escalation and action abuse
- Class imbalance makes accuracy alone a misleading test metric
- Robustness tests omitted realistic noisy and adversarial perturbations
- Resilience tests excluded degraded dependency states
Explanation
Correct Answer
A:Correct. The observed condition specifically indicates security negative testing omitted privilege escalation and action abuse. A meaningful audit response is to verify this diagnosis and then recommend execute denied tool-call, excessive-permission and approval-bypass test cases.
Incorrect Answers
B:“Class imbalance makes accuracy alone a misleading test metric” concerns a separate audit weakness, not the missing assurance described by security negative testing omitted privilege escalation and action abuse. The correct cause is “Security negative testing omitted privilege escalation and action abuse”.
C:“Robustness tests omitted realistic noisy and adversarial perturbations” concerns a separate audit weakness, not the missing assurance described by security negative testing omitted privilege escalation and action abuse. The correct cause is “Security negative testing omitted privilege escalation and action abuse”.
D:“Resilience tests excluded degraded dependency states” concerns a separate audit weakness, not the missing assurance described by security negative testing omitted privilege escalation and action abuse. The correct cause is “Security negative testing omitted privilege escalation and action abuse”.
Question 331
A financial model is validated with point predictions but not the stability of its uncertainty estimates. From an AI audit perspective, which corrective action is the MOST direct response to the weakness described, rather than a useful control for some different AI risk?
- Report uncertainty and obtain representative subgroup samples before fairness signoff
- Validate missing, out-of-range and conflicting input behavior under error paths
- Build field-condition holdouts and test illumination, occlusion and device shifts
- Evaluate calibration across score bands and changing populations
Explanation
Correct Answer
D:Correct. Evaluate calibration across score bands and changing populations is aimed at the actual failure: calibration and confidence reliability were excluded from acceptance testing. The audit should subsequently obtain reliability diagrams, brier scores and out-of-sample calibration tables to test execution, not merely the existence of a policy.
Incorrect Answers
A:“Report uncertainty and obtain representative subgroup samples before fairness signoff” is a control for subgroup sample size is inadequate to support equity conclusions, but the audit finding here is calibration and confidence reliability were excluded from acceptance testing. The relevant measure is “Evaluate calibration across score bands and changing populations”.
B:“Validate missing, out-of-range and conflicting input behavior under error paths” is a control for negative testing did not exercise invalid or incomplete feature contracts, but the audit finding here is calibration and confidence reliability were excluded from acceptance testing. The relevant measure is “Evaluate calibration across score bands and changing populations”.
C:“Build field-condition holdouts and test illumination, occlusion and device shifts” is a control for test dataset lacks representative camera and environmental conditions, but the audit finding here is calibration and confidence reliability were excluded from acceptance testing. The relevant measure is “Evaluate calibration across score bands and changing populations”.
Question 332
In an audit of Testing Techniques for AI Solutions, managers say this measure is live: Evaluate calibration across score bands and changing populations. They claim it corrected a material audit gap. What direct operating evidence would allow the auditor to substantiate that particular claim across a defensible sample?
- Confidence intervals, power assessment and subgroup sample-augmentation plan
- Failing schema test records and explicit rejection responses
- Reliability diagrams, Brier scores and out-of-sample calibration tables
- Production image stratification and condition-specific error rates
Explanation
Correct Answer
C:Correct. The proposed records—reliability diagrams, brier scores and out-of-sample calibration tables—are directly tied to the asserted safeguard and allow its actual operation to be corroborated. The gap being tested is calibration and confidence reliability were excluded from acceptance testing.
Incorrect Answers
A:“Confidence intervals, power assessment and subgroup sample-augmentation plan” would test subgroup sample size is inadequate to support equity conclusions, not the present safeguard against calibration and confidence reliability were excluded from acceptance testing. The expected audit evidence is “Reliability diagrams, Brier scores and out-of-sample calibration tables”.
B:“Failing schema test records and explicit rejection responses” would test negative testing did not exercise invalid or incomplete feature contracts, not the present safeguard against calibration and confidence reliability were excluded from acceptance testing. The expected audit evidence is “Reliability diagrams, Brier scores and out-of-sample calibration tables”.
D:“Production image stratification and condition-specific error rates” would test test dataset lacks representative camera and environmental conditions, not the present safeguard against calibration and confidence reliability were excluded from acceptance testing. The expected audit evidence is “Reliability diagrams, Brier scores and out-of-sample calibration tables”.
Question 333
In a review of Testing Techniques for AI Solutions, investigators cannot establish the expected control proof: reliability diagrams, brier scores and out-of-sample calibration tables. The underlying event was that a financial model is validated with point predictions but not the stability of its uncertainty estimates. Which control weakness is the MOST likely cause requiring focused audit work?
- Subgroup sample size is inadequate to support equity conclusions
- Calibration and confidence reliability were excluded from acceptance testing
- Negative testing did not exercise invalid or incomplete feature contracts
- Test dataset lacks representative camera and environmental conditions
Explanation
Correct Answer
B:Correct. The observed condition specifically indicates calibration and confidence reliability were excluded from acceptance testing. A meaningful audit response is to verify this diagnosis and then recommend evaluate calibration across score bands and changing populations.
Incorrect Answers
A:“Subgroup sample size is inadequate to support equity conclusions” concerns a separate audit weakness, not the missing assurance described by calibration and confidence reliability were excluded from acceptance testing. The correct cause is “Calibration and confidence reliability were excluded from acceptance testing”.
C:“Negative testing did not exercise invalid or incomplete feature contracts” concerns a separate audit weakness, not the missing assurance described by calibration and confidence reliability were excluded from acceptance testing. The correct cause is “Calibration and confidence reliability were excluded from acceptance testing”.
D:“Test dataset lacks representative camera and environmental conditions” concerns a separate audit weakness, not the missing assurance described by calibration and confidence reliability were excluded from acceptance testing. The correct cause is “Calibration and confidence reliability were excluded from acceptance testing”.
Question 334
An attacker plants instructions in a knowledge-base article to make a support bot export internal files. From an AI audit perspective, which corrective action is the MOST direct response to the weakness described, rather than a useful control for some different AI risk?
- Isolate untrusted content and enforce tool-level authorization and egress checks
- Authenticate data feeds, detect anomalous submissions and quarantine suspicious updates
- Treat OCR/image content as untrusted task data and constrain allowable actions
- Enforce server-side tenant authorization on every vector search
Explanation
Correct Answer
A:Correct. Isolate untrusted content and enforce tool-level authorization and egress checks is aimed at the actual failure: retrieved content is allowed to cross the instruction trust boundary. The audit should subsequently obtain red-team poisoned-document results and blocked file-export calls to test execution, not merely the existence of a policy.
Incorrect Answers
B:“Authenticate data feeds, detect anomalous submissions and quarantine suspicious updates” is a control for feedback data lacks integrity and poisoning detection, but the audit finding here is retrieved content is allowed to cross the instruction trust boundary. The relevant measure is “Isolate untrusted content and enforce tool-level authorization and egress checks”.
C:“Treat OCR/image content as untrusted task data and constrain allowable actions” is a control for visual text from documents was promoted to trusted system instruction, but the audit finding here is retrieved content is allowed to cross the instruction trust boundary. The relevant measure is “Isolate untrusted content and enforce tool-level authorization and egress checks”.
D:“Enforce server-side tenant authorization on every vector search” is a control for tenant isolation is enforced only by client-provided retrieval hints, but the audit finding here is retrieved content is allowed to cross the instruction trust boundary. The relevant measure is “Isolate untrusted content and enforce tool-level authorization and egress checks”.
Question 335
In an audit of Threats and Vulnerabilities Specific to AI, managers say this measure is live: Isolate untrusted content and enforce tool-level authorization and egress checks. They claim it corrected a material audit gap. What direct operating evidence would allow the auditor to substantiate that particular claim across a defensible sample?
- Quarantine decisions and signed provenance for accepted feedback
- Adversarial invoice tests showing extraction without instruction execution
- Cross-tenant negative queries and datastore policy test results
- Red-team poisoned-document results and blocked file-export calls
Explanation
Correct Answer
D:Correct. The proposed records—red-team poisoned-document results and blocked file-export calls—are directly tied to the asserted safeguard and allow its actual operation to be corroborated. The gap being tested is retrieved content is allowed to cross the instruction trust boundary.
Incorrect Answers
A:“Quarantine decisions and signed provenance for accepted feedback” would test feedback data lacks integrity and poisoning detection, not the present safeguard against retrieved content is allowed to cross the instruction trust boundary. The expected audit evidence is “Red-team poisoned-document results and blocked file-export calls”.
B:“Adversarial invoice tests showing extraction without instruction execution” would test visual text from documents was promoted to trusted system instruction, not the present safeguard against retrieved content is allowed to cross the instruction trust boundary. The expected audit evidence is “Red-team poisoned-document results and blocked file-export calls”.
C:“Cross-tenant negative queries and datastore policy test results” would test tenant isolation is enforced only by client-provided retrieval hints, not the present safeguard against retrieved content is allowed to cross the instruction trust boundary. The expected audit evidence is “Red-team poisoned-document results and blocked file-export calls”.
Question 336
In a review of Threats and Vulnerabilities Specific to AI, investigators cannot establish the expected control proof: red-team poisoned-document results and blocked file-export calls. The underlying event was that an attacker plants instructions in a knowledge-base article to make a support bot export internal files. Which control weakness is the MOST likely cause requiring focused audit work?
- Feedback data lacks integrity and poisoning detection
- Visual text from documents was promoted to trusted system instruction
- Retrieved content is allowed to cross the instruction trust boundary
- Tenant isolation is enforced only by client-provided retrieval hints
Explanation
Correct Answer
C:Correct. The observed condition specifically indicates retrieved content is allowed to cross the instruction trust boundary. A meaningful audit response is to verify this diagnosis and then recommend isolate untrusted content and enforce tool-level authorization and egress checks.
Incorrect Answers
A:“Feedback data lacks integrity and poisoning detection” concerns a separate audit weakness, not the missing assurance described by retrieved content is allowed to cross the instruction trust boundary. The correct cause is “Retrieved content is allowed to cross the instruction trust boundary”.
B:“Visual text from documents was promoted to trusted system instruction” concerns a separate audit weakness, not the missing assurance described by retrieved content is allowed to cross the instruction trust boundary. The correct cause is “Retrieved content is allowed to cross the instruction trust boundary”.
D:“Tenant isolation is enforced only by client-provided retrieval hints” concerns a separate audit weakness, not the missing assurance described by retrieved content is allowed to cross the instruction trust boundary. The correct cause is “Retrieved content is allowed to cross the instruction trust boundary”.
Question 337
A public model API permits unlimited repeated queries that reveal sensitive training examples. From an AI audit perspective, which corrective action is the MOST direct response to the weakness described, rather than a useful control for some different AI risk?
- Sign and verify model weight hashes at promotion and load
- Apply rate limits, anomaly detection and output disclosure controls
- Test membership inference and reduce memorization or public access as warranted
- Add adversarial interaction monitoring, guardrail hardening and escalation
Explanation
Correct Answer
B:Correct. Apply rate limits, anomaly detection and output disclosure controls is aimed at the actual failure: query-rate and extraction defenses are inadequate for model privacy. The audit should subsequently obtain extractive attack simulation, query telemetry and throttling events to test execution, not merely the existence of a policy.
Incorrect Answers
A:“Sign and verify model weight hashes at promotion and load” is a control for model artifacts themselves lack integrity controls independent of container signing, but the audit finding here is query-rate and extraction defenses are inadequate for model privacy. The relevant measure is “Apply rate limits, anomaly detection and output disclosure controls”.
C:“Test membership inference and reduce memorization or public access as warranted” is a control for membership-inference exposure was not included in privacy threat assessment, but the audit finding here is query-rate and extraction defenses are inadequate for model privacy. The relevant measure is “Apply rate limits, anomaly detection and output disclosure controls”.
D:“Add adversarial interaction monitoring, guardrail hardening and escalation” is a control for adaptive jailbreak testing and abuse monitoring were inadequate, but the audit finding here is query-rate and extraction defenses are inadequate for model privacy. The relevant measure is “Apply rate limits, anomaly detection and output disclosure controls”.
Question 338
In an audit of Threats and Vulnerabilities Specific to AI, managers say this measure is live: Apply rate limits, anomaly detection and output disclosure controls. They claim it corrected a material audit gap. What direct operating evidence would allow the auditor to substantiate that particular claim across a defensible sample?
- Extractive attack simulation, query telemetry and throttling events
- Failed hash validation on tampered model artifact in a release test
- Privacy attack baseline, mitigation results and endpoint authorization tests
- Red-team adaptive prompt series and risk alert response evidence
Explanation
Correct Answer
A:Correct. The proposed records—extractive attack simulation, query telemetry and throttling events—are directly tied to the asserted safeguard and allow its actual operation to be corroborated. The gap being tested is query-rate and extraction defenses are inadequate for model privacy.
Incorrect Answers
B:“Failed hash validation on tampered model artifact in a release test” would test model artifacts themselves lack integrity controls independent of container signing, not the present safeguard against query-rate and extraction defenses are inadequate for model privacy. The expected audit evidence is “Extractive attack simulation, query telemetry and throttling events”.
C:“Privacy attack baseline, mitigation results and endpoint authorization tests” would test membership-inference exposure was not included in privacy threat assessment, not the present safeguard against query-rate and extraction defenses are inadequate for model privacy. The expected audit evidence is “Extractive attack simulation, query telemetry and throttling events”.
D:“Red-team adaptive prompt series and risk alert response evidence” would test adaptive jailbreak testing and abuse monitoring were inadequate, not the present safeguard against query-rate and extraction defenses are inadequate for model privacy. The expected audit evidence is “Extractive attack simulation, query telemetry and throttling events”.
Question 339
In a review of Threats and Vulnerabilities Specific to AI, investigators cannot establish the expected control proof: extractive attack simulation, query telemetry and throttling events. The underlying event was that a public model API permits unlimited repeated queries that reveal sensitive training examples. Which control weakness is the MOST likely cause requiring focused audit work?
- Model artifacts themselves lack integrity controls independent of container signing
- Membership-inference exposure was not included in privacy threat assessment
- Adaptive jailbreak testing and abuse monitoring were inadequate
- Query-rate and extraction defenses are inadequate for model privacy
Explanation
Correct Answer
D:Correct. The observed condition specifically indicates query-rate and extraction defenses are inadequate for model privacy. A meaningful audit response is to verify this diagnosis and then recommend apply rate limits, anomaly detection and output disclosure controls.
Incorrect Answers
A:“Model artifacts themselves lack integrity controls independent of container signing” concerns a separate audit weakness, not the missing assurance described by query-rate and extraction defenses are inadequate for model privacy. The correct cause is “Query-rate and extraction defenses are inadequate for model privacy”.
B:“Membership-inference exposure was not included in privacy threat assessment” concerns a separate audit weakness, not the missing assurance described by query-rate and extraction defenses are inadequate for model privacy. The correct cause is “Query-rate and extraction defenses are inadequate for model privacy”.
C:“Adaptive jailbreak testing and abuse monitoring were inadequate” concerns a separate audit weakness, not the missing assurance described by query-rate and extraction defenses are inadequate for model privacy. The correct cause is “Query-rate and extraction defenses are inadequate for model privacy”.
Question 340
A recommendation model changes behavior after malicious reviews enter a feedback stream. From an AI audit perspective, which corrective action is the MOST direct response to the weakness described, rather than a useful control for some different AI risk?
- Restrict external tool destinations and inspect sensitive content before egress
- Verify provenance and execute untrusted packages in restricted build environments
- Authenticate data feeds, detect anomalous submissions and quarantine suspicious updates
- Isolate untrusted content and enforce tool-level authorization and egress checks
Explanation
Correct Answer
C:Correct. Authenticate data feeds, detect anomalous submissions and quarantine suspicious updates is aimed at the actual failure: feedback data lacks integrity and poisoning detection. The audit should subsequently obtain quarantine decisions and signed provenance for accepted feedback to test execution, not merely the existence of a policy.
Incorrect Answers
A:“Restrict external tool destinations and inspect sensitive content before egress” is a control for tool egress and data-classification checks are missing, but the audit finding here is feedback data lacks integrity and poisoning detection. The relevant measure is “Authenticate data feeds, detect anomalous submissions and quarantine suspicious updates”.
B:“Verify provenance and execute untrusted packages in restricted build environments” is a control for model supply-chain files were not scanned or sandboxed before execution, but the audit finding here is feedback data lacks integrity and poisoning detection. The relevant measure is “Authenticate data feeds, detect anomalous submissions and quarantine suspicious updates”.
D:“Isolate untrusted content and enforce tool-level authorization and egress checks” is a control for retrieved content is allowed to cross the instruction trust boundary, but the audit finding here is feedback data lacks integrity and poisoning detection. The relevant measure is “Authenticate data feeds, detect anomalous submissions and quarantine suspicious updates”.