ISACA Advanced in AI Audit AAIA Practice Test 43
Question 420
An audit estimates a model's error rate from 20 examples but claims population-level precision. From an AI audit perspective, which corrective action is the MOST direct response to the weakness described, rather than a useful control for some different AI risk?
- Preserve deterministic sample selection, population snapshots and trace IDs
- Apply correct weights and report group-specific intervals separately
- Set sample size and confidence limits appropriate to the test objective
- Include evolving attack strategies, threat-informed mutations and retesting
Explanation
Correct Answer
C:Correct. Set sample size and confidence limits appropriate to the test objective is aimed at the actual failure: sampling uncertainty and population representativeness were ignored. The audit should subsequently obtain sampling calculation, inclusion probabilities and confidence intervals to test execution, not merely the existence of a policy.
Incorrect Answers
A:“Preserve deterministic sample selection, population snapshots and trace IDs” is a control for sampling reproducibility and evidence chain are absent, but the audit finding here is sampling uncertainty and population representativeness were ignored. The relevant measure is “Set sample size and confidence limits appropriate to the test objective”.
B:“Apply correct weights and report group-specific intervals separately” is a control for stratified sample estimates were aggregated without sampling weights, but the audit finding here is sampling uncertainty and population representativeness were ignored. The relevant measure is “Set sample size and confidence limits appropriate to the test objective”.
D:“Include evolving attack strategies, threat-informed mutations and retesting” is a control for test selection does not reflect adaptable AI attack patterns, but the audit finding here is sampling uncertainty and population representativeness were ignored. The relevant measure is “Set sample size and confidence limits appropriate to the test objective”.
Question 421
In an audit of Audit Testing and Sampling Methodologies, managers say this measure is live: Set sample size and confidence limits appropriate to the test objective. They claim it corrected a material audit gap. What direct operating evidence would allow the auditor to substantiate that particular claim across a defensible sample?
- Seeded sampling script, frozen population and replayable sample manifests
- Sampling calculation, inclusion probabilities and confidence intervals
- Weighted estimator worksheet and sample-design documentation
- Versioned adversarial sample corpus and observed defense outcomes
Explanation
Correct Answer
B:Correct. The proposed records—sampling calculation, inclusion probabilities and confidence intervals—are directly tied to the asserted safeguard and allow its actual operation to be corroborated. The gap being tested is sampling uncertainty and population representativeness were ignored.
Incorrect Answers
A:“Seeded sampling script, frozen population and replayable sample manifests” would test sampling reproducibility and evidence chain are absent, not the present safeguard against sampling uncertainty and population representativeness were ignored. The expected audit evidence is “Sampling calculation, inclusion probabilities and confidence intervals”.
C:“Weighted estimator worksheet and sample-design documentation” would test stratified sample estimates were aggregated without sampling weights, not the present safeguard against sampling uncertainty and population representativeness were ignored. The expected audit evidence is “Sampling calculation, inclusion probabilities and confidence intervals”.
D:“Versioned adversarial sample corpus and observed defense outcomes” would test test selection does not reflect adaptable AI attack patterns, not the present safeguard against sampling uncertainty and population representativeness were ignored. The expected audit evidence is “Sampling calculation, inclusion probabilities and confidence intervals”.
Question 422
In a review of Audit Testing and Sampling Methodologies, investigators cannot establish the expected control proof: sampling calculation, inclusion probabilities and confidence intervals. The underlying event was that an audit estimates a model's error rate from 20 examples but claims population-level precision. Which control weakness is the MOST likely cause requiring focused audit work?
- Sampling uncertainty and population representativeness were ignored
- Sampling reproducibility and evidence chain are absent
- Stratified sample estimates were aggregated without sampling weights
- Test selection does not reflect adaptable AI attack patterns
Explanation
Correct Answer
A:Correct. The observed condition specifically indicates sampling uncertainty and population representativeness were ignored. A meaningful audit response is to verify this diagnosis and then recommend set sample size and confidence limits appropriate to the test objective.
Incorrect Answers
B:“Sampling reproducibility and evidence chain are absent” concerns a separate audit weakness, not the missing assurance described by sampling uncertainty and population representativeness were ignored. The correct cause is “Sampling uncertainty and population representativeness were ignored”.
C:“Stratified sample estimates were aggregated without sampling weights” concerns a separate audit weakness, not the missing assurance described by sampling uncertainty and population representativeness were ignored. The correct cause is “Sampling uncertainty and population representativeness were ignored”.
D:“Test selection does not reflect adaptable AI attack patterns” concerns a separate audit weakness, not the missing assurance described by sampling uncertainty and population representativeness were ignored. The correct cause is “Sampling uncertainty and population representativeness were ignored”.
Question 423
Auditors test random days but the AI scoring pipeline fails mainly during weekend batch jobs. From an AI audit perspective, which corrective action is the MOST direct response to the weakness described, rather than a useful control for some different AI risk?
- Apply correct weights and report group-specific intervals separately
- Draw independent samples from complete decision and override logs
- Select risk-stratified cases including high-impact exceptions and adversarial inputs
- Stratify samples by weekday, weekend and peak processing conditions
Explanation
Correct Answer
D:Correct. Stratify samples by weekday, weekend and peak processing conditions is aimed at the actual failure: selection method omits temporal operational risk windows. The audit should subsequently obtain date-stratified sample list and observed control execution to test execution, not merely the existence of a policy.
Incorrect Answers
A:“Apply correct weights and report group-specific intervals separately” is a control for stratified sample estimates were aggregated without sampling weights, but the audit finding here is selection method omits temporal operational risk windows. The relevant measure is “Stratify samples by weekday, weekend and peak processing conditions”.
B:“Draw independent samples from complete decision and override logs” is a control for selection bias undermines audit conclusions about control performance, but the audit finding here is selection method omits temporal operational risk windows. The relevant measure is “Stratify samples by weekday, weekend and peak processing conditions”.
C:“Select risk-stratified cases including high-impact exceptions and adversarial inputs” is a control for sampling design misses severity-stratified tail events, but the audit finding here is selection method omits temporal operational risk windows. The relevant measure is “Stratify samples by weekday, weekend and peak processing conditions”.
Question 424
In an audit of Audit Testing and Sampling Methodologies, managers say this measure is live: Stratify samples by weekday, weekend and peak processing conditions. They claim it corrected a material audit gap. What direct operating evidence would allow the auditor to substantiate that particular claim across a defensible sample?
- Weighted estimator worksheet and sample-design documentation
- Source-population reconciliation and independent random seed record
- Date-stratified sample list and observed control execution
- Sampling frame and documented coverage of high-severity scenarios
Explanation
Correct Answer
C:Correct. The proposed records—date-stratified sample list and observed control execution—are directly tied to the asserted safeguard and allow its actual operation to be corroborated. The gap being tested is selection method omits temporal operational risk windows.
Incorrect Answers
A:“Weighted estimator worksheet and sample-design documentation” would test stratified sample estimates were aggregated without sampling weights, not the present safeguard against selection method omits temporal operational risk windows. The expected audit evidence is “Date-stratified sample list and observed control execution”.
B:“Source-population reconciliation and independent random seed record” would test selection bias undermines audit conclusions about control performance, not the present safeguard against selection method omits temporal operational risk windows. The expected audit evidence is “Date-stratified sample list and observed control execution”.
D:“Sampling frame and documented coverage of high-severity scenarios” would test sampling design misses severity-stratified tail events, not the present safeguard against selection method omits temporal operational risk windows. The expected audit evidence is “Date-stratified sample list and observed control execution”.
Question 425
In a review of Audit Testing and Sampling Methodologies, investigators cannot establish the expected control proof: date-stratified sample list and observed control execution. The underlying event was that auditors test random days but the AI scoring pipeline fails mainly during weekend batch jobs. Which control weakness is the MOST likely cause requiring focused audit work?
- Stratified sample estimates were aggregated without sampling weights
- Selection method omits temporal operational risk windows
- Selection bias undermines audit conclusions about control performance
- Sampling design misses severity-stratified tail events
Explanation
Correct Answer
B:Correct. The observed condition specifically indicates selection method omits temporal operational risk windows. A meaningful audit response is to verify this diagnosis and then recommend stratify samples by weekday, weekend and peak processing conditions.
Incorrect Answers
A:“Stratified sample estimates were aggregated without sampling weights” concerns a separate audit weakness, not the missing assurance described by selection method omits temporal operational risk windows. The correct cause is “Selection method omits temporal operational risk windows”.
C:“Selection bias undermines audit conclusions about control performance” concerns a separate audit weakness, not the missing assurance described by selection method omits temporal operational risk windows. The correct cause is “Selection method omits temporal operational risk windows”.
D:“Sampling design misses severity-stratified tail events” concerns a separate audit weakness, not the missing assurance described by selection method omits temporal operational risk windows. The correct cause is “Selection method omits temporal operational risk windows”.
Question 426
A fraud system makes thousands of decisions but the auditor cannot link sample IDs to archived inputs. From an AI audit perspective, which corrective action is the MOST direct response to the weakness described, rather than a useful control for some different AI risk?
- Preserve deterministic sample selection, population snapshots and trace IDs
- Draw independent samples from complete decision and override logs
- Include evolving attack strategies, threat-informed mutations and retesting
- Set sample size and confidence limits appropriate to the test objective
Explanation
Correct Answer
A:Correct. Preserve deterministic sample selection, population snapshots and trace IDs is aimed at the actual failure: sampling reproducibility and evidence chain are absent. The audit should subsequently obtain seeded sampling script, frozen population and replayable sample manifests to test execution, not merely the existence of a policy.
Incorrect Answers
B:“Draw independent samples from complete decision and override logs” is a control for selection bias undermines audit conclusions about control performance, but the audit finding here is sampling reproducibility and evidence chain are absent. The relevant measure is “Preserve deterministic sample selection, population snapshots and trace IDs”.
C:“Include evolving attack strategies, threat-informed mutations and retesting” is a control for test selection does not reflect adaptable AI attack patterns, but the audit finding here is sampling reproducibility and evidence chain are absent. The relevant measure is “Preserve deterministic sample selection, population snapshots and trace IDs”.
D:“Set sample size and confidence limits appropriate to the test objective” is a control for sampling uncertainty and population representativeness were ignored, but the audit finding here is sampling reproducibility and evidence chain are absent. The relevant measure is “Preserve deterministic sample selection, population snapshots and trace IDs”.
Question 427
In an audit of Audit Testing and Sampling Methodologies, managers say this measure is live: Preserve deterministic sample selection, population snapshots and trace IDs. They claim it corrected a material audit gap. What direct operating evidence would allow the auditor to substantiate that particular claim across a defensible sample?
- Source-population reconciliation and independent random seed record
- Versioned adversarial sample corpus and observed defense outcomes
- Sampling calculation, inclusion probabilities and confidence intervals
- Seeded sampling script, frozen population and replayable sample manifests
Explanation
Correct Answer
D:Correct. The proposed records—seeded sampling script, frozen population and replayable sample manifests—are directly tied to the asserted safeguard and allow its actual operation to be corroborated. The gap being tested is sampling reproducibility and evidence chain are absent.
Incorrect Answers
A:“Source-population reconciliation and independent random seed record” would test selection bias undermines audit conclusions about control performance, not the present safeguard against sampling reproducibility and evidence chain are absent. The expected audit evidence is “Seeded sampling script, frozen population and replayable sample manifests”.
B:“Versioned adversarial sample corpus and observed defense outcomes” would test test selection does not reflect adaptable AI attack patterns, not the present safeguard against sampling reproducibility and evidence chain are absent. The expected audit evidence is “Seeded sampling script, frozen population and replayable sample manifests”.
C:“Sampling calculation, inclusion probabilities and confidence intervals” would test sampling uncertainty and population representativeness were ignored, not the present safeguard against sampling reproducibility and evidence chain are absent. The expected audit evidence is “Seeded sampling script, frozen population and replayable sample manifests”.
Question 428
In a review of Audit Testing and Sampling Methodologies, investigators cannot establish the expected control proof: seeded sampling script, frozen population and replayable sample manifests. The underlying event was that a fraud system makes thousands of decisions but the auditor cannot link sample IDs to archived inputs. Which control weakness is the MOST likely cause requiring focused audit work?
- Selection bias undermines audit conclusions about control performance
- Test selection does not reflect adaptable AI attack patterns
- Sampling reproducibility and evidence chain are absent
- Sampling uncertainty and population representativeness were ignored
Explanation
Correct Answer
C:Correct. The observed condition specifically indicates sampling reproducibility and evidence chain are absent. A meaningful audit response is to verify this diagnosis and then recommend preserve deterministic sample selection, population snapshots and trace ids.
Incorrect Answers
A:“Selection bias undermines audit conclusions about control performance” concerns a separate audit weakness, not the missing assurance described by sampling reproducibility and evidence chain are absent. The correct cause is “Sampling reproducibility and evidence chain are absent”.
B:“Test selection does not reflect adaptable AI attack patterns” concerns a separate audit weakness, not the missing assurance described by sampling reproducibility and evidence chain are absent. The correct cause is “Sampling reproducibility and evidence chain are absent”.
D:“Sampling uncertainty and population representativeness were ignored” concerns a separate audit weakness, not the missing assurance described by sampling reproducibility and evidence chain are absent. The correct cause is “Sampling reproducibility and evidence chain are absent”.
Question 429
A fairness review selects the same number from each group but reports unweighted population error rates. From an AI audit perspective, which corrective action is the MOST direct response to the weakness described, rather than a useful control for some different AI risk?
- Include evolving attack strategies, threat-informed mutations and retesting
- Apply correct weights and report group-specific intervals separately
- Select risk-stratified cases including high-impact exceptions and adversarial inputs
- Stratify samples by weekday, weekend and peak processing conditions
Explanation
Correct Answer
B:Correct. Apply correct weights and report group-specific intervals separately is aimed at the actual failure: stratified sample estimates were aggregated without sampling weights. The audit should subsequently obtain weighted estimator worksheet and sample-design documentation to test execution, not merely the existence of a policy.
Incorrect Answers
A:“Include evolving attack strategies, threat-informed mutations and retesting” is a control for test selection does not reflect adaptable AI attack patterns, but the audit finding here is stratified sample estimates were aggregated without sampling weights. The relevant measure is “Apply correct weights and report group-specific intervals separately”.
C:“Select risk-stratified cases including high-impact exceptions and adversarial inputs” is a control for sampling design misses severity-stratified tail events, but the audit finding here is stratified sample estimates were aggregated without sampling weights. The relevant measure is “Apply correct weights and report group-specific intervals separately”.
D:“Stratify samples by weekday, weekend and peak processing conditions” is a control for selection method omits temporal operational risk windows, but the audit finding here is stratified sample estimates were aggregated without sampling weights. The relevant measure is “Apply correct weights and report group-specific intervals separately”.
Question 430
In an audit of Audit Testing and Sampling Methodologies, managers say this measure is live: Apply correct weights and report group-specific intervals separately. They claim it corrected a material audit gap. What direct operating evidence would allow the auditor to substantiate that particular claim across a defensible sample?
- Weighted estimator worksheet and sample-design documentation
- Versioned adversarial sample corpus and observed defense outcomes
- Sampling frame and documented coverage of high-severity scenarios
- Date-stratified sample list and observed control execution
Explanation
Correct Answer
A:Correct. The proposed records—weighted estimator worksheet and sample-design documentation—are directly tied to the asserted safeguard and allow its actual operation to be corroborated. The gap being tested is stratified sample estimates were aggregated without sampling weights.
Incorrect Answers
B:“Versioned adversarial sample corpus and observed defense outcomes” would test test selection does not reflect adaptable AI attack patterns, not the present safeguard against stratified sample estimates were aggregated without sampling weights. The expected audit evidence is “Weighted estimator worksheet and sample-design documentation”.
C:“Sampling frame and documented coverage of high-severity scenarios” would test sampling design misses severity-stratified tail events, not the present safeguard against stratified sample estimates were aggregated without sampling weights. The expected audit evidence is “Weighted estimator worksheet and sample-design documentation”.
D:“Date-stratified sample list and observed control execution” would test selection method omits temporal operational risk windows, not the present safeguard against stratified sample estimates were aggregated without sampling weights. The expected audit evidence is “Weighted estimator worksheet and sample-design documentation”.