Topic 14 Practice Test 2 covers FM Evaluation, Quality Metrics, Model Comparison, A/B Testing, and Canary Validation in Domain 5 (Task 5.1). For broader exam preparation, review the AIP-C01 Exam Dumps. Every option includes focused technical reasoning tied to the tested decision.
Question 1
A model update passes accuracy checks, but the team is concerned that the semantic character of responses may have shifted in ways that break downstream expectations. Which validation best detects this risk? Choose ONE.
- Compare only the number of deployed model copies as an adjacent validation technique
- Increase maxTokens on the updated model as an efficiency or response-size measurement
- Run the same synthetic user workflows against both versions and compare semantic-output characteristics for drift
- Use only a throughput load test with no output inspection as a capacity-focused operational measurement
Correct Answer: C
Correct Answer
Answer C is correct because Use Run the same synthetic user workflows against both versions and compare semantic-output characteristics for drift. Deployment validation with stable synthetic workflows makes before-and-after behavior directly comparable. Applied here, A semantic-drift check on matched scenarios can reveal changed meaning even when headline accuracy remains acceptable. This addresses accuracy alone passes.
Incorrect Answers
Answer A is incorrect because Compare only the number of deployed model copies as an adjacent validation technique can support another concern. Deployment count is an infrastructure property. For the stated evaluation, It does not reveal semantic changes in model responses. That fails accuracy alone passes.
Answer B is incorrect because Increase maxTokens on the updated model as an efficiency or response-size measurement can support another concern. Token budget may alter response length. For the stated evaluation, Changing it would confound the validation rather than measure drift. That fails accuracy alone passes.
Answer D is incorrect because Use only a throughput load test with no output inspection as a capacity-focused operational measurement can support another concern. Load testing can verify capacity. For the stated evaluation, It cannot detect semantic changes when outputs are not evaluated. That fails accuracy alone passes.
Question 2
A knowledge assistant sometimes produces unsupported claims that sound plausible. Trusted reference answers are available for the evaluation set. Which metric most directly quantifies the risk? Choose ONE.
- Hallucination or factual-grounding rate against the trusted references
- Peak requests per second during testing as a narrower supporting signal
- Number of model versions stored in the account
- Average characters per response as a different evaluation dimension
Correct Answer: A
Correct Answer
Answer A is correct because Hallucination or factual-grounding rate against the trusted references is the relevant measure or method. Hallucination-oriented evaluation measures whether generated claims are unsupported or conflict with the reference evidence. Specifically, The defect is plausible unsupported content, so grounding or factuality must be scored explicitly. It meets trusted references exist.
Incorrect Answers
Answer B is incorrect because Peak requests per second during testing as a narrower supporting signal is an adjacent but insufficient choice. Throughput measures system capacity. Under these conditions, It does not establish whether response claims are supported. The key need remains trusted references exist.
Answer C is incorrect because Number of model versions stored in the account is an adjacent but insufficient choice. Version count supports lifecycle management. Under these conditions, It is unrelated to hallucination prevalence. The key need remains trusted references exist.
Answer D is incorrect because Average characters per response as a different evaluation dimension is an adjacent but insufficient choice. Character count measures length. Under these conditions, Unsupported claims can occur in either short or long responses. The key need remains trusted references exist.
Question 3
A creative-copy model is expected to vary wording, but a structured field called product_category must remain stable across repeated runs. Which evaluation should focus specifically on that requirement? Choose ONE.
- Only total inference cost per run as an adjacent validation technique
- Exact full-response string equality across every run as a separate operational measure
- Only user ratings for overall writing style as an adjacent validation technique
- Consistency of the required structured field across repeated equivalent requests
Correct Answer: D
Correct Answer
Answer D is correct because Consistency of the required structured field across repeated equivalent requests. Consistency can be scoped to the deterministic portion of an otherwise variable response. Here, The test can allow creative wording while checking whether product_category remains stable. It directly satisfies creative wording may vary.
Incorrect Answers
Answer A is incorrect because Only total inference cost per run as an adjacent validation technique measures or controls something different. Cost does not indicate whether product_category stays stable. Here, It cannot validate the field requirement. It does not resolve creative wording may vary.
Answer B is incorrect because Exact full-response string equality across every run as a separate operational measure measures or controls something different. Exact equality would reject intended creative variation. Here, It is stricter than the business requirement and would misclassify acceptable outputs. It does not resolve creative wording may vary.
Answer C is incorrect because Only user ratings for overall writing style as an adjacent validation technique measures or controls something different. Style ratings do not reliably verify a required structured field. Here, The deterministic field needs an explicit consistency check. It does not resolve creative wording may vary.
Question 4
A response is grammatically excellent and highly relevant, but it gives a wrong numerical answer. What does this demonstrate about the evaluation framework? Choose ONE.
- Fluency alone is a sufficient proxy for answer correctness as language-form quality rather than the stated decision criterion
- Quality must be measured across multiple dimensions because fluency and relevance do not guarantee factual correctness
- Relevance alone is a sufficient proxy for answer correctness as a different evaluation dimension
- Higher token usage would necessarily fix the numerical error as an efficiency or response-size measurement
Correct Answer: B
Correct Answer
Answer B is correct because GenAI quality dimensions capture different failure modes and should not be collapsed into one superficial signal. For this case, Quality must be measured across multiple dimensions because fluency and relevance do not guarantee factual correctness is appropriate because The response can pass language and relevance checks while failing factual accuracy. The deciding need is wrong numerical answer.
Incorrect Answers
Answer A is incorrect because Fluency measures how well language is formed. That makes Fluency alone is a sufficient proxy for answer correctness as language-form quality rather than the stated decision criterion useful elsewhere, but A fluent response can still contain an incorrect number, as the scenario demonstrates. The unmet requirement is wrong numerical answer.
Answer C is incorrect because Relevance measures whether the response addresses the request. That makes Relevance alone is a sufficient proxy for answer correctness as a different evaluation dimension useful elsewhere, but An on-topic answer can still be factually wrong. The unmet requirement is wrong numerical answer.
Answer D is incorrect because Token usage affects length and cost, not guaranteed correctness. That makes Higher token usage would necessarily fix the numerical error as an efficiency or response-size measurement useful elsewhere, but More tokens do not ensure the correct calculation or fact. The unmet requirement is wrong numerical answer.
Question 5
Product managers want a direct comparison of two answer styles for subjective helpfulness and tone. There is no objective ground-truth label. Which evaluation is most appropriate? Choose ONE.
- Pairwise human preference evaluation using a consistent rubric
- Only token-per-second measurement as an efficiency or response-size measurement
- Only retrieval recall from the knowledge base
- Exact string match against a single reference answer
Correct Answer: A
Correct Answer
Answer A is correct because Pairwise human preference evaluation using a consistent rubric fits the evaluation goal: Pairwise preference asks reviewers to compare alternatives directly on subjective criteria such as helpfulness and tone. In the stated scenario, Human judgment is appropriate when no objective ground truth exists and the distinction is experiential. Required condition: subjective helpfulness and tone.
Incorrect Answers
Answer B is incorrect because Only token-per-second measurement as an efficiency or response-size measurement is the wrong signal here. Throughput does not capture subjective helpfulness or tone. In this case, It is an operational rather than user-experience metric. It misses subjective helpfulness and tone.
Answer C is incorrect because Only retrieval recall from the knowledge base is the wrong signal here. Retrieval recall measures evidence acquisition. In this case, It cannot decide which writing style users prefer. It misses subjective helpfulness and tone.
Answer D is incorrect because Exact string match against a single reference answer is the wrong signal here. Exact match is unsuitable for valid free-form alternatives. In this case, Different wording can be equally helpful and appropriate. It misses subjective helpfulness and tone.
Question 6
Three models must be compared fairly on the same business tasks. Which dataset design best supports the comparison? Choose ONE.
- Compare only the list price of the models as a narrower supporting signal
- Give each model a different prompt set selected by its provider as an adjacent validation technique
- Use one versioned representative prompt set with the same scoring rubric for all three models
- Let each model choose only prompts it is confident about as a different evaluation dimension
Correct Answer: C
Correct Answer
Answer C is correct because Use Use one versioned representative prompt set with the same scoring rubric for all three models. A common versioned dataset and rubric keep evaluation inputs and criteria constant across candidates. Applied here, That isolates model/configuration differences from test-set changes. This addresses three models.
Incorrect Answers
Answer A is incorrect because Compare only the list price of the models as a narrower supporting signal can support another concern. Price matters for cost-performance analysis but does not measure task quality. For the stated evaluation, The team first needs a fair performance comparison on the same work. That fails three models.
Answer B is incorrect because Give each model a different prompt set selected by its provider as an adjacent validation technique can support another concern. Different datasets confound model quality with task difficulty. For the stated evaluation, Scores cannot be compared fairly when inputs differ. That fails three models.
Answer D is incorrect because Let each model choose only prompts it is confident about as a different evaluation dimension can support another concern. Self-selection biases the evaluation toward each model’s strengths. For the stated evaluation, It prevents comparable coverage of the target workload. That fails three models.
Question 7
A canary model receives 5% of traffic. Its average quality remains acceptable, but error rate for one critical workflow doubles. What should the canary controller do? Choose ONE.
- Increase the canary to 50% because the overall average still passes
- Rollback or stop the canary because the critical-workflow guardrail threshold is breached
- Ignore workflow segmentation and use only a global mean as an adjacent validation technique
- Delete the error metric from the dashboard as visualization without enforcing the required gate
Correct Answer: B
Correct Answer
Answer B is correct because Rollback or stop the canary because the critical-workflow guardrail threshold is breached is the relevant measure or method. Canary validation should monitor predeclared quality and reliability guardrails, including critical segmented metrics. Specifically, An aggregate average must not hide a material regression in a protected workflow. It meets 5 percent canary.
Incorrect Answers
Answer A is incorrect because Increase the canary to 50% because the overall average still passes is an adjacent but insufficient choice. Expanding exposure despite a critical regression increases risk. Under these conditions, The segmented guardrail is specifically intended to stop that outcome. The key need remains 5 percent canary.
Answer C is incorrect because Ignore workflow segmentation and use only a global mean as an adjacent validation technique is an adjacent but insufficient choice. A global mean can mask serious failures affecting a smaller but critical segment. Under these conditions, The protected workflow requires its own threshold. The key need remains 5 percent canary.
Answer D is incorrect because Delete the error metric from the dashboard as visualization without enforcing the required gate is an adjacent but insufficient choice. Removing the metric hides evidence instead of resolving the regression. Under these conditions, It defeats the purpose of canary validation. The key need remains 5 percent canary.
Question 8
An A/B model experiment routes A during morning hours and B during evening hours. B has higher satisfaction, but user populations differ strongly by time. What is the main correction? Choose ONE.
- Increase model B temperature to match morning traffic by changing generation randomness rather than evaluation design
- Use only model-provider benchmark scores instead of experiment data as external reference evidence rather than workload-specific validation
- Keep time-based assignment and compare only raw satisfaction means as a narrower supporting signal
- Randomize contemporaneous eligible requests between A and B instead of assigning by time of day
Correct Answer: D
Correct Answer
Answer D is correct because Randomize contemporaneous eligible requests between A and B instead of assigning by time of day. Randomization balances known and unknown traffic differences more effectively than deterministic time blocks. Here, It reduces time-of-day confounding when comparing satisfaction outcomes. It directly satisfies morning and evening users differ.
Incorrect Answers
Answer A is incorrect because Increase model B temperature to match morning traffic by changing generation randomness rather than evaluation design measures or controls something different. Temperature changes generation behavior but does not fix assignment bias. Here, The confounding remains. It does not resolve morning and evening users differ.
Answer B is incorrect because Use only model-provider benchmark scores instead of experiment data as external reference evidence rather than workload-specific validation measures or controls something different. External benchmarks do not answer the product’s real-user satisfaction question. Here, The experiment should be corrected, not discarded for unrelated evidence. It does not resolve morning and evening users differ.
Answer C is incorrect because Keep time-based assignment and compare only raw satisfaction means as a narrower supporting signal measures or controls something different. Raw means remain confounded by different user populations. Here, The experiment cannot attribute the difference cleanly to the model. It does not resolve morning and evening users differ.
Question 9
Model A costs less per request but succeeds on fewer user tasks. Model B costs more per request and succeeds more often. Which metric best compares economic efficiency? Choose ONE.
- Only the model with the fewest configuration parameters
- Raw cost per request without success measurement as an adjacent validation technique
- Cost per successful task that also meets the quality requirement
- Only the highest model accuracy score as a narrower supporting signal
Correct Answer: C
Correct Answer
Answer C is correct because Cost per successful qualified outcome accounts for both spending and whether the model actually completes the intended work. For this case, Cost per successful task that also meets the quality requirement is appropriate because It avoids falsely favoring cheap requests that fail more often. The deciding need is different request costs.
Incorrect Answers
Answer A is incorrect because Configuration complexity is not a direct cost-per-outcome metric. That makes Only the model with the fewest configuration parameters useful elsewhere, but It cannot quantify economic efficiency. The unmet requirement is different request costs.
Answer B is incorrect because Per-request cost ignores the additional failures and retries associated with poorer task performance. That makes Raw cost per request without success measurement as an adjacent validation technique useful elsewhere, but A cheaper request is not cheaper if it frequently fails the business task. The unmet requirement is different request costs.
Answer D is incorrect because Accuracy alone ignores the cost difference. That makes Only the highest model accuracy score as a narrower supporting signal useful elsewhere, but The team is explicitly comparing economic efficiency. The unmet requirement is different request costs.
Question 10
Two models meet the same quality target. One has better average latency but occasional very slow responses that violate the user SLA. Which performance statistic should be emphasized? Choose ONE.
- Only the arithmetic mean latency while measuring runtime performance rather than semantic correctness
- A tail-latency percentile such as p95 or p99 alongside the quality threshold
- Only total tokens processed per day as an efficiency or response-size measurement
- Only the number of evaluation prompts as an adjacent validation technique
Correct Answer: B
Correct Answer
Answer B is correct because A tail-latency percentile such as p95 or p99 alongside the quality threshold fits the evaluation goal: Tail percentiles expose slow outliers that averages can conceal. In the stated scenario, The SLA violation is caused by occasional high latency, so a tail metric is needed for model comparison. Required condition: same quality target.
Incorrect Answers
Answer A is incorrect because Only the arithmetic mean latency while measuring runtime performance rather than semantic correctness is the wrong signal here. The mean can remain good while a significant tail exceeds the SLA. In this case, It does not characterize the slow-response risk described. It misses same quality target.
Answer C is incorrect because Only total tokens processed per day as an efficiency or response-size measurement is the wrong signal here. Daily tokens measure volume and cost exposure. In this case, They do not reveal tail response-time behavior. It misses same quality target.
Answer D is incorrect because Only the number of evaluation prompts as an adjacent validation technique is the wrong signal here. Dataset size affects confidence in the test. In this case, It is not a latency statistic and cannot detect SLA outliers. It misses same quality target.
Question 11
A feedback form records a one-to-five rating, but the team cannot tell whether low scores came from factual errors, tone, or missing citations. What change most improves diagnostic value? Choose ONE.
- Replace ratings with total prompt-token count as an efficiency or response-size measurement
- Collect ratings only from users who submit support tickets as a separate operational measure
- Round all ratings to either zero or one as a separate operational measure
- Add structured reason labels and optional reviewer comments linked to each rating
Correct Answer: D
Correct Answer
Answer D is correct because Use Add structured reason labels and optional reviewer comments linked to each rating. Reason labels decompose a general score into actionable failure categories while comments retain nuanced context. Applied here, The team can then separate factual, tone, and citation defects for targeted improvements. This addresses rating alone lacks cause.
Incorrect Answers
Answer A is incorrect because Replace ratings with total prompt-token count as an efficiency or response-size measurement can support another concern. Token count does not express user satisfaction or the reason for a poor answer. For the stated evaluation, It cannot diagnose quality defects. That fails rating alone lacks cause.
Answer B is incorrect because Collect ratings only from users who submit support tickets as a separate operational measure can support another concern. Ticket submitters are a self-selected group. For the stated evaluation, Restricting feedback to them can introduce bias without adding the missing failure categories. That fails rating alone lacks cause.
Answer C is incorrect because Round all ratings to either zero or one as a separate operational measure can support another concern. Binarizing the score removes information and still does not identify the failure cause. For the stated evaluation, It makes diagnostics weaker. That fails rating alone lacks cause.
Question 12
Two annotators frequently disagree on whether responses satisfy a safety-policy rubric. What is the best next step for the evaluation process? Choose ONE.
- Measure inter-annotator agreement, refine ambiguous rubric guidance, and adjudicate disputed examples
- Increase model temperature before annotation by changing generation randomness rather than evaluation design
- Randomly keep one annotator’s label for each disagreement
- Discard every disputed example permanently as an adjacent validation technique
Correct Answer: A
Correct Answer
Answer A is correct because Measure inter-annotator agreement, refine ambiguous rubric guidance, and adjudicate disputed examples is the relevant measure or method. Agreement analysis reveals inconsistent criteria, while rubric refinement and adjudication create a more reliable labeling standard. Specifically, The repeated disagreement signals an evaluation-process problem rather than a reason to average labels blindly. It meets frequent annotator disagreement.
Incorrect Answers
Answer B is incorrect because Increase model temperature before annotation by changing generation randomness rather than evaluation design is an adjacent but insufficient choice. Generator randomness does not resolve inconsistent human rubric interpretation. Under these conditions, It can create even more variable examples. The key need remains frequent annotator disagreement.
Answer C is incorrect because Randomly keep one annotator’s label for each disagreement is an adjacent but insufficient choice. Random selection preserves noise and gives no insight into why the rubric is interpreted differently. Under these conditions, It does not improve label reliability. The key need remains frequent annotator disagreement.
Answer D is incorrect because Discard every disputed example permanently as an adjacent validation technique is an adjacent but insufficient choice. Removing hard examples can create an unrealistically easy evaluation set. Under these conditions, Those cases often reveal the exact boundaries the rubric needs to clarify. The key need remains frequent annotator disagreement.
Question 13
A production model is stable today, but the organization wants early warning when quality gradually degrades as traffic patterns change. Which process best supports this goal? Choose ONE.
- Evaluate only once before the original launch as an adjacent validation technique
- Run continuous evaluation on representative production samples and trend quality metrics against baselines
- Track only infrastructure CPU utilization as a separate operational measure
- Increase prompt length whenever a weekly metric changes as a different evaluation dimension
Correct Answer: B
Correct Answer
Answer B is correct because Run continuous evaluation on representative production samples and trend quality metrics against baselines. Continuous evaluation turns current traffic into recurring quality measurements that can reveal gradual degradation. Here, Trending against an established baseline provides early drift or regression evidence. It directly satisfies need early warning.
Incorrect Answers
Answer A is incorrect because Evaluate only once before the original launch as an adjacent validation technique measures or controls something different. A one-time prelaunch test cannot observe future changes in traffic or behavior. Here, It provides no ongoing warning. It does not resolve need early warning.
Answer C is incorrect because Track only infrastructure CPU utilization as a separate operational measure measures or controls something different. CPU can reveal resource pressure but not semantic output quality. Here, Quality may degrade while utilization remains normal. It does not resolve need early warning.
Answer D is incorrect because Increase prompt length whenever a weekly metric changes as a different evaluation dimension measures or controls something different. Longer prompts are not a generic drift remedy. Here, The team first needs reliable continuous measurement and diagnosis. It does not resolve need early warning.
Question 14
A release team wants to know whether model version 12 is worse than version 11 on previously approved behavior. Which comparison creates the strongest regression evidence? Choose ONE.
- Compare only deployment timestamps for the two versions as lifecycle metadata rather than a response-quality metric
- Compare only the number of tokens each version supports as an efficiency or response-size measurement
- Evaluate version 12 on a new dataset and compare it with old version-11 dashboard scores
- Evaluate both versions on the same frozen regression set with identical scoring rules
Correct Answer: D
Correct Answer
Answer D is correct because A frozen regression set controls the inputs and criteria so differences reflect version behavior rather than test changes. For this case, Evaluate both versions on the same frozen regression set with identical scoring rules is appropriate because Direct baseline-versus-candidate comparison is the clearest evidence of regression. The deciding need is version 12 vs version 11.
Incorrect Answers
Answer A is incorrect because Timestamps establish chronology, not output quality. That makes Compare only deployment timestamps for the two versions as lifecycle metadata rather than a response-quality metric useful elsewhere, but They cannot measure regression. The unmet requirement is version 12 vs version 11.
Answer B is incorrect because Token capacity is not the approved behavior being tested. That makes Compare only the number of tokens each version supports as an efficiency or response-size measurement useful elsewhere, but It cannot reveal quality regression on known cases. The unmet requirement is version 12 vs version 11.
Answer C is incorrect because Different datasets can have different difficulty and composition. That makes Evaluate version 12 on a new dataset and compare it with old version-11 dashboard scores useful elsewhere, but The score difference may come from the test set rather than the model version. The unmet requirement is version 12 vs version 11.
Question 15
A RAG assistant must be evaluated for both evidence retrieval and final answer quality. Which design gives the most complete view? Choose ONE.
- Evaluate retrieval quality and retrieve-and-generate response quality with ground truth for the representative queries
- Measure only the number of retrieved chunks as a different evaluation dimension
- Evaluate only final response fluency as language-form quality rather than the stated decision criterion
- Measure only vector-index storage size as a separate operational measure
Correct Answer: A
Correct Answer
Answer A is correct because Evaluate retrieval quality and retrieve-and-generate response quality with ground truth for the representative queries fits the evaluation goal: RAG evaluation can assess the retriever and the final generated response as separate but related stages. In the stated scenario, Ground-truth queries let the team identify whether failures arise before or after generation. Required condition: RAG assistant.
Incorrect Answers
Answer B is incorrect because Measure only the number of retrieved chunks as a different evaluation dimension is the wrong signal here. Chunk count says how many items were returned, not whether they were relevant or used correctly. In this case, It is not a complete RAG quality evaluation. It misses RAG assistant.
Answer C is incorrect because Evaluate only final response fluency as language-form quality rather than the stated decision criterion is the wrong signal here. Fluency can be high even when retrieved evidence is irrelevant or the answer is unsupported. In this case, It omits the retrieval dimension entirely. It misses RAG assistant.
Answer D is incorrect because Measure only vector-index storage size as a separate operational measure is the wrong signal here. Storage size is an infrastructure characteristic. In this case, It does not score evidence relevance or final response quality. It misses RAG assistant.
Question 16
A team suspects its generator is fine but retrieval is poor. It wants the evaluation to exclude generation entirely. Which configuration best meets that need? Choose ONE.
- Run only a human preference test on final generated prose
- Run an A/B test of two prompt styles with the same retriever
- Run a retrieve-only RAG evaluation on representative queries with expected context
- Compare only the models’ context-window limits as a narrower supporting signal
Correct Answer: C
Correct Answer
Answer C is correct because Use Run a retrieve-only RAG evaluation on representative queries with expected context. Retrieve-only evaluation focuses on whether relevant information is found without scoring generated answers. Applied here, It isolates the suspected retriever and avoids generator confounding. This addresses generator believed fine.
Incorrect Answers
Answer A is incorrect because Run only a human preference test on final generated prose can support another concern. Final-answer preference includes generator effects. For the stated evaluation, It cannot isolate retrieval quality. That fails generator believed fine.
Answer B is incorrect because Run an A/B test of two prompt styles with the same retriever can support another concern. Prompt-style testing changes generation instructions. For the stated evaluation, It leaves the suspected retrieval behavior unmeasured. That fails generator believed fine.
Answer D is incorrect because Compare only the models’ context-window limits as a narrower supporting signal can support another concern. Context capacity does not indicate which passages the retriever selects. For the stated evaluation, It is unrelated to retrieve-only relevance. That fails generator believed fine.
Question 17
An LLM judge scores customer-service responses. The team notices it consistently favors verbose answers even when users prefer concise ones. What should the team do? Choose ONE.
- Increase the generator maxTokens so all answers become longer as an efficiency or response-size measurement
- Remove human preference data from the validation process as a narrower supporting signal
- Accept the judge bias because automated evaluation must be objective as an adjacent validation technique
- Revise and calibrate the judge rubric against human-labeled examples that represent the desired concision preference
Correct Answer: D
Correct Answer
Answer D is correct because Revise and calibrate the judge rubric against human-labeled examples that represent the desired concision preference is the relevant measure or method. Judge behavior depends on the evaluation criteria and can exhibit systematic bias. Specifically, Human-labeled calibration examples reveal the preference mismatch and guide a rubric that rewards the intended response style. It meets judge favors verbosity.
Incorrect Answers
Answer A is incorrect because Increase the generator maxTokens so all answers become longer as an efficiency or response-size measurement is an adjacent but insufficient choice. That adapts the product to the evaluator’s bias instead of fixing the evaluation. Under these conditions, It moves responses further from user preference. The key need remains judge favors verbosity.
Answer B is incorrect because Remove human preference data from the validation process as a narrower supporting signal is an adjacent but insufficient choice. Human evidence is exactly what revealed the misalignment. Under these conditions, Discarding it would hide rather than correct the judge bias. The key need remains judge favors verbosity.
Answer C is incorrect because Accept the judge bias because automated evaluation must be objective as an adjacent validation technique is an adjacent but insufficient choice. LLM judges are not inherently unbiased. Under these conditions, Known systematic disagreement with desired human preference should be corrected or bounded. The key need remains judge favors verbosity.
Question 18
An AI agent usually returns a polished final answer, but traces show it calls the same external tool repeatedly and sometimes fails to complete the requested action. Which evaluation metrics should be emphasized? Choose ONE.
- Task completion rate and tool-usage effectiveness
- Only the foundation model’s public benchmark rank
- Only final-answer fluency as language-form quality rather than the stated decision criterion
- Only the number of words in the final response
Correct Answer: A
Correct Answer
Answer A is correct because Task completion rate and tool-usage effectiveness. Agent evaluation must measure whether the goal is achieved and whether tools are used correctly and efficiently across multi-step execution. Here, Polished language alone can hide failed tasks and wasteful repeated tool calls. It directly satisfies repeated tool calls.
Incorrect Answers
Answer B is incorrect because Only the foundation model’s public benchmark rank measures or controls something different. Public model benchmarks do not observe this agent’s tool sequence or task completion. Here, The failure is in agent execution behavior. It does not resolve repeated tool calls.
Answer C is incorrect because Only final-answer fluency as language-form quality rather than the stated decision criterion measures or controls something different. Fluency measures the surface quality of the final text. Here, It does not show whether the agent completed the action or used tools effectively. It does not resolve repeated tool calls.
Answer D is incorrect because Only the number of words in the final response measures or controls something different. Response length is not an agent-performance measure. Here, A concise or verbose answer can accompany the same tool-loop failure. It does not resolve repeated tool calls.
Question 19
A model update will change a production workflow. Before sending any real-user traffic, the team wants a reproducible end-to-end check that exercises login, retrieval, model response, and downstream action. What should it use? Choose ONE.
- Only a unit test of the prompt-template string as a narrower supporting signal
- Only a model throughput benchmark as a capacity-focused operational measurement
- Synthetic user workflows that exercise the complete path with automated output validation
- Only a manual check of one production user after release
Correct Answer: C
Correct Answer
Answer C is correct because Synthetic workflows reproduce representative end-to-end interactions without requiring real-user exposure. For this case, Synthetic user workflows that exercise the complete path with automated output validation is appropriate because They can validate both integration steps and AI-specific output criteria before deployment. The deciding need is no real-user traffic yet.
Incorrect Answers
Answer A is incorrect because A prompt unit test covers one component. That makes Only a unit test of the prompt-template string as a narrower supporting signal useful elsewhere, but It cannot validate login, retrieval, downstream integration, and final output together. The unmet requirement is no real-user traffic yet.
Answer B is incorrect because Throughput verifies capacity but not end-to-end workflow correctness. That makes Only a model throughput benchmark as a capacity-focused operational measurement useful elsewhere, but The requirement spans multiple application steps. The unmet requirement is no real-user traffic yet.
Answer D is incorrect because That introduces real-user exposure before validation and is not reproducible. That makes Only a manual check of one production user after release useful elsewhere, but It fails the predeployment requirement. The unmet requirement is no real-user traffic yet.
Question 20
Executives need a recurring report to compare two model configurations. They care about user quality, latency, cost, and completed business tasks. What should the report contain? Choose ONE.
- Only average token count without quality or outcome data as an efficiency or response-size measurement
- A versioned scorecard showing quality, latency, cost efficiency, and business-outcome metrics for both configurations
- Only one selected best-case example from each model as an adjacent validation technique
- Only the model identifier and launch date as a narrower supporting signal
Correct Answer: B
Correct Answer
Answer B is correct because A versioned scorecard showing quality, latency, cost efficiency, and business-outcome metrics for both configurations fits the evaluation goal: A multi-dimensional comparison report communicates the trade-offs stakeholders actually need to make a deployment decision. In the stated scenario, Including common definitions and model versions prevents a single metric from hiding important consequences. Required condition: executive comparison.
Incorrect Answers
Answer A is incorrect because Only average token count without quality or outcome data as an efficiency or response-size measurement is the wrong signal here. Token usage helps explain cost, but does not establish whether users receive better results or complete tasks. In this case, It is one dimension, not the requested report. It misses executive comparison.
Answer C is incorrect because Only one selected best-case example from each model as an adjacent validation technique is the wrong signal here. Anecdotes can illustrate behavior but are not representative metrics. In this case, They cannot summarize quality, latency, cost, and business outcomes. It misses executive comparison.
Answer D is incorrect because Only the model identifier and launch date as a narrower supporting signal is the wrong signal here. Identity and chronology provide context but no performance evidence. In this case, They cannot support the requested trade-off decision. It misses executive comparison.
