Topic 14 Practice Test 1 covers FM Evaluation, Quality Metrics, Model Comparison, A/B Testing, and Canary Validation in Domain 5 (Task 5.1). For broader exam preparation, review the AIP-C01 Exam Dumps. Every option includes focused technical reasoning tied to the tested decision.
Question 1
A medical-answering pilot is fluent and concise, but reviewers found several statements that conflict with the approved clinical reference. Which evaluation signal should be prioritized before release? Choose ONE.
- Peak token throughput during batch inference as an efficiency or response-size measurement
- Factual accuracy against the approved reference set
- Average response length across all test prompts
- Fluency ratings from a general language rubric
Correct Answer: B
Correct Answer
Answer B is correct because Factual accuracy against the approved reference set fits the evaluation goal: Factual-accuracy evaluation checks whether claims agree with trusted evidence rather than merely reading smoothly. In the stated scenario, The observed defect is incorrect factual content, so the evaluation must directly measure agreement with the approved reference. Required condition: trusted clinical reference exists.
Incorrect Answers
Answer A is incorrect because Peak token throughput during batch inference as an efficiency or response-size measurement is the wrong signal here. Throughput measures how much inference work is processed over time. In this case, Performance capacity does not establish whether medical claims are correct. It misses trusted clinical reference exists.
Answer C is incorrect because Average response length across all test prompts is the wrong signal here. Response length is a format or efficiency characteristic. In this case, A shorter or longer answer can still contain the same factual errors, so length does not diagnose the reported risk. It misses trusted clinical reference exists.
Answer D is incorrect because Fluency ratings from a general language rubric is the wrong signal here. Fluency measures readability and linguistic quality. In this case, The pilot is already fluent; this signal can remain high while the answer contradicts the clinical reference. It misses trusted clinical reference exists.
Question 2
A customer assistant answers the right topic but often rambles, repeats itself, and uses awkward grammar. Facts are correct. Which metric best targets the remaining quality problem? Choose ONE.
- Retrieval latency at the vector database while measuring runtime performance rather than semantic correctness
- Hallucination rate against a source corpus as a separate operational measure
- Provisioned throughput utilization for the model as a capacity-focused operational measurement
- Fluency and coherence scoring on representative responses
Correct Answer: D
Correct Answer
Answer D is correct because Use Fluency and coherence scoring on representative responses. Fluency and coherence assess readability, grammatical quality, and whether the response is well formed. Applied here, Those dimensions match the awkward and repetitive output while leaving factual correctness as a separate already-passing measure. This addresses facts already correct.
Incorrect Answers
Answer A is incorrect because Retrieval latency at the vector database while measuring runtime performance rather than semantic correctness can support another concern. Retrieval latency measures the time required to obtain supporting context. For the stated evaluation, The scenario reports a language-quality defect after correct content is produced, not slow retrieval. That fails facts already correct.
Answer B is incorrect because Hallucination rate against a source corpus as a separate operational measure can support another concern. Hallucination evaluation checks unsupported or false claims. For the stated evaluation, The facts are stated to be correct, so hallucination rate does not directly measure rambling grammar and poor coherence. That fails facts already correct.
Answer C is incorrect because Provisioned throughput utilization for the model as a capacity-focused operational measurement can support another concern. Utilization indicates how much reserved inference capacity is consumed. For the stated evaluation, Capacity usage cannot determine whether wording is readable or repetitive. That fails facts already correct.
Question 3
The same deterministic extraction request is run repeatedly with unchanged input and configuration. The output schema is valid every time, but extracted categories vary unexpectedly. Which quality dimension should the test emphasize? Choose ONE.
- Consistency across repeated equivalent invocations
- Human preference between two writing styles as a separate operational measure
- Knowledge-base ingestion completion percentage as a separate operational measure
- Average prompt-token count per request as an efficiency or response-size measurement
Correct Answer: A
Correct Answer
Answer A is correct because Consistency across repeated equivalent invocations is the relevant measure or method. Consistency measures whether equivalent requests produce suitably stable outputs under the intended configuration. Specifically, Unexpected category variation across otherwise identical tests is directly a consistency problem. It meets same input and configuration.
Incorrect Answers
Answer B is incorrect because Human preference between two writing styles as a separate operational measure is an adjacent but insufficient choice. Preference testing is useful for subjective presentation choices. Under these conditions, The issue is repeatability of extracted categories, not which prose style users prefer. The key need remains same input and configuration.
Answer C is incorrect because Knowledge-base ingestion completion percentage as a separate operational measure is an adjacent but insufficient choice. Ingestion completion indicates whether source documents were processed. Under these conditions, The test concerns repeated model output from the same request, not document ingestion. The key need remains same input and configuration.
Answer D is incorrect because Average prompt-token count per request as an efficiency or response-size measurement is an adjacent but insufficient choice. Prompt-token count measures input size and cost exposure. Under these conditions, The request is unchanged, so token count does not explain whether outputs remain stable. The key need remains same input and configuration.
Question 4
A support bot gives factually correct background information but frequently fails to answer the user’s actual question. Which metric should receive the strongest weight in the evaluation rubric? Choose ONE.
- Model invocation logging retention period as a different evaluation dimension
- Total number of output tokens generated as an efficiency or response-size measurement
- Relevance of the response to the user request
- Availability-zone count for the inference path as a different evaluation dimension
Correct Answer: C
Correct Answer
Answer C is correct because Relevance of the response to the user request. Relevance measures whether the response addresses the question or task that was asked. Here, Correct but off-topic background can score well on factuality while failing relevance, which is the stated defect. It directly satisfies facts can be correct.
Incorrect Answers
Answer A is incorrect because Model invocation logging retention period as a different evaluation dimension measures or controls something different. Log retention controls how long operational evidence is kept. Here, Retention can support auditability but does not itself measure response relevance. It does not resolve facts can be correct.
Answer B is incorrect because Total number of output tokens generated as an efficiency or response-size measurement measures or controls something different. Token count tracks response size and may contribute to cost. Here, A response can be short or long and still avoid the user’s requested task. It does not resolve facts can be correct.
Answer D is incorrect because Availability-zone count for the inference path as a different evaluation dimension measures or controls something different. Zone topology contributes to infrastructure resiliency. Here, It does not score whether an answer addresses the user request. It does not resolve facts can be correct.
Question 5
A legal drafting assistant must meet minimum thresholds for factual accuracy, relevance, and fluency. A candidate model excels on fluency but falls below the factual threshold. How should the release scorecard treat it? Choose ONE.
- Average all three scores and release if the mean is high
- Fail the candidate because a mandatory quality threshold is not met
- Ignore factual accuracy when the response is well written
- Replace the scorecard with only token-cost measurement as an efficiency or response-size measurement
Correct Answer: B
Correct Answer
Answer B is correct because A multi-metric scorecard can enforce non-negotiable floors for dimensions whose failure is unacceptable. For this case, Fail the candidate because a mandatory quality threshold is not met is appropriate because High fluency should not compensate for factual accuracy below a legal-quality release threshold. The deciding need is legal use case.
Incorrect Answers
Answer A is incorrect because A simple average permits a strong dimension to offset a critical weakness. That makes Average all three scores and release if the mean is high useful elsewhere, but That would allow fluency to mask a factuality failure even though the factual threshold is mandatory. The unmet requirement is legal use case.
Answer C is incorrect because Writing quality and factual correctness are independent dimensions. That makes Ignore factual accuracy when the response is well written useful elsewhere, but A polished legal answer can still be unsafe when its claims are incorrect. The unmet requirement is legal use case.
Answer D is incorrect because Token cost measures efficiency rather than output validity. That makes Replace the scorecard with only token-cost measurement as an efficiency or response-size measurement useful elsewhere, but Cost alone cannot enforce the stated quality requirements. The unmet requirement is legal use case.
Question 6
A team must compare two foundation models on its own representative prompts before choosing one for a production summarization workload. It wants repeatable model-level metrics without exposing real users. Which approach is best? Choose ONE.
- Run an Amazon Bedrock model evaluation on the same representative prompt dataset for both models
- Choose the model with the largest context window without testing outputs as a model-capacity characteristic rather than an outcome metric
- Compare only each model provider’s published benchmark score as external reference evidence rather than workload-specific validation
- Send all production users to model A for one month and then all users to model B
Correct Answer: A
Correct Answer
Answer A is correct because Run an Amazon Bedrock model evaluation on the same representative prompt dataset for both models fits the evaluation goal: Bedrock model evaluations can assess model responses using controlled datasets and defined evaluation methods. In the stated scenario, Using the same representative prompts creates a repeatable offline comparison before production traffic is involved. Required condition: no real-user exposure.
Incorrect Answers
Answer B is incorrect because Choose the model with the largest context window without testing outputs as a model-capacity characteristic rather than an outcome metric is the wrong signal here. Context capacity is only one technical characteristic. In this case, It does not establish summarization quality on the organization’s representative inputs. It misses no real-user exposure.
Answer C is incorrect because Compare only each model provider’s published benchmark score as external reference evidence rather than workload-specific validation is the wrong signal here. Published benchmarks may not reflect the organization’s prompts and constraints. In this case, The team needs evidence on its own representative workload. It misses no real-user exposure.
Answer D is incorrect because Send all production users to model A for one month and then all users to model B is the wrong signal here. Sequential exposure mixes model effects with changes in traffic, seasonality, and user behavior. In this case, The requirement is a controlled pre-production comparison, not a time-confounded production trial. It misses no real-user exposure.
Question 7
Two approved models are ready for production. The product team wants to know which one improves completed purchases for real users while keeping prompt mix comparable. Which experiment is most appropriate? Choose ONE.
- Route all traffic to the lower-cost model and inspect logs later
- Use model A during weekdays and model B during weekends as an adjacent validation technique
- Compare the models only by parameter count as a narrower supporting signal
- Randomly assign comparable live requests between the two models and compare conversion outcomes
Correct Answer: D
Correct Answer
Answer D is correct because Use Randomly assign comparable live requests between the two models and compare conversion outcomes. A/B testing distributes contemporaneous traffic between alternatives so outcome differences can be attributed more credibly to the treatment. Applied here, Random assignment directly tests the business outcome on comparable real-user traffic. This addresses real production users.
Incorrect Answers
Answer A is incorrect because Route all traffic to the lower-cost model and inspect logs later can support another concern. Cost-first routing does not create a controlled comparison of conversion outcomes. For the stated evaluation, It cannot determine whether another model would improve the target business result. That fails real production users.
Answer B is incorrect because Use model A during weekdays and model B during weekends as an adjacent validation technique can support another concern. Time-based assignment confounds model effects with day-of-week traffic differences. For the stated evaluation, The prompt mix and user population may change systematically between the periods. That fails real production users.
Answer C is incorrect because Compare the models only by parameter count as a narrower supporting signal can support another concern. Parameter count is not a business-outcome metric. For the stated evaluation, A larger model does not necessarily increase completed purchases. That fails real production users.
Question 8
A replacement model has passed offline evaluation. The team wants production evidence while limiting the blast radius and automatically reverting if hallucination rate exceeds the approved threshold. Which rollout is best? Choose ONE.
- Replace the old model for all traffic and compare incidents afterward as a narrower supporting signal
- Increase the new model temperature during the rollout by changing generation randomness rather than evaluation design
- Send a small controlled share of production traffic to the new model as a canary with rollback thresholds
- Run only a synthetic load test against the old model as a narrower supporting signal
Correct Answer: C
Correct Answer
Answer C is correct because Send a small controlled share of production traffic to the new model as a canary with rollback thresholds is the relevant measure or method. Canary testing exposes a limited cohort to a candidate while monitoring defined health and quality metrics. Specifically, It supplies real production evidence with bounded risk and supports rollback when the hallucination threshold is breached. It meets offline evaluation already passed.
Incorrect Answers
Answer A is incorrect because Replace the old model for all traffic and compare incidents afterward as a narrower supporting signal is an adjacent but insufficient choice. A full cutover maximizes the blast radius before real-world behavior is known. Under these conditions, It does not satisfy the requirement to limit exposure during validation. The key need remains offline evaluation already passed.
Answer B is incorrect because Increase the new model temperature during the rollout by changing generation randomness rather than evaluation design is an adjacent but insufficient choice. Temperature changes sampling behavior but is not a deployment-validation strategy. Under these conditions, It neither limits traffic exposure nor provides controlled rollback criteria. The key need remains offline evaluation already passed.
Answer D is incorrect because Run only a synthetic load test against the old model as a narrower supporting signal is an adjacent but insufficient choice. Load testing the existing model may verify capacity. Under these conditions, It provides no production evidence about the replacement model. The key need remains offline evaluation already passed.
Question 9
Model X is slightly more accurate but has much higher latency. Model Y meets the quality floor and responds far faster. The application has a strict interactive SLA. Which comparison measure best supports the decision? Choose ONE.
- Latency-to-quality analysis with the SLA and quality floor applied together
- Select the model with the smallest model identifier string
- Compare only the models’ training-data cutoff dates as an adjacent validation technique
- Select the model with the highest standalone accuracy score
Correct Answer: A
Correct Answer
Answer A is correct because Latency-to-quality analysis with the SLA and quality floor applied together. Latency-to-quality analysis evaluates response speed in relation to achieved output quality rather than treating either dimension alone. Here, The decision requires acceptable quality while respecting a strict interactive latency constraint. It directly satisfies strict interactive SLA.
Incorrect Answers
Answer B is incorrect because Select the model with the smallest model identifier string measures or controls something different. Identifier length has no relationship to runtime behavior or output quality. Here, It cannot support the stated trade-off. It does not resolve strict interactive SLA.
Answer C is incorrect because Compare only the models’ training-data cutoff dates as an adjacent validation technique measures or controls something different. Training cutoff can matter for knowledge recency. Here, It does not quantify the latency versus quality trade-off in this workload. It does not resolve strict interactive SLA.
Answer D is incorrect because Select the model with the highest standalone accuracy score measures or controls something different. Accuracy alone ignores the production response-time requirement. Here, A marginally higher score is not sufficient if the model violates the SLA. It does not resolve strict interactive SLA.
Question 10
Two models both satisfy the required response-quality threshold. One consumes 45% fewer input and output tokens for the same successful tasks. Which metric should favor that model? Choose ONE.
- Maximum context-window size regardless of actual use
- Raw response accuracy without any cost normalization
- Number of model aliases configured in the account
- Token efficiency or cost per successful quality-qualified response
Correct Answer: D
Correct Answer
Answer D is correct because Token-efficiency analysis relates consumed inference resources to successful outcomes that already meet quality requirements. For this case, Token efficiency or cost per successful quality-qualified response is appropriate because When quality is equivalent, lower token consumption directly improves cost-performance efficiency. The deciding need is both models meet quality floor.
Incorrect Answers
Answer A is incorrect because Maximum capacity does not indicate how many tokens this workload consumes. That makes Maximum context-window size regardless of actual use useful elsewhere, but The decision concerns realized efficiency for equivalent successful tasks. The unmet requirement is both models meet quality floor.
Answer B is incorrect because Accuracy is important, but both candidates already meet the same required quality threshold. That makes Raw response accuracy without any cost normalization useful elsewhere, but It does not capture the material token-efficiency difference. The unmet requirement is both models meet quality floor.
Answer C is incorrect because Alias count is a deployment-management detail. That makes Number of model aliases configured in the account useful elsewhere, but It has no direct relationship to per-task token efficiency. The unmet requirement is both models meet quality floor.
Question 11
Users can rate generated troubleshooting answers, but a simple thumbs-up count gives little insight into why a response failed. What improvement makes the feedback more useful for model refinement? Choose ONE.
- Store only the total number of thumbs-up events each day as a narrower supporting signal
- Capture the rating with a short failure-reason category and the associated task context
- Increase the model maximum-token setting after each negative rating as an efficiency or response-size measurement
- Disable ratings for responses that contain citations as a separate operational measure
Correct Answer: B
Correct Answer
Answer B is correct because Capture the rating with a short failure-reason category and the associated task context fits the evaluation goal: Structured feedback links a user preference to the scenario and reason, making recurring quality defects analyzable. In the stated scenario, It produces actionable labels rather than an undifferentiated approval count. Required condition: thumbs-up alone lacks diagnosis.
Incorrect Answers
Answer A is incorrect because Store only the total number of thumbs-up events each day as a narrower supporting signal is the wrong signal here. Daily totals show aggregate sentiment but discard the task and failure reason. In this case, The team cannot identify what should be changed from that total alone. It misses thumbs-up alone lacks diagnosis.
Answer C is incorrect because Increase the model maximum-token setting after each negative rating as an efficiency or response-size measurement is the wrong signal here. Token budget is unrelated to many causes of dissatisfaction. In this case, Automatically changing it would confound the evaluation without diagnosing the reason. It misses thumbs-up alone lacks diagnosis.
Answer D is incorrect because Disable ratings for responses that contain citations as a separate operational measure is the wrong signal here. Citations do not make user feedback unnecessary. In this case, Suppressing a class of ratings can bias the evidence and hide quality problems. It misses thumbs-up alone lacks diagnosis.
Question 12
A domain-specific contract assistant produces nuanced answers that require legal interpretation. The team needs labels for whether each answer is acceptable and why. Which mechanism is most appropriate? Choose ONE.
- Use token count as the acceptance label
- Treat every response not blocked by a guardrail as acceptable
- Use a subject-matter-expert annotation workflow with a defined rubric
- Infer acceptability only from average response latency
Correct Answer: C
Correct Answer
Answer C is correct because Use Use a subject-matter-expert annotation workflow with a defined rubric. Annotation workflows let qualified reviewers apply consistent labels and rationale using explicit evaluation criteria. Applied here, The legal interpretation requires domain expertise that a generic operational metric cannot substitute for. This addresses legal interpretation requires expertise.
Incorrect Answers
Answer A is incorrect because Use token count as the acceptance label can support another concern. Token count measures length or cost exposure. For the stated evaluation, It does not capture legal interpretation quality. That fails legal interpretation requires expertise.
Answer B is incorrect because Treat every response not blocked by a guardrail as acceptable can support another concern. Guardrails enforce configured safety policies. For the stated evaluation, Passing a safety filter does not prove a contract interpretation is legally acceptable. That fails legal interpretation requires expertise.
Answer D is incorrect because Infer acceptability only from average response latency can support another concern. Latency measures performance, not legal correctness or adequacy. For the stated evaluation, A fast response can still be unacceptable. That fails legal interpretation requires expertise.
Question 13
A team samples only responses that users explicitly complain about and concludes the model is failing on 40% of traffic. What should it change to make the user-centered estimate more representative? Choose ONE.
- Increase the canary traffic percentage until complaints decline as a separate operational measure
- Keep only complaints because they contain the strongest negative signal as an adjacent validation technique
- Remove all negative ratings from the evaluation set as a narrower supporting signal
- Sample across the broader request population while retaining complaint signals as a separate segment
Correct Answer: D
Correct Answer
Answer D is correct because Sample across the broader request population while retaining complaint signals as a separate segment is the relevant measure or method. Representative sampling prevents the evaluation set from being dominated by users who self-selected into complaint reporting. Specifically, The current complaint-only dataset cannot estimate failure prevalence across all traffic. It meets complaint reporters self-select.
Incorrect Answers
Answer A is incorrect because Increase the canary traffic percentage until complaints decline as a separate operational measure is an adjacent but insufficient choice. Changing traffic allocation does not correct biased sampling of evaluation evidence. Under these conditions, The estimate remains based on a self-selected subset. The key need remains complaint reporters self-select.
Answer B is incorrect because Keep only complaints because they contain the strongest negative signal as an adjacent validation technique is an adjacent but insufficient choice. Complaint data is useful for defect discovery but is selection-biased. Under these conditions, Using it alone exaggerates prevalence when estimating population-level quality. The key need remains complaint reporters self-select.
Answer C is incorrect because Remove all negative ratings from the evaluation set as a narrower supporting signal is an adjacent but insufficient choice. Discarding negative evidence would bias the result in the opposite direction. Under these conditions, It would hide real failures rather than make the sample representative. The key need remains complaint reporters self-select.
Question 14
A prompt revision passes ad hoc spot checks, but the team wants to detect whether known previously solved cases regress. Which test design is best? Choose ONE.
- Test only newly invented examples after each change as an adjacent validation technique
- Run a fixed regression suite containing representative and previously failing cases against the old and new prompt versions
- Raise temperature so the tests explore more outputs by changing generation randomness rather than evaluation design
- Evaluate only the prompt-template file size as a separate operational measure
Correct Answer: B
Correct Answer
Answer B is correct because Run a fixed regression suite containing representative and previously failing cases against the old and new prompt versions. A regression suite preserves known expectations and reruns them consistently after changes. Here, Including previously failing cases verifies that fixes remain intact while comparing the candidate with the baseline. It directly satisfies detect recurrence of solved defects.
Incorrect Answers
Answer A is incorrect because Test only newly invented examples after each change as an adjacent validation technique measures or controls something different. New examples expand coverage but do not prove previously fixed behavior remains correct. Here, Known regressions can reappear unnoticed. It does not resolve detect recurrence of solved defects.
Answer C is incorrect because Raise temperature so the tests explore more outputs by changing generation randomness rather than evaluation design measures or controls something different. Higher sampling variability does not create a controlled regression baseline. Here, It makes comparisons harder and does not encode expected results. It does not resolve detect recurrence of solved defects.
Answer D is incorrect because Evaluate only the prompt-template file size as a separate operational measure measures or controls something different. File size says nothing about output behavior. Here, It cannot detect a return of known quality defects. It does not resolve detect recurrence of solved defects.
Question 15
A deployment pipeline should block a new model configuration when its representative evaluation score falls below the approved minimum. Which control should implement this requirement? Choose ONE.
- An automated quality gate that fails the deployment when the evaluation threshold is missed
- A larger context window on every candidate model as a model-capacity characteristic rather than an outcome metric
- A monthly manual review after all candidates reach production as a different evaluation dimension
- A dashboard that displays the score but never affects deployment as visualization without enforcing the required gate
Correct Answer: A
Correct Answer
Answer A is correct because A deployment quality gate converts evaluation results into an enforceable release decision. For this case, An automated quality gate that fails the deployment when the evaluation threshold is missed is appropriate because It prevents promotion whenever the measured score is below the approved minimum. The deciding need is pipeline must block release.
Incorrect Answers
Answer B is incorrect because Context capacity is a model characteristic, not a release decision mechanism. That makes A larger context window on every candidate model as a model-capacity characteristic rather than an outcome metric useful elsewhere, but It cannot block a candidate based on measured evaluation quality. The unmet requirement is pipeline must block release.
Answer C is incorrect because Post-deployment review occurs too late for the stated pre-release block. That makes A monthly manual review after all candidates reach production as a different evaluation dimension useful elsewhere, but It does not provide an automated deployment gate. The unmet requirement is pipeline must block release.
Answer D is incorrect because Dashboards improve visibility but do not enforce the threshold. That makes A dashboard that displays the score but never affects deployment as visualization without enforcing the required gate useful elsewhere, but A failing candidate could still be promoted. The unmet requirement is pipeline must block release.
Question 16
A team needs to score thousands of generated answers against a detailed rubric and wants an explanation for each score. Human review of every answer is too slow. Which evaluation technique best fits? Choose ONE.
- Select a model solely from its public benchmark ranking as external reference evidence rather than workload-specific validation
- Measure only output token count and infer answer quality from length as an efficiency or response-size measurement
- Use an LLM-as-a-judge evaluation with the defined rubric, then calibrate it on a human-reviewed sample
- Use a canary deployment without any response scoring as a different evaluation dimension
Correct Answer: C
Correct Answer
Answer C is correct because Use an LLM-as-a-judge evaluation with the defined rubric, then calibrate it on a human-reviewed sample fits the evaluation goal: A judge model can score large response sets and explain its assessments using specified criteria. In the stated scenario, Human calibration checks whether the scalable judge aligns with the intended rubric before its scores are trusted broadly. Required condition: thousands of answers.
Incorrect Answers
Answer A is incorrect because Select a model solely from its public benchmark ranking as external reference evidence rather than workload-specific validation is the wrong signal here. A public ranking is not an evaluation of the team’s own answers under its rubric. In this case, It does not produce per-response scores or explanations. It misses thousands of answers.
Answer B is incorrect because Measure only output token count and infer answer quality from length as an efficiency or response-size measurement is the wrong signal here. Length is easy to compute but weakly related to correctness or usefulness. In this case, It cannot apply the detailed evaluation rubric. It misses thousands of answers.
Answer D is incorrect because Use a canary deployment without any response scoring as a different evaluation dimension is the wrong signal here. Canary routing limits rollout risk but does not itself score thousands of answers. In this case, The need is scalable rubric-based assessment. It misses thousands of answers.
Question 17
A high-stakes policy assistant has subtle acceptable-answer boundaries that subject-matter experts often debate. Which evaluation method should remain part of the final acceptance process even if automated scores are available? Choose ONE.
- Only the average model response latency while measuring runtime performance rather than semantic correctness
- Human evaluation by qualified reviewers using an explicit acceptance rubric
- Only an automated word-count threshold as a separate operational measure
- Only the number of retrieved passages per request
Correct Answer: B
Correct Answer
Answer B is correct because Use Human evaluation by qualified reviewers using an explicit acceptance rubric. Human evaluation captures domain judgment and nuanced boundaries that automated metrics may not reliably represent. Applied here, For high-stakes ambiguous policy decisions, qualified reviewers provide the necessary final perspective. This addresses subtle boundaries.
Incorrect Answers
Answer A is incorrect because Only the average model response latency while measuring runtime performance rather than semantic correctness can support another concern. Latency is operationally useful but does not decide whether subtle policy content is acceptable. For the stated evaluation, It cannot replace expert judgment. That fails subtle boundaries.
Answer C is incorrect because Only an automated word-count threshold as a separate operational measure can support another concern. Word count can enforce formatting limits. For the stated evaluation, It cannot resolve nuanced policy correctness. That fails subtle boundaries.
Answer D is incorrect because Only the number of retrieved passages per request can support another concern. Retrieval count says how much context was returned, not whether the final policy answer is acceptable. For the stated evaluation, It is insufficient for high-stakes semantic review. That fails subtle boundaries.
Question 18
An automated judge is being introduced for response quality. Before using its scores as a deployment gate, what validation is most important? Choose ONE.
- Delete low-scoring examples from the judge dataset as a separate operational measure
- Use only one trivial example because the judge is another LLM
- Increase the generator model temperature until judge scores spread out
- Compare judge scores with a human-reviewed calibration set and investigate systematic disagreement
Correct Answer: D
Correct Answer
Answer D is correct because Compare judge scores with a human-reviewed calibration set and investigate systematic disagreement is the relevant measure or method. Calibration against trusted human labels reveals whether the automated evaluator applies the intended criteria consistently. Specifically, Systematic disagreement must be understood before judge scores can control deployments. It meets judge will control deployments.
Incorrect Answers
Answer A is incorrect because Delete low-scoring examples from the judge dataset as a separate operational measure is an adjacent but insufficient choice. Removing difficult examples can hide evaluator weaknesses. Under these conditions, It biases validation rather than measuring agreement. The key need remains judge will control deployments.
Answer B is incorrect because Use only one trivial example because the judge is another LLM is an adjacent but insufficient choice. LLM judges can have biases and failure modes. Under these conditions, A single trivial example cannot establish reliable alignment. The key need remains judge will control deployments.
Answer C is incorrect because Increase the generator model temperature until judge scores spread out is an adjacent but insufficient choice. Generator sampling does not validate the evaluator’s alignment with the rubric. Under these conditions, It changes the evaluated output distribution instead of checking judge reliability. The key need remains judge will control deployments.
Question 19
A RAG system returns poor answers. The team wants to determine whether the retriever itself fails to surface relevant evidence before evaluating generation. Which test should it run first? Choose ONE.
- A model-context-window size comparison as a narrower supporting signal
- A fluency evaluation of the final generated answers only as language-form quality rather than the stated decision criterion
- A retrieve-only evaluation against queries with expected relevant context or ground truth
- A canary rollout of a new generator with the same retriever
Correct Answer: C
Correct Answer
Answer C is correct because A retrieve-only evaluation against queries with expected relevant context or ground truth. Retrieve-only evaluation isolates retrieval performance without mixing in generator behavior. Here, It shows whether the correct supporting context is being found before response generation is assessed. It directly satisfies poor RAG answers.
Incorrect Answers
Answer A is incorrect because A model-context-window size comparison as a narrower supporting signal measures or controls something different. Context capacity can affect how much evidence can be passed. Here, It does not directly score whether the retriever finds the relevant evidence. It does not resolve poor RAG answers.
Answer B is incorrect because A fluency evaluation of the final generated answers only as language-form quality rather than the stated decision criterion measures or controls something different. Fluency measures language quality at the response stage. Here, It cannot isolate whether poor evidence retrieval caused the answer failure. It does not resolve poor RAG answers.
Answer D is incorrect because A canary rollout of a new generator with the same retriever measures or controls something different. Changing the generator can alter final answers but leaves the suspected retriever unmeasured. Here, The team first needs retrieval-specific evidence. It does not resolve poor RAG answers.
Question 20
A retrieval stack has good relevance scores but users still experience slow responses. The team wants to know whether retrieval contributes materially to end-to-end delay. Which measurement should it add? Choose ONE.
- Retrieval latency measured separately from generation latency
- Only the final answer’s fluency score
- Only the count of prompt variables as template substitution rather than evaluation execution
- Only the human preference score between two answer styles
Correct Answer: A
Correct Answer
Answer A is correct because Separating stage latency shows how much time retrieval contributes before the model generates a response. For this case, Retrieval latency measured separately from generation latency is appropriate because That lets the team preserve relevance while diagnosing the performance bottleneck. The deciding need is relevance already good.
Incorrect Answers
Answer B is incorrect because Fluency says nothing about where latency occurs. That makes Only the final answer’s fluency score useful elsewhere, but A fluent response can still have a slow retrieval stage. The unmet requirement is relevance already good.
Answer C is incorrect because Prompt-variable count is unrelated to retrieval duration in most architectures. That makes Only the count of prompt variables as template substitution rather than evaluation execution useful elsewhere, but It cannot quantify stage latency. The unmet requirement is relevance already good.
Answer D is incorrect because Preference can assess user-perceived response quality. That makes Only the human preference score between two answer styles useful elsewhere, but It does not isolate retrieval time from generation time. The unmet requirement is relevance already good.
