Generative AI on Databricks<\/a>, scorers are the measurement layer that connects experiments to live behavior. They help answer whether a prompt change, model route, retrieval update, tool change, or application release actually improved the criteria the product cares about.<\/p>\nCurrent MLflow 3 offers built-in LLM judges, custom LLM judges, code-based scorers, and integrations with third-party scorer libraries.<\/p>\n
Built-in judges cover common GenAI quality dimensions<\/h3>\n Databricks provides built-in judges such as RelevanceToQuery, RetrievalRelevance, Safety, and other quality-oriented scorers for common evaluation needs.<\/p>\n
Groundedness, correctness, and context-related judges can be added where the evaluation data includes the evidence or expected answer they require.<\/p>\n
Built-ins are the fastest place to start because they implement common scoring logic and integrate directly with MLflow evaluation results.<\/p>\n
Correctness needs ground truth<\/h3>\n A correctness judge compares the application response with expected facts or an expected response. It is useful when the evaluation set has a known answer or required fact set.<\/p>\n
It should not be used as a magical factuality detector for open-world questions with no reference. In those cases, groundedness or task-specific evidence may be more appropriate.<\/p>\n
Evaluation quality depends on the quality of the expected answer supplied to the judge.<\/p>\n
Relevance and retrieval relevance answer different questions<\/h3>\n Response relevance asks whether the answer addresses the user\u2019s request. Retrieval relevance asks whether the context or chunks retrieved for the request are themselves useful.<\/p>\n
A RAG application can fail either layer. Good chunks can be ignored by the model, or the model can answer fluently from weak context.<\/p>\n
Separate scorers make it easier to fix the correct component.<\/p>\n
Groundedness focuses on support from retrieved evidence<\/h3>\n Groundedness evaluates whether the answer is supported by the context available to the application. This is particularly important for enterprise RAG where the product promises answers based on governed internal sources.<\/p>\n
A grounded answer can still be incomplete, irrelevant, or unsafe, so groundedness should be combined with other scorers rather than treated as a single overall quality score.<\/p>\n
The chunking and retrieval pipeline should be examined when groundedness falls after a search change.<\/p>\n
Safety can be evaluated independently from gateway guardrails<\/h3>\n An MLflow Safety scorer measures output quality after or during evaluation. Unity Gateway guardrails can block or redact interactions at runtime.<\/p>\n
These are complementary. The scorer tells the team how safe application behavior is across an evaluation set; the guardrail enforces a runtime policy on each request or response.<\/p>\n
Evaluation can reveal false negatives or unsafe tendencies before the runtime policy becomes the only protection.<\/p>\n
Custom LLM judges add domain-specific qualitative criteria<\/h3>\n When built-in scorers do not capture the business requirement, a custom LLM judge can evaluate criteria such as tone, completeness, regulatory phrasing, citation quality, or domain-specific correctness guidelines.<\/p>\n
The rubric should be explicit enough that the judge can make repeatable decisions across the evaluation set.<\/p>\n
Custom judges should themselves be validated against human-reviewed examples before their scores become a release gate.<\/p>\n
Code-based scorers are best for deterministic rules<\/h3>\n MLflow supports Python scorers for checks that do not need another LLM. Examples include exact match, JSON schema validation, regex rules, numeric ranges, latency thresholds, citation-count requirements, tool-sequence checks, or business invariants.<\/p>\n
Deterministic logic should stay deterministic where possible. Using an LLM judge to check whether a JSON field exists is slower, more expensive, and less reliable than parsing the JSON directly.<\/p>\n
Code-based scorers can still call judges selectively when a hybrid rule needs both deterministic and qualitative assessment.<\/p>\n
Third-party scorers can join the same evaluation workflow<\/h3>\n MLflow integrates with external evaluation frameworks such as DeepEval and Guardrails AI so specialized metrics can be used alongside built-in scorers.<\/p>\n
This is useful when an organization already trusts a particular evaluator or needs niche metrics not covered by Databricks built-ins.<\/p>\n
The platform should avoid collecting redundant scorers merely because integrations exist. Each scorer should correspond to a quality question the team intends to act on.<\/p>\n
Production monitoring samples live traces with the same scorers<\/h3>\n MLflow production monitoring can register scorers against an experiment and apply them to a configurable sample of live traces. Feedback is attached to the evaluated trace, allowing quality trends to be monitored over time.<\/p>\n
Sampling controls cost. High-volume systems can evaluate a fraction of traffic while increasing coverage for high-risk features or after a release.<\/p>\n
The key advantage is continuity: the scorer that accepted the release can watch the same criterion in production.<\/p>\n
Scorer lifecycle should be governed like application logic<\/h3>\n Production scorers can be registered, started, stopped, updated, and managed through an explicit lifecycle. If a scorer definition changes, historical and current scores may no longer be directly comparable.<\/p>\n
Version the scorer, record the model\/judge if one is used, and keep test examples for the scorer itself.<\/p>\n
A quality metric is only useful when the team knows what it meant at the time it was produced.<\/p>\n
Scorers create value when poor scores lead to engineering action<\/h3>\n Evaluation should connect each scorer to a likely remediation path: retrieval relevance \u2192 search\/chunking, groundedness \u2192 context\/evidence, correctness \u2192 model\/prompt\/data, safety \u2192 policy and model behavior, schema validity \u2192 application output controls.<\/p>\n
Production failures can be curated into the evaluation dataset so the next release is tested against real weaknesses.<\/p>\n
That feedback loop\u2014trace, score, curate, fix, reevaluate, monitor\u2014is the core reason MLflow scorers matter for LLM applications.<\/p>\n
Evaluation datasets should be versioned alongside scorers. A score increase can come from a better application, an easier dataset, or a changed evaluator. Keeping dataset and scorer versions stable makes release comparisons interpretable.<\/p>\n
Human feedback can be used to calibrate judges. Sample scorer disagreements and ask domain reviewers whether the judge is too strict, too lenient, or misunderstanding specialized terminology. Scorer quality matters because a bad evaluator can push the application in the wrong direction.<\/p>\n
Multi-turn conversations need conversation-level criteria in addition to single-turn response checks. A support agent can answer every individual message reasonably while still forgetting prior commitments or failing to resolve the user\u2019s goal by the end of the session.<\/p>\n
Tool-using agents also need action-level evaluation. Code-based scorers can check whether required tools were called, forbidden tools were avoided, or the final state matches the business outcome rather than scoring only the natural-language answer.<\/p>\n
Latency and cost can be scorers too when release gates include operational quality. A model that improves correctness slightly but doubles p95 latency or evaluator cost may not be the right production choice.<\/p>\n
Production sampling should be stratified where risk differs. High-value transactions, new releases, unusual tenants, or fallback routes may deserve a higher scorer sample rate than routine traffic.<\/p>\n
Scorer dashboards should avoid collapsing everything into one composite number. Separate dimensions make remediation actionable. A release with excellent relevance and poor safety requires a different response from one with strong safety but weak retrieval relevance.<\/p>\n
The strongest MLflow evaluation loop treats scorers as executable product requirements. They define what \u201cgood\u201d means, apply that definition before release, and keep checking the same definition after real users arrive.<\/p>\n
Scorers should have clear pass thresholds only where the metric supports one. A relevance judge can produce nuanced feedback, and forcing every score into one binary gate may hide useful trade-offs.<\/p>\n
Release gates can combine several scorers with hard and soft requirements. Safety or schema validity may be non-negotiable, while relevance and latency may have target ranges that allow engineering judgment.<\/p>\n
Scorer prompts and judge models can themselves change. For LLM-based evaluation, record the judge model and scorer implementation so historical score trends are not compared as though the evaluator were constant when it was not.<\/p>\n
Ground-truth maintenance is ongoing. Expected answers become stale when product policy, source data, or business rules change. Evaluation data should have owners and effective dates just like the application knowledge base.<\/p>\n
Production monitoring should feed incident triage, not only dashboards. A sudden drop in groundedness after an index sync should alert the retrieval owner; a safety regression after a model route change should alert the model-platform owner.<\/p>\n
Evaluation cost should be monitored because LLM judges can be expensive at large scale. Use deterministic scorers where they fit, sample production traces, and reserve heavyweight judges for criteria that genuinely require model reasoning.<\/p>\n
Scorer disagreement can be informative. When two judges or a judge and human reviewer disagree consistently, the issue may be the rubric rather than the application.<\/p>\n
Scorers should remain close to product requirements so teams can explain each release gate to stakeholders without translating a pile of abstract AI metrics after the fact.<\/p>\n
Keep the scorer set small enough that every metric has an owner, threshold or interpretation, and a clear remediation path.<\/p>\n
That discipline keeps evaluation useful as the application, model portfolio, and production traffic evolve.<\/p>\n
Teams should also connect scorer output to release gates and incident review. When a score falls, the next question is whether the cause is the prompt, model, retrieval layer, tool behavior, or evaluation set, because each failure requires a different engineering response.<\/p>\n","protected":false},"excerpt":{"rendered":"
MLflow scorers are the evaluation interface for GenAI applications, agents, and RAG systems in MLflow 3. A scorer turns an application trace or input\/output pair into feedback such as pass\/fail, true\/false, a numerical score, a category, or structured assessment. The same scorer can be used during development evaluation and production monitoring, which gives teams one […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-19813","post","type-post","status-publish","format-standard","hentry","category-general"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t