Model monitoring is easy to reduce to a chart labeled \u201cdrift,\u201d but production monitoring is really a decision system. A team chooses a baseline, chooses the production data to compare against it, selects a distance metric, defines thresholds, and then decides what an alert should cause humans or automation to do. Google\u2019s current Model Monitoring v2 documentation keeps that structure explicit: a model monitor holds reusable configuration, while monitoring jobs run on demand or on a schedule against selected baseline and target data.<\/p>\n
The planned term Vertex AI Model Monitoring<\/strong> remains recognizable, although Google\u2019s 2026 platform naming moved much of Vertex AI under Gemini Enterprise Agent Platform. For an AI on Google Cloud<\/a> design, the practical point is that model monitoring is an operational control for data and model behavior, not a substitute for application observability. Candidates preparing for the Generative AI Leader<\/a> exam should distinguish statistical drift in model inputs or outputs from generative-AI quality failures such as unsupported answers, tool misuse, retrieval misses, or unsafe content.<\/p>\n A drift score has meaning only relative to something. The baseline can be training data, a known-good production period, or another reference dataset that represents expected behavior. The target is the data being evaluated now. Model Monitoring v2 lets teams choose whole datasets or time windows and can compare a recent window against an earlier offset window. That flexibility is useful, but it also means two teams can monitor the same model and reach different conclusions because they chose different reference populations.<\/p>\n The design problem is explored more broadly in model monitoring and drift<\/a>. Before setting thresholds, identify what operational change the metric is meant to detect. If the business legitimately enters a new geography, a feature distribution may shift sharply without the model being broken. Conversely, a stable overall distribution can hide failure for a small but important customer segment. Baseline choice should follow the risk you are trying to surface.<\/p>\n Feature drift tells you that the inputs reaching the model no longer look like the reference inputs. Prediction-output drift tells you that model outputs have changed distribution. Neither one proves causality. Input drift can occur without harming accuracy, and output drift can be an intended response to a legitimate change in the population. Monitoring is therefore evidence that deserves investigation, not an automatic declaration that the model must be retrained.<\/p>\n Google\u2019s current monitoring interfaces support objectives around raw-feature drift, prediction-output drift, and feature-attribution drift for supported tabular monitoring scenarios. Teams can configure metrics such as Jensen-Shannon divergence or L-infinity distance and alert thresholds. Those values should be calibrated with historical data. A threshold chosen because it \u201clooks small\u201d can generate noisy alerts, while a threshold chosen after reviewing known incidents can become a meaningful operating signal.<\/p>\n Two datasets can have similar input distributions while the model relies on those features differently. Feature-attribution monitoring helps surface that possibility by comparing how much different inputs contribute to predictions. This can be useful when a model appears statistically stable at the feature level but begins making decisions for different reasons because relationships between features have changed.<\/p>\n Attribution metrics still require interpretation. A shift may reflect an intentional model update, a changed feature pipeline, or a real change in the domain. It can also point toward a data-quality defect. The operational response should connect the alert to deployment records, feature-pipeline changes, and business events. This is where AI observability<\/a> becomes more than metric collection: operators need enough context to explain what changed before they change the model.<\/p>\n Continuous monitoring is usually implemented as repeated monitoring jobs over a defined window. Small windows react quickly but can produce volatile metrics when sample counts are low. Larger windows are more stable but can dilute a recent failure inside a large body of older traffic. Google\u2019s guidance explicitly notes that inadequate volume can make drift analysis volatile, so window size should reflect traffic patterns rather than a generic hourly or daily cadence.<\/p>\n Seasonality matters as well. Comparing Sunday traffic to a weekday baseline, holiday retail traffic to a normal week, or month-end finance behavior to mid-month activity can create predictable \u201cdrift\u201d that is not a defect. An offset baseline can help compare similar periods, but the monitoring plan should document the business calendar. Alerting that ignores predictable operating cycles trains teams to ignore the monitoring system itself.<\/p>\n Model monitoring requires access to inputs, outputs, or derived statistics that can contain sensitive information. Teams should decide what data is retained, where it is stored, how long it remains available, and who can inspect it. Exporting rich prediction payloads for monitoring can accidentally create a second copy of regulated data with weaker access controls than the primary application.<\/p>\n The model monitor should therefore fit the same project, IAM, encryption, retention, and audit strategy as the application. If a production model uses personal or proprietary features, monitoring is part of that data flow. The broader governance lesson in private data and model access<\/a> applies directly: observability is not exempt from least privilege merely because the data is being collected for reliability.<\/p>\n Traditional model monitoring is strongest when there is a clear schema and measurable feature or prediction distributions. Generative AI can fail while those signals remain stable. A support assistant can keep receiving the same kinds of questions yet begin producing less grounded answers after a prompt change. A retrieval system can return plausible but less relevant evidence. An agent can complete the same number of tasks while selecting riskier tools.<\/p>\nDefine the baseline before you define the alert<\/h3>\n
Input drift and output drift answer different questions<\/h3>\n
Feature-attribution drift can reveal changing decision logic<\/h3>\n
Scheduled jobs need enough data to avoid unstable conclusions<\/h3>\n
Monitoring data must be governed like production data<\/h3>\n
Generative AI requires more than tabular drift metrics<\/h3>\n