Feature-store design is about keeping model inputs consistent across training and serving. For Generative AI Leader, that means preserving feature definition, entity identity, time semantics, and freshness across the complete model path. In Google Cloud’s current architecture, BigQuery is the offline feature source while Agent Platform Feature Store manages feature views for online serving; a feature view must be synchronized before it can serve online feature values. For new deployments, Google recommends Bigtable online serving; Optimized online serving has been deprecated since May 17, 2026 and is scheduled to sunset on February 17, 2027.
Within Google Cloud AI, Feature Store sits between data engineering and model serving. Its job is not merely to make lookup fast; it is to make the feature definition, entity key, freshness, and point-in-time meaning stable enough that training and production inference refer to the same concept.
The hardest failures are usually not outages. They are silent mismatches between what the model learned during training and what the online application provides later.
This article focuses on the design contract of features across training and online serving—not on a general overview of every Google Cloud AI service. Current Google documentation calls the capability Agent Platform Feature Store on Gemini Enterprise Agent Platform; the historical Vertex AI Feature Store terminology still appears in older tutorials, so the intended product version must be verified for every deployment.
BigQuery feature sources need reproducible definitions
Google Cloud’s feature-serving guidance starts with feature data stored in BigQuery tables. This gives offline training and analysis a durable SQL-accessible source and avoids maintaining an unrelated offline feature store.
BigQuery also makes it practical to derive features with scheduled queries, Dataform, pipelines, or existing analytical transformations before those features are synchronized for online serving.
The feature table should have a clear owner and data contract because every downstream model that depends on it inherits its semantics.
Feature computation should be reproducible. If a feature is generated by a SQL transformation, pipeline, or UDF, keep that logic in source control and link it to the feature definition. Manually edited BigQuery tables undermine the purpose of a shared feature layer because nobody can reconstruct how the current value was derived.
Point-in-time correctness depends on event-time semantics and backfill discipline
Time-sensitive features must represent what would have been known at the historical prediction time. BigQuery provides functions such as ML.FEATURES_AT_TIME and ML.ENTITY_FEATURES_AT_TIME to support point-in-time lookups. Without temporal correctness, a training query can accidentally use a future account status, later purchase, or post-event label information that would not have existed when the prediction should have been made.
This type of leakage can produce excellent offline metrics and disappointing production performance, so time semantics belong in the feature contract. For time-sensitive features, the timestamp should reflect when the feature value became true or valid in the business process. Using only warehouse-load time can distort point-in-time training when pipelines are delayed or backfilled.
Separating event time from ingestion time also makes freshness monitoring clearer. A row loaded today may represent an event that occurred yesterday. The feature contract should document which timestamp is used for historical correctness and which timestamp is used for pipeline operations.
Feature backfills should preserve history rather than overwrite it when point-in-time training matters. Recomputing a feature with today’s logic over old events can change historical values and make old model experiments difficult to reproduce unless the transformation version is also preserved.
Feature views should synchronize only what online models actually need
Agent Platform Feature Store serves feature values from feature views. A view identifies the BigQuery-backed feature data and the entity identifier used for online retrieval. The online store does not need every feature in the warehouse. It should synchronize only the features required by latency-sensitive production models.
A small, purpose-built online feature surface is easier to govern and cheaper to keep fresh than replicating every analytical column.
Cost review should identify features whose online serving is rarely used. BigQuery may remain the correct offline source while an expensive online copy provides little value. Usage metrics can support decommissioning or slower sync for low-demand features.
Sync state and capacity determine freshness at serving time
A feature view must be synchronized into the online store before it can serve values. That creates an operational dependency between BigQuery data freshness and the online store’s sync state. Monitoring should track last successful sync, sync duration, failed rows, and source freshness so the application can distinguish “feature value is old” from “feature is legitimately unchanged.”
If stale features create unacceptable model risk, the serving application needs a fallback or fail-closed strategy rather than using old values silently.
Freshness requirements should be feature-specific. A user’s account balance may need near-real-time updates, while a seven-day activity count can tolerate a longer sync interval. One global online-store cadence can be either unnecessarily expensive or too stale for the most critical features.
Online-store capacity should be tested at production read QPS and entity distribution. Hot entities can produce a different load profile from uniformly distributed benchmark keys, and one popular entity should not surprise the platform with a concentrated traffic pattern.
Fallbacks and SLOs are part of the online feature contract
Fallback behavior needs product agreement. If an online feature lookup times out, the application might use a default, use a stale cached value, call a slower source, or fail the prediction. Each choice changes model behavior and should be tested, not improvised during the first outage.
Service-level objectives should cover both freshness and lookup latency. A feature store can be fast while serving yesterday’s value, or perfectly fresh while too slow for the online model path. Both dimensions need independent monitoring and an agreed threshold.
Publish those SLOs with the feature view so model owners know what serving guarantees they can rely on. That makes freshness and latency expectations part of the model contract instead of informal assumptions held by one team.
Entity IDs, grain, and permissions are correctness boundaries
Online serving retrieves feature values for entity identifiers such as customer, account, device, or product IDs. The application should obtain those IDs from trusted business context rather than accepting arbitrary client-supplied identifiers. A feature store can return the correct value for the wrong entity if the caller is allowed to query it. Authorization and tenancy must therefore be enforced around the online lookup.
Entity definitions should also remain stable across training, serving, and evaluation so one model does not silently reinterpret the same key.
Feature definitions should specify grain explicitly. A value can be per user, per user-product pair, per device, or per account-day. Ambiguous grain creates accidental many-to-one joins during training and wrong lookups during serving even when the column name sounds clear.
Feature permissions should follow least privilege. Analysts who can explore offline BigQuery tables do not necessarily need the right to serve every feature online, and applications should receive only the feature views required by their models.
Online and offline skew requires validation and model monitoring
Even when one BigQuery source is used, skew can appear because online synchronization cadence, preprocessing, missing-value behavior, or model version differs from the training workflow. Model drift becomes part of feature-store QA because feature distributions, missingness, and freshness can diverge between historical training data and online requests even when the serving API itself remains healthy. Skew is a data-contract problem as much as a model problem.
Feature validation should happen at both creation and serving time. Range checks, enum membership, null-rate expectations, and freshness thresholds can catch upstream data changes before they silently shift model behavior.
Online/offline parity should be tested with sampled entities. Retrieve the historical or current value from BigQuery and compare it with the online store for the same entity and timestamp expectations. This catches sync and transformation mismatches that dashboards may not reveal.
Feature ownership includes reuse, lineage, privacy, and retirement
A feature can outlive the team that created it. The catalog should record definition, source table, owner, freshness expectation, consumers, and deprecation policy. If a feature changes meaning, downstream models should not be forced to discover that through degraded accuracy. Versioned names or explicit migration periods may be safer than silently redefining the column.
Lineage makes it possible to identify which models and online views depend on a source transformation before that transformation changes.
Feature reuse should be measured, not assumed. Central feature catalogs can accumulate hundreds of nearly identical columns created by different teams. Before publishing a new feature, search for existing definitions and compare semantics, freshness, and entity grain. Reuse is valuable only when the meaning truly matches.
Privacy and retention should follow the source data. A derived feature can still reveal sensitive behavior even if it no longer contains the original raw event. Access controls, deletion processes, and retention periods should cover the feature layer and synchronized online copies.
Model retirement should trigger feature-consumer review. If the last model using a low-latency feature is retired, the online view may no longer need to be synchronized. Removing unused serving paths reduces cost and eliminates stale data products that future teams might mistakenly depend on.
Feature Store succeeds when meaning survives every storage layer
The objective is not to centralize every column. It is to make high-value model features reusable, temporally correct, low-latency when needed, and governed across training and serving. Feature Store materializes selected BigQuery-backed features for online serving, so the operational contract includes synchronization, lookup latency, entity semantics, and access control as well as the SQL definition of the feature.
Missing values need a documented semantic. “No row exists,” “source has not arrived,” “user has no history,” and “pipeline failed” are different situations. The model and application should not collapse them into the same default value without considering the impact on predictions.