Delta Lake Change Data Feed (CDF) exposes row-level inserts, updates, and deletes between table versions so downstream systems can process changes incrementally instead of re-reading an entire table. In current Databricks releases, there are two distinct paths: automatic change data feed for qualifying Unity Catalog tables on newer runtimes, and the older legacy CDF mode that materializes change records during writes and must be enabled per Delta table.
Within Databricks Data Engineering, CDF is a change-propagation boundary. It is useful for incremental ETL, downstream replication, cache synchronization, audit workflows, and any pipeline that needs to distinguish inserts from updates and deletes rather than treating the table as append-only.
The existing Delta Lake fundamentals article provides the transaction-log context that makes versioned change reading possible.
Automatic CDF changes the write-versus-read trade-off
Current Databricks documentation introduces automatic change data feed for qualifying Unity Catalog managed Delta tables with row tracking enabled and for supported Iceberg v3 tables. Instead of materializing every change record during the write, Databricks can compute changes during reads using row-lineage metadata.
This reduces some write overhead and storage cost compared with legacy CDF, especially for updates and merges that previously generated dedicated change files.
The feature requires newer Databricks Runtime versions and compatible table types, so migration should begin with capability checks rather than simply disabling the legacy property.
Legacy CDF remains a table property
The legacy Delta path uses the delta.enableChangeDataFeed=true table property. Changes that occur after the property is enabled become queryable through the CDF APIs.
Turning the feature off creates a gap: changes that occur during the disabled interval are not retroactively captured by the legacy feed. Databricks recommends migrating eligible tables to automatic CDF if the use case benefits from that model.
Teams should record when legacy CDF was enabled so consumers know the earliest reliable version.
Change records include both data and commit metadata
CDF includes table columns plus metadata fields such as _change_type, _commit_version, and _commit_timestamp. Update operations can expose both preimage and postimage rows.
Consumers should use these fields rather than infer operation type by comparing before/after snapshots themselves.
The metadata also makes it possible to checkpoint downstream progress using table version rather than wall-clock time alone.
Batch and streaming consumers use the same underlying feed
Databricks exposes CDF through batch table-change queries and Structured Streaming using readChangeFeed=true. The same change semantics can therefore support one-time backfills and continuous downstream pipelines.
Streaming consumers should persist checkpoint state and use starting versions deliberately. Recreating a consumer from an old version may fail if that history has already aged out of retention.
For periodic catch-up, AvailableNow can process the currently available change feed as a bounded batch-style streaming run.
CDF is transient unless you archive it
Databricks explicitly states that the change data feed is not a permanent audit log. CDF availability depends on table history and retention. Transaction-log cleanup and VACUUM can remove the table versions or change files required for old ranges.
If the business requires a permanent history, stream the change feed into a dedicated append-only archive table with its own retention policy.
Audit obligations should not rely on the default Delta retention window unless that retention has been designed and validated for the requirement.
Schema changes can break version-range reads
Non-additive schema changes such as column renames, column drops, type changes, or nullability changes can limit CDF reads across the affected version range.
Consumers should treat schema evolution and change replay as one design problem. A downstream job that assumes one schema across months of history can fail when the source changed shape in the middle of that range.
Migration plans should define whether the consumer processes pre-change and post-change ranges separately or transforms both into a common downstream schema.
Automatic CDF has additional table restrictions
Current automatic CDF is not supported for every table configuration. Databricks documentation calls out restrictions including tables with row filters or column masks and certain multi-statement transaction scenarios.
This matters for governed tables because the security model can influence the change-propagation technology available.
The platform team should review Unity Catalog policy features together with the desired incremental-processing pattern before standardizing automatic CDF globally.
Rate limits can control downstream catch-up
Streaming reads from CDF support options such as maximum files or bytes per trigger. Databricks applies commit boundaries atomically in the change feed, so a commit is either included entirely in a micro-batch or deferred.
This helps downstream systems control recovery load after a backlog without splitting one logical commit unpredictably.
Rate limiting should be tuned with downstream table and sink capacity, not only with source size.
CDF is preferable to streaming the base table when changes matter
Databricks recommends using change data feed when downstream logic must process inserts, updates, and deletes explicitly. Streaming directly from the Delta table is more appropriate for append-only or carefully constrained patterns.
The base-table stream can skip change commits, but then downstream tables will not receive the actual mutations. CDF is the stronger contract when full change propagation matters.
Later H07 content on Lakeflow Auto CDC extends this idea into higher-level declarative CDC processing.
Change propagation is successful when replay is predictable
A mature CDF design records the source table version processed, downstream checkpoint, schema version, retention assumption, and recovery procedure. Operators should know how far back they can replay and what happens when the requested version has expired.
CDF is powerful because it makes Delta table changes queryable as data. It becomes reliable only when retention, schema evolution, downstream idempotency, and checkpoint ownership are treated as part of the same contract.
Starting-version choice determines whether a downstream consumer begins from a current snapshot or from historical changes. For a new replication target, one pattern is to load a baseline snapshot and then start CDF from the corresponding version. Another is to use streaming semantics that emit the current table state before subsequent changes. The recovery procedure should document exactly which approach the consumer expects.
Update preimages and postimages can double the apparent row volume for update-heavy tables. Consumers that only need the final value should filter or consolidate events appropriately, while audit systems may require both sides of the change. The change type should drive downstream semantics rather than be ignored after ingestion.
Idempotency is essential when applying CDF into external systems. A downstream job can replay a commit after failure, so writes should use commit version, business key, or another deterministic marker to avoid duplicating effects.
Table clones, restores, and rewrites can also affect consumer assumptions about history. A CDF consumer should follow the intended table identity and version lineage rather than assuming any table with the same name has continuous historical change semantics.
Automatic CDF’s dependency on row tracking means table configuration becomes part of the change contract. Platform standards should record which tables are eligible, whether row tracking is enabled, and whether security features such as masks or row filters create an incompatibility.
For audit use cases, archive the CDF into a dedicated immutable or append-only history table with enough metadata to reconstruct ordering. Treating the source table’s transient feed as the audit store creates unnecessary risk around retention and vacuum.
CDF is most valuable when downstream teams can replay deterministically from a known version to a known target state. That requires source retention, schema history, checkpoint ownership, and idempotent application of changes to be designed together.
Downstream consumers should record the source commit version they last applied successfully. Wall-clock timestamps alone can be ambiguous when commits are delayed, replayed, or processed across time zones. Version checkpoints provide a deterministic restart point tied directly to Delta history.
Merge-heavy source tables deserve load testing because update events can produce multiple change rows. The downstream system should be sized for change volume, not only for final table growth. A stable table size can still generate a large CDF during heavy corrections or restatements.
Consumer contracts should specify whether deletes are hard deletes, tombstones, or business-status changes. CDF exposes the source operation, but the destination still needs to decide how that event maps into its own storage and retention model.
Schema version should be stored with archived change records when long-term audit matters. A row image is difficult to interpret years later if the column meaning or type has changed and the consumer only preserved the values.
Downstream materialization should also define ordering across commits. Delta commit version provides a total order for changes in one table, but cross-table business transactions may not share a single commit boundary. Workflows that need cross-table consistency require a higher-level transaction or reconciliation strategy.
When CDF is used for replication, compare source and destination counts or checksums periodically. A streaming checkpoint can continue happily after a logic bug has already caused divergence; reconciliation gives the system a way to detect silent drift.
Retention settings should be reviewed with downstream outage scenarios. If a consumer can be offline for seven days but the source history is retained for less than that, the theoretical replay path will fail during a real incident. Recovery time objectives should therefore inform table-history and vacuum policy.
Consumers should also test schema-change boundaries explicitly. A rename or type change can require splitting a replay into ranges before and after the change. Runbooks should include that procedure so operators do not discover it for the first time during a backlog recovery.
