A Structured Streaming checkpoint preserves the state a query needs to resume processing, including its progress through supported sources and stateful operations. In Databricks, the checkpoint is an operational dependency rather than a disposable temporary folder. Deleting or reusing it blindly can cause duplicate processing, invalid state restoration, or a replay that no longer corresponds to the original sink’s contents.<\/p>\n
Recovery becomes especially difficult when the input schema, query plan, streaming state, or sink transaction contract changes during an outage. A reliable incident procedure starts by identifying what the checkpoint represents, which data remains in the source, and which writes have already committed. Restarting a cluster is only one part of the problem.<\/p>\n
Record the query name, source technology, checkpoint path, target tables, stateful operations, trigger type, runtime version, and ownership. Two queries should not share one checkpoint location. If the checkpoint is moved, copied, or reset, that change must be explicitly planned because it changes which source offsets and state the engine can recover.<\/p>\n
Checkpoint internals can include metadata, offsets, commits, and state-store contents. They are not a general-purpose file format to edit by hand. A missing file or partial checkpoint upload may reflect storage permissions, object-store consistency issues, concurrent writers, or unexpected cleanup policy. Diagnose the failure from query logs before attempting repair.<\/p>\n
Structured Streaming reliability depends on preserving data contracts across processing stages. Distinguish a failed micro-batch whose sink transaction did not commit from one that committed successfully but whose driver lost contact before reporting completion. Those cases have different duplicate-processing implications.<\/p>\n
A streaming query tracks the point it has reached in each supported source. Kafka offsets, cloud file discovery state, and Delta source versions have different retention and replay characteristics. A checkpoint may be intact yet reference input data that the source has already expired or vacuumed. In that situation the problem cannot be solved by simply restarting the query.<\/p>\n
Inspect the most recent completed batch and sink history. A batch might have read records and written output before a failure interrupted acknowledgement. If the sink uses supported transactional semantics, the engine can avoid repeating some effects; external APIs or poorly designed custom sinks may not provide the same protection.<\/p>\n
Define stable business identifiers and idempotent writes for external side effects. An An engineer may replace a source table with a similar table that has the same schema but a different transaction history. Reusing the previous checkpoint can then associate saved offsets with the wrong source identity or state lineage. Source equivalence cannot be inferred from column names alone. Compare the query plan, source identifiers, checkpoint metadata, and intended output contract before approving the restart. If the boundary truly changed, isolate a new stream and reconcile a consistent snapshot. This makes the transition auditable and prevents a superficially healthy process from silently skipping the new source’s early changes.<\/p>\n Some query changes are compatible with a saved checkpoint; others are not. Changes to source identity, state schema, aggregation keys, watermark behavior, or sink configuration can invalidate a straightforward restart. Databricks documents limitations that depend on the query and runtime. A safe release process compares the proposed logical plan with the existing checkpoint before deployment.<\/p>\n If an upgrade fails, do not immediately delete the old checkpoint to force the new code to start. That may replay an entire source, drop state needed for correct aggregation, or advance from a new starting point that skips historical events. First determine whether reverting the code and runtime can restore processing safely from the original checkpoint.<\/p>\n If starting fresh is genuinely required, treat it as a migration. Choose a new checkpoint path, define the initial offset or snapshot boundary, isolate the target output, and reconcile new results before switching consumers. Preserve the original checkpoint according to approved incident-retention rules until the recovery decision is validated.<\/p>\n Stateful aggregations, stream-stream joins, and deduplication depend on saved state beyond a simple source offset. A query that loses that state may produce different results even if it can replay recent source messages. Inspect the operator’s state metrics and the history of watermark progress, state size, and processed rows.<\/p>\n Watermarks manage bounded late data under the query’s event-time semantics. If the recovery starts with a changed watermark setting, some out-of-order events may be handled differently from the original execution. Test with a controlled set of late events and duplicates to establish whether resumed outputs remain consistent with the intended contract.<\/p>\n Changes to the state store or serialization configuration should be planned with runtime compatibility in mind. A corruption alert or inability to restore state is not proof that the data in the source is corrupt. It may be a failed restore of state files or an incompatible application change. Separate source integrity testing from checkpoint and state-store diagnostics.<\/p>\n Source retention limits can make old offsets unavailable. Kafka may delete messages; a Delta source can remove files outside its retention horizon; file-notification services can have their own delivery constraints. Establish the earliest replayable position and compare it with the checkpoint’s expected position before taking corrective action.<\/p>\n Where a gap cannot be replayed, investigate alternate reconstruction sources such as a retained transaction log, an audited snapshot, or a source system export. Rebuilding the target table may be preferable to allowing a resumed stream to skip an unknown interval silently. State explicitly which events were recovered, reconstructed, or lost.<\/p>\n A reliable recovery plan includes source and sink reconciliation. For example, compare the count and keys of orders from an authoritative source window against committed target rows, accounting for updates and deletions rather than assuming one message equals one output row. Capture exceptions and reconcile them before marking the incident closed.<\/p>\n A checkpoint path must remain accessible and writable to the job identity across restarts. Credential rotation, external-location policy changes, or a switch from classic to serverless compute may alter the effective path access. A generic permission-denied message may refer to the checkpoint folder rather than the source table.<\/p>\n Test access using the same principal and compute configuration as production. An interactive administrator notebook that lists checkpoint objects successfully does not prove the scheduled job’s identity can read and atomically update them. Diagnose path ownership, cloud IAM, Unity Catalog governance, and network accessibility as separate layers.<\/p>\n Protect checkpoint data from housekeeping scripts and broad VACUUM or lifecycle rules. Checkpoints are operational state whose retention must align with the maximum expected outage and recovery window. Accidental cleanup can transform a routine cluster restart into a historical reprocessing project.<\/p>\n For a sessionized clickstream, include several events that arrive after the watermark and others that arrive just inside the accepted delay window. During a recovery rehearsal, verify both the final aggregation and the handling of late events, not merely the total number of processed rows. Preserve representative state-store metrics before and after the restart. If a job has a stable input count but produces a different number of sessions after changing state configuration, the code and saved state may no longer represent the same analytical contract.<\/p>\n When recovery requires a new query or target, use isolated output first. Process a bounded test interval, compare counts and business keys, and inspect duplicate and late-event outcomes. Verify the same transformation code and schema version that production will use. A replay that only produces plausible aggregate totals may still miss individual updates.<\/p>\n A recovery rehearsal should stop one test stream after records have been read but before the sink has acknowledged a durable write. At restart, compare source offsets, streaming batch identifiers, and target transaction records to determine whether reprocessing is harmless or creates duplicate business effects. Repeat this test with a stale checkpoint copy and a delayed source event. A clean first-run checkpoint does not exercise the failure path that an on-call team actually needs to recover.<\/p>\n Test a crash between a sink write and progress commit where feasible. For a Delta sink, understand the platform’s transactional handling. For an external service called through foreachBatch<\/code> function that sends messages or modifies a third-party service may repeat a batch during recovery. The application should use batch IDs and record identifiers carefully, with its own durable record of completed effects, rather than assuming an exactly-once guarantee extends to every external action.<\/p>\nDecide whether checkpoint reuse is valid<\/h3>\n
Diagnose stateful operator recovery<\/h3>\n
Recover safely after input retention gaps<\/h3>\n
Validate the checkpoint’s storage permissions<\/h3>\n
Run controlled replay and cutover tests<\/h3>\n
foreachBatch<\/code>, confirm idempotency under intentional batch re-execution. This is a more meaningful test than restarting an idle stream and seeing it report Active.<\/p>\n