Lakeflow Declarative Pipelines can apply data-quality expectations while transforming arriving records into governed tables. Expectations express conditions that data should satisfy, but their action matters as much as their predicate. A policy may record quality metrics, drop invalid records, or fail a pipeline update; those outcomes create different operational and business consequences. Quarantining suspect records requires an explicit design for retaining and reviewing them rather than merely declaring an expectation.<\/p>\n
Strong data-quality engineering separates the source’s observed defect, the pipeline’s chosen reaction, and the downstream consumer’s tolerance. A silently dropped row can preserve a dashboard’s apparent cleanliness while concealing an important financial event. A failed pipeline can protect a contract while blocking unrelated valid updates.<\/p>\n
Begin with the meaning of the field. A non-null customer identifier is a basic structural expectation; an amount expected to be nonnegative may involve business exceptions such as refunds or corrections. Build the predicate from an approved domain rule and identify the data owner authorized to change it. A convenient SQL condition is not necessarily a complete business policy.<\/p>\n
Describe the scope: whether the rule applies before or after deduplication, to raw ingestion or curated data, and whether historical records follow the same constraint as newly arriving events. A condition may be valid for new applications but unsuitable for legacy imported history. Version the contract so consumers can interpret quality metrics consistently over time.<\/p>\n
A useful data-quality pipeline<\/a> catches concrete failure modes rather than asserting generic cleanliness. Distinguish malformed syntax, unknown reference values, duplicate business keys, out-of-range measurements, and records that contradict an earlier accepted state. Each class can require a different operational response.<\/p>\n A warn-style expectation allows records to continue while producing metrics about validity. That can suit monitoring or an early migration phase when the team needs to learn the defect rate. A drop-style action prevents invalid rows from entering the targeted output, but does not by itself preserve them for review. A fail-style expectation halts the affected pipeline update under the supported semantics.<\/p>\n Choose based on consequences. For a financial ledger, dropping a record without traceability may violate reconciliation requirements. For an analytics dashboard, allowing an obviously malformed measurement to affect aggregates can be worse than temporarily excluding it with explicit counts. Define what correctness means for the consuming service, not simply the pipeline job’s green status.<\/p>\n The rules should have escalation thresholds and owners. A sudden five-percent invalid rate after a source deployment is not a harmless metric merely because the pipeline continues running. Monitor the trend, identify the affected producers and schema versions, and trigger a process that either repairs the records or formally changes the expected contract.<\/p>\n Consider a shipping event with a missing order identifier. Rejecting the row protects the curated order table, but operations still need to know which logistics producer emitted it and whether a physical package moved. Attach a stable source reference, producer version, ingestion timestamp, validation code, and replay eligibility to the quarantine record. A support analyst should be able to decide whether the upstream system needs a correction or whether the record represents an invalid duplicate. Without that detail, a quarantine table merely postpones diagnosis while storage accumulates unusable data.<\/p>\n Quarantine usually involves routing the entire suspect record, its source metadata, failure reason, ingestion timestamp, and stable identifier to a separate governed dataset. The pipeline must make that branch explicit. An expectation attached to a curated table cannot be assumed to automatically create a quarantine table without additional logic.<\/p>\n Preserve enough original context to reconstruct the failure while minimizing exposure of sensitive fields. A failed record may include personal data or secrets, so quarantine privileges and retention deserve at least as much care as the production table. Keep the reason for rejection understandable to analysts without duplicating unrestricted raw payloads in log messages.<\/p>\n Plan for multiple simultaneous rule failures. A record may have an invalid currency and missing account identifier; recording only the first predicate can hide the scope of remediation. Whether the pipeline stores an array of failure codes or a precedence-ranked reason, define the interpretation and keep it stable across releases.<\/p>\n A new producer field can break parsing or validation when the receiving pipeline assumes a fixed schema. Distinguish legitimate additive changes from incompatible type changes and accidental field renames. An expectation failure should lead to a review of the schema contract before the team changes a predicate to accept every incoming value.<\/p>\n Late-arriving events can be valid even when timestamps fall outside the usual processing window. Quality conditions based on \u201ccurrent time\u201d must account for replay and backfill. A pipeline that rejects archived legitimate events during recovery can convert an outage into permanent data loss.<\/p>\n For deduplication, define event identity separately from processing time. A replayed message might carry a stable business identifier but a fresh ingestion timestamp. A rule that accepts only unique ingestion times will fail to recognize duplicates. Align quality controls with the intended event semantics and watermark or state retention where applicable.<\/p>\n Track expectation outcomes by pipeline update, source, rule, and relevant domain dimension. Overall pass percentage can mask a concentrated failure in one client application or region. A low global defect rate may still represent every transaction from a newly deployed producer, making a targeted drill-down essential.<\/p>\nChoose warn, drop, or fail behavior deliberately<\/h3>\n
Design a real quarantine flow<\/h3>\n
Handle schema change and delayed arrival<\/h3>\n
Make quality metrics operationally useful<\/h3>\n