Implementing Data Engineering Solutions Using Azure Databricks DP-750 Practice Test 18
Question 170
A platform architect adopts: Specify the stream watermark and an explicit closeout readiness condition before running downstream aggregation. For this requirement: A streaming process depends on a downstream batch closeout but no one defines whether it should wait for the latest watermark. Which rationale MOST directly supports it?
- Orchestration may need error paths that execute regardless of normal success dependencies.
- Task-level repair reduces unnecessary recomputation while preserving dependency correctness.
- Stream progress is not synonymous with completeness for a business reporting interval.
- Declarative pipelines express table dependencies and data quality rules without bespoke notebook orchestration.
Explanation
Correct Answer
C:The decisive mechanism is: Stream progress is not synonymous with completeness for a business reporting interval. That explains the selected action: Specify the stream watermark and an explicit closeout readiness condition before running downstream aggregation. In the original scenario: A streaming process depends on a downstream batch closeout but no one defines whether it should wait for the latest watermark. The specific unmanaged risk is: A downstream report can close before late but permitted events are incorporated. Validate with: A controlled late-event test updates the interval before its approved closeout and leaves later events for correction.
Incorrect Answers
A:This principle concerns a separate data-engineering condition: A source task fails yet the monitoring task never executes because all downstream tasks require upstream success. Its targeted response is: Design a separate failure-handling or completion branch with the supported run-if logic for alerts and cleanup. The current case is governed more directly by: Stream progress is not synonymous with completeness for a business reporting interval.
B:This principle concerns a separate data-engineering condition: A bulk repair executes all pipeline tasks again even when only the final reporting notebook failed. Its targeted response is: Separate tasks and use the supported repair-run workflow at the failed stage when dependencies remain valid. The current case is governed more directly by: Stream progress is not synonymous with completeness for a business reporting interval.
D:This principle concerns a separate data-engineering condition: A continuous feed creates and refreshes tables with consistent declarative dependencies but developers manually chain notebooks. Its targeted response is: Evaluate Lakeflow Spark Declarative Pipelines for the compatible streaming and transformation graph. The current case is governed more directly by: Stream progress is not synonymous with completeness for a business reporting interval.
Question 171
An implementation skips: Store deployment source in Git with reviewed branches and pull requests before promoting changes. Engineers are observing: Engineers edit production notebooks directly and lose the ability to identify which code change broke a pipeline. What consequence is MOST likely?
- Manual post-deployment edits can bypass review and silently change retry or security settings.
- Unreviewed direct edits can break jobs without a reproducible rollback commit.
- A wrong schedule can duplicate loads or miss critical reporting windows.
- Duplicate jobs may run on the same schedule and load data twice.
Explanation
Correct Answer
B:The direct residual exposure is: Unreviewed direct edits can break jobs without a reproducible rollback commit. The triggering condition is: Engineers edit production notebooks directly and lose the ability to identify which code change broke a pipeline. The omitted safeguard is: Store deployment source in Git with reviewed branches and pull requests before promoting changes. This matters because: Version control and code review establish provenance and repeatable release history. A corrective test would show: A production run references an approved immutable commit and a prior version can be restored.
Incorrect Answers
A:This failure mode belongs mainly to another situation: A Databricks bundle deployment differs from what appears in the repository because someone edits the job in the UI afterward. Its preventative action would be: Use drift detection or a release reconciliation that compares deployed objects against the versioned bundle. The most direct exposure in the current scenario is: Unreviewed direct edits can break jobs without a reproducible rollback commit.
C:This failure mode belongs mainly to another situation: A data engineer updates a bundled job trigger but another environment's deployment silently uses the wrong schedule. Its preventative action would be: Validate per-target bundle variable resolution and compare rendered job definitions before deployment. The most direct exposure in the current scenario is: Unreviewed direct edits can break jobs without a reproducible rollback commit.
D:This failure mode belongs mainly to another situation: A REST deployment script creates duplicate jobs every time CI reruns after a timeout. Its preventative action would be: Use idempotent deployment methods or reconcile existing resource IDs before creating replacements. The most direct exposure in the current scenario is: Unreviewed direct edits can break jobs without a reproducible rollback commit.
Question 172
An implementation skips: Use caching only for beneficial reuse and release persisted state when no longer needed. Engineers are observing: A team enables Spark cache for a one-time ETL dataset and never unpersists it before processing the next large table. What consequence is MOST likely?
- Over-maintenance can spend more resources rewriting files than it saves in queries.
- Unused cached data can cause eviction, spilling and unexpected failures for later stages.
- A critical SLA may depend on unrelated teams' scheduling and resource usage.
- Premature deletion may invalidate time travel or a lagging reader's file references.
Explanation
Correct Answer
B:The direct residual exposure is: Unused cached data can cause eviction, spilling and unexpected failures for later stages. The triggering condition is: A team enables Spark cache for a one-time ETL dataset and never unpersists it before processing the next large table. The omitted safeguard is: Use caching only for beneficial reuse and release persisted state when no longer needed. This matters because: Caching trades execution time for memory and may increase pressure when datasets are not reused. A corrective test would show: Spark storage metrics show released cache and unchanged correctness with improved total runtime.
Incorrect Answers
A:This failure mode belongs mainly to another situation: A new Delta table maintenance job combines repeated OPTIMIZE and VACUUM at every microbatch even though changes are tiny. Its preventative action would be: Set a measured maintenance cadence or evaluate supported predictive optimization for eligible managed tables. The most direct exposure in the current scenario is: Unused cached data can cause eviction, spilling and unexpected failures for later stages.
C:This failure mode belongs mainly to another situation: A production notebook intermittently times out only when many other jobs share the same classic cluster. Its preventative action would be: Compare workload contention and resource reservations, then isolate or resize the critical job compute. The most direct exposure in the current scenario is: Unused cached data can cause eviction, spilling and unexpected failures for later stages.
D:This failure mode belongs mainly to another situation: An engineer runs VACUUM with a zero-hour retention on a shared Delta table used for historical audit queries. Its preventative action would be: Respect documented safe retention windows and coordinate VACUUM with readers, streams and time-travel needs. The most direct exposure in the current scenario is: Unused cached data can cause eviction, spilling and unexpected failures for later stages.
Question 173
A data product owner adopts: Use fully qualified names and a controlled default namespace in deployment SQL and notebooks. The reason is: An automation pipeline issues CREATE TABLE without qualifying the target catalog and schema. What observed result BEST proves effectiveness?
- Production jobs resolve fully qualified names and fail safely when pointed at a nonproduction catalog.
- Permitted users query only expected columns and rows through the view without a separately exported file.
- Discovery tests distinguish raw from certified curated objects and owners confirm their assigned namespaces.
- The same deployment command creates objects only in its designated staging or production namespace.
Explanation
Correct Answer
D:The strongest acceptance proof is: The same deployment command creates objects only in its designated staging or production namespace. It specifically verifies: Use fully qualified names and a controlled default namespace in deployment SQL and notebooks. The original issue was: An automation pipeline issues CREATE TABLE without qualifying the target catalog and schema. The underlying mechanism is: Explicit namespace resolution makes deployment deterministic across different workspaces. Without it, an environment-default schema can route production data definitions into an incorrect catalog.
Incorrect Answers
A:This result would validate a different measure: Create environment-specific Unity Catalog catalogs or schemas and enforce clear ownership and naming conventions. That is relevant to the distinct problem: Production and development teams create identically named data objects in one catalog and accidentally overwrite test tables. Here the acceptance criterion is: The same deployment command creates objects only in its designated staging or production namespace.
B:This result would validate a different measure: Create a Unity Catalog view with approved permissions and verify underlying access semantics. That is relevant to the distinct problem: An analyst wants to expose a filtered projection of a governed table without copying sensitive source rows. Here the acceptance criterion is: The same deployment command creates objects only in its designated staging or production namespace.
C:This result would validate a different measure: Define a consistent schema and naming architecture separating layers and business domains by purpose. That is relevant to the distinct problem: Multiple data products put raw, refined and curated assets into a single schema with confusing table names. Here the acceptance criterion is: The same deployment command creates objects only in its designated staging or production namespace.
Question 174
The recommended measure is: Tune supported SQL warehouse capacity and concurrency scaling based on simultaneous production query measurements. The operational context is: A BI warehouse repeatedly slows during morning sign-in peaks even though a single analyst query runs quickly. Which data engineering concept applies?
- Concurrency queuing is separate from a single-query execution performance problem.
- End-to-end task time includes compute startup and orchestration overhead, not just transformation CPU time.
- Compute isolation reduces interference between unrelated application environments and execution libraries.
- Serverless and classic compute have different configuration surfaces and supported dependency patterns.
Explanation
Correct Answer
A:The decisive mechanism is: Concurrency queuing is separate from a single-query execution performance problem. That explains the selected action: Tune supported SQL warehouse capacity and concurrency scaling based on simultaneous production query measurements. In the original scenario: A BI warehouse repeatedly slows during morning sign-in peaks even though a single analyst query runs quickly. The specific unmanaged risk is: Sizing from isolated benchmark queries can leave hundreds of dashboard viewers waiting in a query queue. Validate with: A concurrent-user benchmark meets percentile latency and queue-depth targets with acceptable cost.
Incorrect Answers
B:This principle concerns a separate data-engineering condition: A data team uses a high-power job cluster for a low-volume data-quality check that spends most runtime starting the cluster. Its targeted response is: Separate the small validation into an eligible lightweight or serverless task and measure total end-to-end runtime. The current case is governed more directly by: Concurrency queuing is separate from a single-query execution performance problem.
C:This principle concerns a separate data-engineering condition: A notebook pipeline uses a shared cluster with unstable third-party dependencies while a critical finance job needs controlled isolation. Its targeted response is: Run the finance job on dedicated supported job compute with its own pinned libraries and permissions. The current case is governed more directly by: Concurrency queuing is separate from a single-query execution performance problem.
D:This principle concerns a separate data-engineering condition: A long-running job using an unsupported custom system dependency is moved to serverless without checking feature compatibility. Its targeted response is: Review current serverless limitations and select compatible classic jobs compute when necessary. The current case is governed more directly by: Concurrency queuing is separate from a single-query execution performance problem.
Question 175
An implementation skips: Implement idempotent foreachBatch or compatible transactional sink logic keyed by epoch or business identity. Engineers are observing: A stream writes into an external sink without stable batch identities and causes duplicate side effects on retry. What consequence is MOST likely?
- Uncoordinated retries can run transformations before ingestion finishes or under the wrong identity.
- Retries can duplicate invoices or notifications even if the source checkpoint is correct.
- Driver out-of-memory can fail the load despite adequate executor resources.
- Replayed events can double-count orders in downstream aggregates.
Explanation
Correct Answer
B:The direct residual exposure is: Retries can duplicate invoices or notifications even if the source checkpoint is correct. The triggering condition is: A stream writes into an external sink without stable batch identities and causes duplicate side effects on retry. The omitted safeguard is: Implement idempotent foreachBatch or compatible transactional sink logic keyed by epoch or business identity. This matters because: Structured Streaming may retry microbatches, so external effects need idempotency. A corrective test would show: Replaying the same microbatch produces one target effect per business key.
Incorrect Answers
A:This failure mode belongs mainly to another situation: A data product needs to copy a supported Azure Data Factory pipeline output into governed Databricks tables. Its preventative action would be: Coordinate ADF orchestration with the appropriate Databricks task and Unity Catalog access model. The most direct exposure in the current scenario is: Retries can duplicate invoices or notifications even if the source checkpoint is correct.
C:This failure mode belongs mainly to another situation: A Python notebook reads a massive table into pandas on the driver before processing data intended for Spark executors. Its preventative action would be: Use Spark-native distributed read and transformation APIs or partition-safe processing. The most direct exposure in the current scenario is: Retries can duplicate invoices or notifications even if the source checkpoint is correct.
D:This failure mode belongs mainly to another situation: An ingestion pipeline reads cloud events with at-least-once delivery and assumes every event ID appears only once. Its preventative action would be: Deduplicate by stable event identity and business time using documented state and watermark rules. The most direct exposure in the current scenario is: Retries can duplicate invoices or notifications even if the source checkpoint is correct.
Question 176
During a lakehouse implementation review, engineers observe: A multi-team interactive cluster permits users to attach arbitrary unapproved libraries and run notebooks under broad shared privileges. What is the BEST next action?
- Define a job-owned execution identity and eligible task compute separate from ad hoc analyst cluster lifecycle.
- Choose node families and driver/worker sizes using CPU, memory, shuffle and workload profile evidence.
- Set appropriate auto-termination or idle controls for interactive compute while excluding workloads requiring continuous availability.
- Apply suitable compute access modes, policies and entitlements that restrict who can attach or modify shared compute.
Explanation
Correct Answer
D:The most appropriate action is: Apply suitable compute access modes, policies and entitlements that restrict who can attach or modify shared compute. It addresses the stated condition: A multi-team interactive cluster permits users to attach arbitrary unapproved libraries and run notebooks under broad shared privileges. The technical basis is: Workspace compute permissions and access modes help maintain workload and security boundaries. Without this control, an untrusted library or unrestricted cluster edit can affect other users' sessions or access to protected data. A direct acceptance check is: Permission tests show approved analysts run their notebooks while unauthorized cluster policy edits and installs fail.
Incorrect Answers
A:This alternative primarily addresses a different Azure Databricks problem: A scheduled ETL run uses shared analyst compute and fails whenever the cluster owner restarts a notebook session. The corresponding measure there would be: Define a job-owned execution identity and eligible task compute separate from ad hoc analyst cluster lifecycle. The stated situation instead calls for: Apply suitable compute access modes, policies and entitlements that restrict who can attach or modify shared compute.
B:This alternative primarily addresses a different Azure Databricks problem: A new job uses a large memory-optimized node merely because its source data is large, although profiling shows compute-bound transforms. The corresponding measure there would be: Choose node families and driver/worker sizes using CPU, memory, shuffle and workload profile evidence. The stated situation instead calls for: Apply suitable compute access modes, policies and entitlements that restrict who can attach or modify shared compute.
C:This alternative primarily addresses a different Azure Databricks problem: A newly created cluster remains available for hours after manual exploration even though the organization has a strict idle-cost policy. The corresponding measure there would be: Set appropriate auto-termination or idle controls for interactive compute while excluding workloads requiring continuous availability. The stated situation instead calls for: Apply suitable compute access modes, policies and entitlements that restrict who can attach or modify shared compute.
Question 177
A production data platform experiences this condition: A continuously running pipeline is scheduled to restart automatically but restart storms consume a growing compute budget. Which targeted measure should be selected?
- Define validated job parameters and propagate them through supported task parameter bindings.
- Evaluate supported table update triggers or a clear upstream job dependency based on actual availability.
- Use a supported file-arrival trigger where its prerequisites and source location are compatible.
- Configure bounded restarts and alerting with root-cause investigation for repeated failures.
Explanation
Correct Answer
D:The most appropriate action is: Configure bounded restarts and alerting with root-cause investigation for repeated failures. It addresses the stated condition: A continuously running pipeline is scheduled to restart automatically but restart storms consume a growing compute budget. The technical basis is: Automatic recovery should be limited by failure budgets and should not hide persistent faults. Without this control, unbounded restarts can amplify resource consumption while no useful records are processed. A direct acceptance check is: Repeated injected failures hit the configured stop threshold and create a human-actionable incident.
Incorrect Answers
A:This alternative primarily addresses a different Azure Databricks problem: A workflow needs to pass a date parameter from a scheduled trigger into all its downstream notebook tasks. The corresponding measure there would be: Define validated job parameters and propagate them through supported task parameter bindings. The stated situation instead calls for: Configure bounded restarts and alerting with root-cause investigation for repeated failures.
B:This alternative primarily addresses a different Azure Databricks problem: A pipeline must run when an upstream Databricks table is updated but analysts have hardcoded a daily time estimate. The corresponding measure there would be: Evaluate supported table update triggers or a clear upstream job dependency based on actual availability. The stated situation instead calls for: Configure bounded restarts and alerting with root-cause investigation for repeated failures.
C:This alternative primarily addresses a different Azure Databricks problem: A cloud file feed needs a job triggered by arriving files instead of polling every minute. The corresponding measure there would be: Use a supported file-arrival trigger where its prerequisites and source location are compatible. The stated situation instead calls for: Configure bounded restarts and alerting with root-cause investigation for repeated failures.
Question 178
An operator introduces: Revoke inappropriate base-table access and publish only the authorized view or policy-governed object. The situation is: A data owner builds a masked view but analysts can still query the underlying sensitive table directly. Which validation should be required before signoff?
- An approved DDL operation creates the external table at a registered location and an arbitrary path is rejected.
- Direct source queries are denied for analysts while the governed view produces the approved masked data.
- The analyst can SELECT the authorized table but cannot UPDATE it or read an unrelated protected table.
- The visiting user reads the designated dataset before expiry and loses access afterwards without ownership changes.
Explanation
Correct Answer
B:The strongest acceptance proof is: Direct source queries are denied for analysts while the governed view produces the approved masked data. It specifically verifies: Revoke inappropriate base-table access and publish only the authorized view or policy-governed object. The original issue was: A data owner builds a masked view but analysts can still query the underlying sensitive table directly. The underlying mechanism is: Security of a view depends on the complete authorization path, not merely its SELECT projection. Without it, A user may bypass the masked presentation by reading the unmasked source table.
Incorrect Answers
A:This result would validate a different measure: Configure a supported external location and specific creation privileges rather than bypassing the catalog to write directly. That is relevant to the distinct problem: A user is granted CREATE TABLE but cannot create an external table at an unregistered cloud path. Here the acceptance criterion is: Direct source queries are denied for analysts while the governed view produces the approved masked data.
C:This result would validate a different measure: Grant only the required USE and SELECT privileges to the analyst group and inspect inherited grants. That is relevant to the distinct problem: A finance analyst must read an approved table but was granted broad MODIFY rights at the schema level. Here the acceptance criterion is: Direct source queries are denied for analysts while the governed view produces the approved masked data.
D:This result would validate a different measure: Issue time-bounded group membership or an approved scoped SELECT grant with an access review. That is relevant to the distinct problem: A data steward wants to grant temporary read access without handing over Unity Catalog ownership of a schema. Here the acceptance criterion is: Direct source queries are denied for analysts while the governed view produces the approved masked data.
Question 179
A data product owner adopts: Create environment-specific Unity Catalog catalogs or schemas and enforce clear ownership and naming conventions. The reason is: Production and development teams create identically named data objects in one catalog and accidentally overwrite test tables. What observed result BEST proves effectiveness?
- Contract tests continue to select documented columns through the published view after a staged schema change.
- A source update followed by the documented refresh produces reconciled aggregate totals.
- Production jobs resolve fully qualified names and fail safely when pointed at a nonproduction catalog.
- A capability test documents supported query and write operations and prevents unsupported mutations.
Explanation
Correct Answer
C:The strongest acceptance proof is: Production jobs resolve fully qualified names and fail safely when pointed at a nonproduction catalog. It specifically verifies: Create environment-specific Unity Catalog catalogs or schemas and enforce clear ownership and naming conventions. The original issue was: Production and development teams create identically named data objects in one catalog and accidentally overwrite test tables. The underlying mechanism is: Catalog and schema hierarchy can isolate environments while making object ownership and purpose explicit. Without it, an unqualified table reference may redirect a production job to a developer's test dataset.
Incorrect Answers
A:This result would validate a different measure: Define a stable contract through managed views and test compatibility before changing source objects. That is relevant to the distinct problem: A reporting service needs to keep a stable presentation schema while underlying source columns are renamed by engineering. Here the acceptance criterion is: Production jobs resolve fully qualified names and fail safely when pointed at a nonproduction catalog.
B:This result would validate a different measure: Create an eligible materialized view or managed derived table with appropriate refresh and access rules. That is relevant to the distinct problem: A BI group needs to query a materialized aggregate while the source transaction Delta table changes hourly. Here the acceptance criterion is: Production jobs resolve fully qualified names and fail safely when pointed at a nonproduction catalog.
D:This result would validate a different measure: Check federation connector capabilities and distinguish read-only remote access from supported writable local tables. That is relevant to the distinct problem: A foreign catalog is connected to a remote database but an analyst expects standard managed Delta writes against it. Here the acceptance criterion is: Production jobs resolve fully qualified names and fail safely when pointed at a nonproduction catalog.
Question 180
A data engineering team faces this Azure Databricks requirement: A data product needs to copy a supported Azure Data Factory pipeline output into governed Databricks tables. Which action is MOST appropriate?
- Coordinate ADF orchestration with the appropriate Databricks task and Unity Catalog access model.
- Specify the correct input delimiter, header and schema and validate parsing on a representative file.
- Use Spark-native distributed read and transformation APIs or partition-safe processing.
- Apply ordered CDC sequencing using source change version or timestamp and consistent delete handling.
Explanation
Correct Answer
A:The most appropriate action is: Coordinate ADF orchestration with the appropriate Databricks task and Unity Catalog access model. It addresses the stated condition: A data product needs to copy a supported Azure Data Factory pipeline output into governed Databricks tables. The technical basis is: An external orchestrator can trigger Databricks processing while governance and failure handling remain explicit. Without this control, uncoordinated retries can run transformations before ingestion finishes or under the wrong identity. A direct acceptance check is: ADF completion evidence and Databricks job dependencies show an end-to-end successful load.
Incorrect Answers
B:This alternative primarily addresses a different Azure Databricks problem: An overnight load reads tab-separated files but an engineer assumes comma separators and casts malformed columns to null. The corresponding measure there would be: Specify the correct input delimiter, header and schema and validate parsing on a representative file. The stated situation instead calls for: Coordinate ADF orchestration with the appropriate Databricks task and Unity Catalog access model.
C:This alternative primarily addresses a different Azure Databricks problem: A Python notebook reads a massive table into pandas on the driver before processing data intended for Spark executors. The corresponding measure there would be: Use Spark-native distributed read and transformation APIs or partition-safe processing. The stated situation instead calls for: Coordinate ADF orchestration with the appropriate Databricks task and Unity Catalog access model.
D:This alternative primarily addresses a different Azure Databricks problem: A transactional feed produces out-of-order CDC events and a naive MERGE overwrites a newer status with an older update. The corresponding measure there would be: Apply ordered CDC sequencing using source change version or timestamp and consistent delete handling. The stated situation instead calls for: Coordinate ADF orchestration with the appropriate Databricks task and Unity Catalog access model.