Certified Data Engineer Associate Practice Test 57
Question 560
At a tax accounting service, one shuffle stage has a single task reading gigabytes more data than its peers and dominates total execution time. The dataset `supplier_orders` is in scope, and the proposed fix needs before-and-after measurements. Before release, which acceptance test would actually demonstrate that the root problem is controlled?
- Check that the intended configuration exists for `supplier_orders` and that a healthy run succeeded, without inducing the earlier failure.
- Apply the selected partitioning change and compare longest task time, spill and aggregate correctness
- Review a platform-level availability report for `supplier_orders` without testing whether one heavily skewed key concentrates work in a straggling shuffle partition.
- Run only a normal, unchanged `supplier_orders` sample and review max versus median shuffle read.
Explanation
Correct Answer
B:Apply the selected partitioning change and compare longest task time, spill and aggregate correctness. This reproduces the scenario-dependent condition, verifies max versus median shuffle read, stage task times, spill bytes, and heavy key counts, and directly tests whether one heavily skewed key concentrates work in a straggling shuffle partition is resolved. Observed results, not intent, determine approval.
Incorrect Answers
A:Check that the intended configuration exists for `supplier_orders` and that a healthy run succeeded, without inducing the earlier failure. Installed configuration and a healthy run are insufficient to prove the failure mode is gone. The original issue was one heavily skewed key concentrates work in a straggling shuffle partition.
C:Review a platform-level availability report for `supplier_orders` without testing whether one heavily skewed key concentrates work in a straggling shuffle partition. Aggregate availability does not establish the behavior of affected `supplier_orders` records. It cannot replace the direct exercise: apply the selected partitioning change and compare longest task time, spill and aggregate correctness.
D:Run only a normal, unchanged `supplier_orders` sample and review max versus median shuffle read. A normal sample does not challenge one heavily skewed key concentrates work in a straggling shuffle partition.
Question 561
At a telecom operations group, join stages spill heavily to disk after a new wide dataset was introduced, even though CPU utilization looks moderate. The dataset `research_events` is in scope, and a successful task does not excuse excessive resource usage. What evidence should the engineer collect to establish the cause rather than a coincidental symptom?
- Review completed run status and aggregate cluster health without record-level correlation
- Correlate memory/disk spill bytes, shuffle exchange size across affected and unaffected `R034-CASE-113` runs.
- Compare the current configuration and the signed change approval, without source-row evidence
- Inspect normal-period success metrics for the stated dataset without reproducing the source error
Explanation
Correct Answer
B:Correlate memory/disk spill bytes, shuffle exchange size across affected and unaffected `R034-CASE-113` runs.. By joining memory/disk spill bytes, shuffle exchange size, executor GC, and task duration across the affected and unaffected conditions, the investigation directly tests the recorded mechanism: wide shuffle partitions exceed available execution memory and spill intermediate data to disk. It also sets a measurable baseline for replay the wide join after a bounded partition/memory change and compare spill plus end-to-end time.
Incorrect Answers
A:Review completed run status and aggregate cluster health without record-level correlation. A successful aggregate platform state can coexist with wide shuffle partitions exceed available execution memory and spill intermediate data to disk. This check never observes the affected `research_events` records or their causal path.
C:Compare the current configuration and the signed change approval, without source-row evidence. A desired configuration and approval ticket are not runtime proof. In this case they omit memory/disk spill bytes, shuffle exchange size, executor GC, and task duration, the direct means of distinguishing wide shuffle partitions exceed available execution memory and spill intermediate data to disk.
D:Inspect normal-period success metrics for the stated dataset without reproducing the source error. A healthy-run trend can be useful, but it does not expose wide shuffle partitions exceed available execution memory and spill intermediate data to disk. The failing-path memory/disk spill bytes, shuffle exchange size, executor GC, and task duration has to be correlated before attributing cause.
Question 562
At a telecom operations group, join stages spill heavily to disk after a new wide dataset was introduced, even though CPU utilization looks moderate. The dataset `research_events` is in scope, and a successful task does not excuse excessive resource usage. Which platform configuration or data-engineering change is most defensible?
- Treat low average CPU as proof that the join cannot be resource constrained; use that approach for `research_events`.
- Tune only the notebook driver heap based on its idle CPU graph; use that approach for `research_events`.
- Cache every large input dataset without measuring memory use; use that approach for `research_events`.
- Use Spark UI spill and shuffle metrics to evaluate partition size and executor-memory pressure before tuning
Explanation
Correct Answer
D:Use Spark UI spill and shuffle metrics to evaluate partition size and executor-memory pressure before tuning. This directly addresses wide shuffle partitions exceed available execution memory and spill intermediate data to disk. The design should be accepted only after replay the wide join after a bounded partition/memory change and compare spill plus end-to-end time; the related operational boundary is to increase only justified memory or partition capacity and verify cost versus performance.
Incorrect Answers
A:Treat low average CPU as proof that the join cannot be resource constrained; use that approach for `research_events`. Disk I/O and executor memory can dominate even with moderate CPU usage. For `research_events`, this cannot remove the causal limitation: wide shuffle partitions exceed available execution memory and spill intermediate data to disk.
B:Tune only the notebook driver heap based on its idle CPU graph; use that approach for `research_events`. Executor shuffle spill is not cured simply by changing driver heap. For `research_events`, this cannot remove the causal limitation: wide shuffle partitions exceed available execution memory and spill intermediate data to disk.
C:Cache every large input dataset without measuring memory use; use that approach for `research_events`. Unbounded caching can intensify memory pressure and spilling. For `research_events`, this cannot remove the causal limitation: wide shuffle partitions exceed available execution memory and spill intermediate data to disk.
Question 563
At a telecom operations group, join stages spill heavily to disk after a new wide dataset was introduced, even though CPU utilization looks moderate. The dataset `research_events` is in scope, and a successful task does not excuse excessive resource usage. A teammate proposes: "Tune only the notebook driver heap based on its idle CPU graph." Which technical limitation is the decisive reason not to approve that specific proposal?
- Disk I/O and executor memory can dominate even with moderate CPU usage. This would apply to another option for `research_events`.
- Executor shuffle spill is not cured simply by changing driver heap. The proposal does not repair `research_events`.
- The task is waiting for its next schedule; Spark stage metrics and task failure records are irrelevant.
- Unbounded caching can intensify memory pressure and spilling. This would apply to an alternative change for `research_events`.
Explanation
Correct Answer
B:Executor shuffle spill is not cured simply by changing driver heap. The proposal does not repair `research_events`. This identifies the known limitation of the exact proposed action. Executor shuffle spill is not cured simply by changing driver heap. The underlying problem remains wide shuffle partitions exceed available execution memory and spill intermediate data to disk; Use Spark UI spill and shuffle metrics to evaluate partition size and executor-memory pressure before tuning instead targets that failure.
Incorrect Answers
A:Disk I/O and executor memory can dominate even with moderate CPU usage. This would apply to another option for `research_events`. This limitation belongs to the different alternative "Treat low average CPU as proof that the join cannot be resource constrained". The stem calls for the specific failure of "Tune only the notebook driver heap based on its idle CPU graph".
C:The task is waiting for its next schedule; Spark stage metrics and task failure records are irrelevant. The case provides a concrete failing data or control path; a mere alternate theory does not address wide shuffle partitions exceed available execution memory and spill intermediate data to disk.
D:Unbounded caching can intensify memory pressure and spilling. This would apply to an alternative change for `research_events`. This failure applies to the separate proposal "Cache every large input dataset without measuring memory use".
Question 564
At a telecom operations group, join stages spill heavily to disk after a new wide dataset was introduced, even though CPU utilization looks moderate. The dataset `research_events` is in scope, and a successful task does not excuse excessive resource usage. What risk-and-cost compromise should the owner accept instead of the proposed shortcuts?
- Increase only justified memory or partition capacity and verify cost versus performance
- Tune only the notebook driver heap based on its idle CPU graph
- Cache every large input dataset without measuring memory use
- Treat low average CPU as proof that the join cannot be resource constrained
Explanation
Correct Answer
A:Increase only justified memory or partition capacity and verify cost versus performance. This explicitly bounds the compromise while still addressing wide shuffle partitions exceed available execution memory and spill intermediate data to disk. Acceptance relies on replay the wide join after a bounded partition/memory change and compare spill plus end-to-end time rather than simply declaring the change complete.
Incorrect Answers
B:Tune only the notebook driver heap based on its idle CPU graph. Executor shuffle spill is not cured simply by changing driver heap. Monitoring after deployment does not make this approach a safe bounded tradeoff for `research_events`.
C:Cache every large input dataset without measuring memory use. Unbounded caching can intensify memory pressure and spilling. Monitoring after deployment does not make this approach a safe bounded tradeoff for `research_events`.
D:Treat low average CPU as proof that the join cannot be resource constrained. Disk I/O and executor memory can dominate even with moderate CPU usage. Monitoring after deployment does not make this approach a safe bounded tradeoff for `research_events`.
Question 565
At a telecom operations group, join stages spill heavily to disk after a new wide dataset was introduced, even though CPU utilization looks moderate. The dataset `research_events` is in scope, and a successful task does not excuse excessive resource usage. Which direct test could falsify the proposed fix instead of only confirming deployment?
- Replay the wide join after a bounded partition/memory change and compare spill plus end-to-end time
- Run only a normal, unchanged `research_events` sample and review memory/disk spill bytes.
- Check that the intended configuration exists for `research_events` and that a healthy run succeeded, without inducing the earlier failure.
- Review a platform-level availability report for `research_events` without testing whether wide shuffle partitions exceed available execution memory and spill intermediate data to disk.
Explanation
Correct Answer
A:Replay the wide join after a bounded partition/memory change and compare spill plus end-to-end time. This reproduces the scenario-dependent condition, verifies memory/disk spill bytes, shuffle exchange size, executor GC, and task duration, and directly tests whether wide shuffle partitions exceed available execution memory and spill intermediate data to disk is resolved. Observed results, not intent, determine approval.
Incorrect Answers
B:Run only a normal, unchanged `research_events` sample and review memory/disk spill bytes. A normal sample does not challenge wide shuffle partitions exceed available execution memory and spill intermediate data to disk.
C:Check that the intended configuration exists for `research_events` and that a healthy run succeeded, without inducing the earlier failure. Installed configuration and a healthy run are insufficient to prove the failure mode is gone. The original issue was wide shuffle partitions exceed available execution memory and spill intermediate data to disk.
D:Review a platform-level availability report for `research_events` without testing whether wide shuffle partitions exceed available execution memory and spill intermediate data to disk. Aggregate availability does not establish the behavior of affected `research_events` records. It cannot replace the direct exercise: replay the wide join after a bounded partition/memory change and compare spill plus end-to-end time.
Question 566
At a city utilities office, one shuffle stage has a single task reading gigabytes more data than its peers and dominates total execution time. The dataset `experiment_samples` is in scope, and the proposed fix needs before-and-after measurements. Which correlated observation is the strongest basis for the change decision?
- Compare the saved configuration with the signed change ticket for `experiment_samples` while omitting source and output records.
- Examine only a normal successful run of `experiment_samples` and trend max versus median shuffle read without recreating the fault.
- Correlate max versus median shuffle read, stage task times across affected and unaffected `R034-CASE-114` runs.
- Review aggregate cluster availability and job completion for `experiment_samples` without correlating the observed error path.
Explanation
Correct Answer
C:Correlate max versus median shuffle read, stage task times across affected and unaffected `R034-CASE-114` runs.. By joining max versus median shuffle read, stage task times, spill bytes, and heavy key counts across the affected and unaffected conditions, the investigation directly tests the recorded mechanism: one heavily skewed key concentrates work in a straggling shuffle partition. It also sets a measurable baseline for apply the selected partitioning change and compare longest task time, spill and aggregate correctness.
Incorrect Answers
A:Compare the saved configuration with the signed change ticket for `experiment_samples` while omitting source and output records. A desired configuration and approval ticket are not runtime proof. In this case they omit max versus median shuffle read, stage task times, spill bytes, and heavy key counts, the direct means of distinguishing one heavily skewed key concentrates work in a straggling shuffle partition.
B:Examine only a normal successful run of `experiment_samples` and trend max versus median shuffle read without recreating the fault. A healthy-run trend can be useful, but it does not expose one heavily skewed key concentrates work in a straggling shuffle partition. The failing-path max versus median shuffle read, stage task times, spill bytes, and heavy key counts has to be correlated before attributing cause.
D:Review aggregate cluster availability and job completion for `experiment_samples` without correlating the observed error path. A successful aggregate platform state can coexist with one heavily skewed key concentrates work in a straggling shuffle partition. This check never observes the affected `experiment_samples` records or their causal path.
Question 567
At a city utilities office, one shuffle stage has a single task reading gigabytes more data than its peers and dominates total execution time. The dataset `experiment_samples` is in scope, and the proposed fix needs before-and-after measurements. What implementation should the engineer select for the failing workflow?
- Diagnose key skew in Spark UI and apply a bounded skew-aware partitioning or salting strategy for the affected operation
- Ignore task-level metrics and double all executors permanently; use that approach for `experiment_samples`.
- Reduce the shuffle partition count sharply across the whole application; use that approach for `experiment_samples`.
- Increase only driver memory and leave the skewed execution plan unchanged; use that approach for `experiment_samples`.
Explanation
Correct Answer
A:Diagnose key skew in Spark UI and apply a bounded skew-aware partitioning or salting strategy for the affected operation. This directly addresses one heavily skewed key concentrates work in a straggling shuffle partition. The design should be accepted only after apply the selected partitioning change and compare longest task time, spill and aggregate correctness; the related operational boundary is to target the skewed operation instead of scaling the whole application without evidence.
Incorrect Answers
B:Ignore task-level metrics and double all executors permanently; use that approach for `experiment_samples`. Additional idle executors do not split one skewed key and inflate cost. For `experiment_samples`, this cannot remove the causal limitation: one heavily skewed key concentrates work in a straggling shuffle partition.
C:Reduce the shuffle partition count sharply across the whole application; use that approach for `experiment_samples`. Larger shuffle partitions can worsen stragglers and disk spill. For `experiment_samples`, this cannot remove the causal limitation: one heavily skewed key concentrates work in a straggling shuffle partition.
D:Increase only driver memory and leave the skewed execution plan unchanged; use that approach for `experiment_samples`. Driver memory does not redistribute an oversized executor-side shuffle partition. For `experiment_samples`, this cannot remove the causal limitation: one heavily skewed key concentrates work in a straggling shuffle partition.
Question 568
At a city utilities office, one shuffle stage has a single task reading gigabytes more data than its peers and dominates total execution time. The dataset `experiment_samples` is in scope, and the proposed fix needs before-and-after measurements. A teammate proposes: "Ignore task-level metrics and double all executors permanently." Which technical limitation is the decisive reason not to approve that specific proposal?
- Additional idle executors do not split one skewed key and inflate cost. The proposal does not repair `experiment_samples`.
- Larger shuffle partitions can worsen stragglers and disk spill. This would apply to another option for `experiment_samples`.
- Driver memory does not redistribute an oversized executor-side shuffle partition. This would apply to an alternative change for `experiment_samples`.
- The task is waiting for its next schedule; Spark stage metrics and task failure records are irrelevant.
Explanation
Correct Answer
A:Additional idle executors do not split one skewed key and inflate cost. The proposal does not repair `experiment_samples`. This identifies the known limitation of the exact proposed action. Additional idle executors do not split one skewed key and inflate cost. The underlying problem remains one heavily skewed key concentrates work in a straggling shuffle partition; Diagnose key skew in Spark UI and apply a bounded skew-aware partitioning or salting strategy for the affected operation instead targets that failure.
Incorrect Answers
B:Larger shuffle partitions can worsen stragglers and disk spill. This would apply to another option for `experiment_samples`. This limitation belongs to the different alternative "Reduce the shuffle partition count sharply across the whole application". The stem calls for the specific failure of "Ignore task-level metrics and double all executors permanently".
C:Driver memory does not redistribute an oversized executor-side shuffle partition. This would apply to an alternative change for `experiment_samples`. This failure applies to the separate proposal "Increase only driver memory and leave the skewed execution plan unchanged".
D:The task is waiting for its next schedule; Spark stage metrics and task failure records are irrelevant. The case provides a concrete failing data or control path; a mere alternate theory does not address one heavily skewed key concentrates work in a straggling shuffle partition.
Question 569
At a city utilities office, one shuffle stage has a single task reading gigabytes more data than its peers and dominates total execution time. The dataset `experiment_samples` is in scope, and the proposed fix needs before-and-after measurements. Which decision preserves the stated data contract while keeping the change scope proportionate?
- Reduce the shuffle partition count sharply across the whole application; monitor results only after launch of `experiment_samples`.
- Increase only driver memory and leave the skewed execution plan unchanged; monitor results only after launch of `experiment_samples`.
- Ignore task-level metrics and double all executors permanently; monitor results only after launch of `experiment_samples`.
- Target the skewed operation instead of scaling the whole application without evidence
Explanation
Correct Answer
D:Target the skewed operation instead of scaling the whole application without evidence. This explicitly bounds the compromise while still addressing one heavily skewed key concentrates work in a straggling shuffle partition. Acceptance relies on apply the selected partitioning change and compare longest task time, spill and aggregate correctness rather than simply declaring the change complete.
Incorrect Answers
A:Reduce the shuffle partition count sharply across the whole application; monitor results only after launch of `experiment_samples`. Larger shuffle partitions can worsen stragglers and disk spill. Monitoring after deployment does not make this approach a safe bounded tradeoff for `experiment_samples`.
B:Increase only driver memory and leave the skewed execution plan unchanged; monitor results only after launch of `experiment_samples`. Driver memory does not redistribute an oversized executor-side shuffle partition. Monitoring after deployment does not make this approach a safe bounded tradeoff for `experiment_samples`.
C:Ignore task-level metrics and double all executors permanently; monitor results only after launch of `experiment_samples`. Additional idle executors do not split one skewed key and inflate cost. Monitoring after deployment does not make this approach a safe bounded tradeoff for `experiment_samples`.
Question 570
At a city utilities office, one shuffle stage has a single task reading gigabytes more data than its peers and dominates total execution time. The dataset `experiment_samples` is in scope, and the proposed fix needs before-and-after measurements. What verification provides evidence of correct behavior under the original adverse condition?
- Run only a normal, unchanged `experiment_samples` sample and review max versus median shuffle read.
- Review a platform-level availability report for `experiment_samples` without testing whether one heavily skewed key concentrates work in a straggling shuffle partition.
- Apply the selected partitioning change and compare longest task time, spill and aggregate correctness
- Check that the intended configuration exists for `experiment_samples` and that a healthy run succeeded, without inducing the earlier failure.
Explanation
Correct Answer
C:Apply the selected partitioning change and compare longest task time, spill and aggregate correctness. This reproduces the scenario-dependent condition, verifies max versus median shuffle read, stage task times, spill bytes, and heavy key counts, and directly tests whether one heavily skewed key concentrates work in a straggling shuffle partition is resolved. Observed results, not intent, determine approval.
Incorrect Answers
A:Run only a normal, unchanged `experiment_samples` sample and review max versus median shuffle read. A normal sample does not challenge one heavily skewed key concentrates work in a straggling shuffle partition.
B:Review a platform-level availability report for `experiment_samples` without testing whether one heavily skewed key concentrates work in a straggling shuffle partition. Aggregate availability does not establish the behavior of affected `experiment_samples` records. It cannot replace the direct exercise: apply the selected partitioning change and compare longest task time, spill and aggregate correctness.
D:Check that the intended configuration exists for `experiment_samples` and that a healthy run succeeded, without inducing the earlier failure. Installed configuration and a healthy run are insufficient to prove the failure mode is gone. The original issue was one heavily skewed key concentrates work in a straggling shuffle partition.