Certified Data Engineer Associate Practice Test 30
Question 290
At a subscription media service, a Spark SQL aggregation creates a few oversized shuffle partitions while most tasks finish quickly. The dataset `fraud_signals` is in scope, and downstream dashboards cannot accept fabricated values. What verification provides evidence of correct behavior under the original adverse condition?
- Check that the intended configuration exists for `fraud_signals` and that a healthy run succeeded, without inducing the earlier failure.
- Run only a normal, unchanged `fraud_signals` sample and review Spark stage task-time spread.
- Run the aggregation before and after the selected partitioning change and compare long-tail duration and spill
- Review a platform-level availability report for `fraud_signals` without testing whether the long tail is caused by uneven shuffle partition sizes, not only an insufficient total number of executors.
Explanation
Correct Answer
C:Run the aggregation before and after the selected partitioning change and compare long-tail duration and spill. This reproduces the scenario-dependent condition, verifies Spark stage task-time spread, max/median shuffle bytes, spill metrics, and skewed key frequencies, and directly tests whether the long tail is caused by uneven shuffle partition sizes, not only an insufficient total number of executors is resolved. Observed results, not intent, determine approval.
Incorrect Answers
A:Check that the intended configuration exists for `fraud_signals` and that a healthy run succeeded, without inducing the earlier failure. Installed configuration and a healthy run are insufficient to prove the failure mode is gone. The original issue was the long tail is caused by uneven shuffle partition sizes, not only an insufficient total number of executors.
B:Run only a normal, unchanged `fraud_signals` sample and review Spark stage task-time spread. A normal sample does not challenge the long tail is caused by uneven shuffle partition sizes, not only an insufficient total number of executors.
D:Review a platform-level availability report for `fraud_signals` without testing whether the long tail is caused by uneven shuffle partition sizes, not only an insufficient total number of executors. Aggregate availability does not establish the behavior of affected `fraud_signals` records. It cannot replace the direct exercise: run the aggregation before and after the selected partitioning change and compare long-tail duration and spill.
Question 291
At a regional banking group, a small dimension join is unexpectedly expensive after the broadcast threshold was lowered in a new runtime configuration. The dataset `claims_records` is in scope, and raw records must remain available for replay. Which investigation best tests the stated constraint with direct data?
- Inspect normal-period success metrics for the stated dataset without reproducing the source error
- Review completed run status and aggregate cluster health without record-level correlation
- Compare the current configuration and the signed change approval, without source-row evidence
- Correlate dimension size, physical join plan across affected and unaffected `R034-CASE-059` runs.
Explanation
Correct Answer
D:Correlate dimension size, physical join plan across affected and unaffected `R034-CASE-059` runs.. By joining dimension size, physical join plan, shuffle read, and query duration before and after tuning across the affected and unaffected conditions, the investigation directly tests the recorded mechanism: the optimizer may choose a shuffle join after a threshold change even when a small dimension could be broadcast. It also sets a measurable baseline for reproduce the query at known dimension sizes and confirm the intended join strategy plus measured improvement.
Incorrect Answers
A:Inspect normal-period success metrics for the stated dataset without reproducing the source error. A healthy-run trend can be useful, but it does not expose the optimizer may choose a shuffle join after a threshold change even when a small dimension could be broadcast. The failing-path dimension size, physical join plan, shuffle read, and query duration before and after tuning has to be correlated before attributing cause.
B:Review completed run status and aggregate cluster health without record-level correlation. A successful aggregate platform state can coexist with the optimizer may choose a shuffle join after a threshold change even when a small dimension could be broadcast. This check never observes the affected `claims_records` records or their causal path.
C:Compare the current configuration and the signed change approval, without source-row evidence. A desired configuration and approval ticket are not runtime proof. In this case they omit dimension size, physical join plan, shuffle read, and query duration before and after tuning, the direct means of distinguishing the optimizer may choose a shuffle join after a threshold change even when a small dimension could be broadcast.
Question 292
At a regional banking group, a small dimension join is unexpectedly expensive after the broadcast threshold was lowered in a new runtime configuration. The dataset `claims_records` is in scope, and raw records must remain available for replay. Which option targets the actual technical dependency rather than an incidental symptom?
- Double executor memory and stop without checking the new plan; use that approach for `claims_records`.
- Force the larger fact DataFrame as the broadcast side; use that approach for `claims_records`.
- Inspect the physical join plan and input sizes, then set a safe broadcast strategy and remeasure the join
- Disable all adaptive query optimizations globally; use that approach for `claims_records`.
Explanation
Correct Answer
C:Inspect the physical join plan and input sizes, then set a safe broadcast strategy and remeasure the join. This directly addresses the optimizer may choose a shuffle join after a threshold change even when a small dimension could be broadcast. The design should be accepted only after reproduce the query at known dimension sizes and confirm the intended join strategy plus measured improvement; the related operational boundary is to tune the affected join in a bounded manner rather than changing unrelated global compute limits.
Incorrect Answers
A:Double executor memory and stop without checking the new plan; use that approach for `claims_records`. Memory alone does not prove the optimizer selected the intended join method. For `claims_records`, this cannot remove the causal limitation: the optimizer may choose a shuffle join after a threshold change even when a small dimension could be broadcast.
B:Force the larger fact DataFrame as the broadcast side; use that approach for `claims_records`. Replicating a large fact DataFrame risks out-of-memory errors. For `claims_records`, this cannot remove the causal limitation: the optimizer may choose a shuffle join after a threshold change even when a small dimension could be broadcast.
D:Disable all adaptive query optimizations globally; use that approach for `claims_records`. Removing adaptive choices may degrade other workloads without fixing the local threshold. For `claims_records`, this cannot remove the causal limitation: the optimizer may choose a shuffle join after a threshold change even when a small dimension could be broadcast.
Question 293
At a regional banking group, a small dimension join is unexpectedly expensive after the broadcast threshold was lowered in a new runtime configuration. The dataset `claims_records` is in scope, and raw records must remain available for replay. A teammate proposes: "Double executor memory and stop without checking the new plan." Which technical limitation is the decisive reason not to approve that specific proposal?
- The dashboard cached the prior result; no DataFrame operation or underlying data requires correction.
- Removing adaptive choices may degrade other workloads without fixing the local threshold. This would apply to an alternative change for `claims_records`.
- Replicating a large fact DataFrame risks out-of-memory errors. This would apply to another option for `claims_records`.
- Memory alone does not prove the optimizer selected the intended join method. The proposal does not repair `claims_records`.
Explanation
Correct Answer
D:Memory alone does not prove the optimizer selected the intended join method. The proposal does not repair `claims_records`. This identifies the known limitation of the exact proposed action. Memory alone does not prove the optimizer selected the intended join method. The underlying problem remains the optimizer may choose a shuffle join after a threshold change even when a small dimension could be broadcast; Inspect the physical join plan and input sizes, then set a safe broadcast strategy and remeasure the join instead targets that failure.
Incorrect Answers
A:The dashboard cached the prior result; no DataFrame operation or underlying data requires correction. The case provides a concrete failing data or control path; a mere alternate theory does not address the optimizer may choose a shuffle join after a threshold change even when a small dimension could be broadcast.
B:Removing adaptive choices may degrade other workloads without fixing the local threshold. This would apply to an alternative change for `claims_records`. This failure applies to the separate proposal "Disable all adaptive query optimizations globally".
C:Replicating a large fact DataFrame risks out-of-memory errors. This would apply to another option for `claims_records`. This limitation belongs to the different alternative "Force the larger fact DataFrame as the broadcast side". The stem calls for the specific failure of "Double executor memory and stop without checking the new plan".
Question 294
At a regional banking group, a small dimension join is unexpectedly expensive after the broadcast threshold was lowered in a new runtime configuration. The dataset `claims_records` is in scope, and raw records must remain available for replay. What operational boundary should be documented for approval?
- Double executor memory and stop without checking the new plan
- Disable all adaptive query optimizations globally
- Tune the affected join in a bounded manner rather than changing unrelated global compute limits
- Force the larger fact DataFrame as the broadcast side
Explanation
Correct Answer
C:Tune the affected join in a bounded manner rather than changing unrelated global compute limits. This explicitly bounds the compromise while still addressing the optimizer may choose a shuffle join after a threshold change even when a small dimension could be broadcast. Acceptance relies on reproduce the query at known dimension sizes and confirm the intended join strategy plus measured improvement rather than simply declaring the change complete.
Incorrect Answers
A:Double executor memory and stop without checking the new plan. Memory alone does not prove the optimizer selected the intended join method. Monitoring after deployment does not make this approach a safe bounded tradeoff for `claims_records`.
B:Disable all adaptive query optimizations globally. Removing adaptive choices may degrade other workloads without fixing the local threshold. Monitoring after deployment does not make this approach a safe bounded tradeoff for `claims_records`.
D:Force the larger fact DataFrame as the broadcast side. Replicating a large fact DataFrame risks out-of-memory errors. Monitoring after deployment does not make this approach a safe bounded tradeoff for `claims_records`.
Question 295
At a regional banking group, a small dimension join is unexpectedly expensive after the broadcast threshold was lowered in a new runtime configuration. The dataset `claims_records` is in scope, and raw records must remain available for replay. Which validation should be required in the release record?
- Check that the intended configuration exists for `claims_records` and that a healthy run succeeded, without inducing the earlier failure.
- Review a platform-level availability report for `claims_records` without testing whether the optimizer may choose a shuffle join after a threshold change even when a small dimension could be broadcast.
- Run only a normal, unchanged `claims_records` sample and review dimension size.
- Reproduce the query at known dimension sizes and confirm the intended join strategy plus measured improvement
Explanation
Correct Answer
D:Reproduce the query at known dimension sizes and confirm the intended join strategy plus measured improvement. This reproduces the scenario-dependent condition, verifies dimension size, physical join plan, shuffle read, and query duration before and after tuning, and directly tests whether the optimizer may choose a shuffle join after a threshold change even when a small dimension could be broadcast is resolved. Observed results, not intent, determine approval.
Incorrect Answers
A:Check that the intended configuration exists for `claims_records` and that a healthy run succeeded, without inducing the earlier failure. Installed configuration and a healthy run are insufficient to prove the failure mode is gone. The original issue was the optimizer may choose a shuffle join after a threshold change even when a small dimension could be broadcast.
B:Review a platform-level availability report for `claims_records` without testing whether the optimizer may choose a shuffle join after a threshold change even when a small dimension could be broadcast. Aggregate availability does not establish the behavior of affected `claims_records` records. It cannot replace the direct exercise: reproduce the query at known dimension sizes and confirm the intended join strategy plus measured improvement.
C:Run only a normal, unchanged `claims_records` sample and review dimension size. A normal sample does not challenge the optimizer may choose a shuffle join after a threshold change even when a small dimension could be broadcast.
Question 296
At a census statistics agency, a Spark SQL aggregation creates a few oversized shuffle partitions while most tasks finish quickly. The dataset `account_statements` is in scope, and downstream dashboards cannot accept fabricated values. Before choosing a remedy, which investigation actually distinguishes the failure mechanism from a green dashboard?
- Correlate Spark stage task-time spread, max/median shuffle bytes across affected and unaffected `R034-CASE-060` runs.
- Compare the saved configuration with the signed change ticket for `account_statements` while omitting source and output records.
- Examine only a normal successful run of `account_statements` and trend Spark stage task-time spread without recreating the fault.
- Review aggregate cluster availability and job completion for `account_statements` without correlating the observed error path.
Explanation
Correct Answer
A:Correlate Spark stage task-time spread, max/median shuffle bytes across affected and unaffected `R034-CASE-060` runs.. By joining Spark stage task-time spread, max/median shuffle bytes, spill metrics, and skewed key frequencies across the affected and unaffected conditions, the investigation directly tests the recorded mechanism: the long tail is caused by uneven shuffle partition sizes, not only an insufficient total number of executors. It also sets a measurable baseline for run the aggregation before and after the selected partitioning change and compare long-tail duration and spill.
Incorrect Answers
B:Compare the saved configuration with the signed change ticket for `account_statements` while omitting source and output records. A desired configuration and approval ticket are not runtime proof. In this case they omit Spark stage task-time spread, max/median shuffle bytes, spill metrics, and skewed key frequencies, the direct means of distinguishing the long tail is caused by uneven shuffle partition sizes, not only an insufficient total number of executors.
C:Examine only a normal successful run of `account_statements` and trend Spark stage task-time spread without recreating the fault. A healthy-run trend can be useful, but it does not expose the long tail is caused by uneven shuffle partition sizes, not only an insufficient total number of executors. The failing-path Spark stage task-time spread, max/median shuffle bytes, spill metrics, and skewed key frequencies has to be correlated before attributing cause.
D:Review aggregate cluster availability and job completion for `account_statements` without correlating the observed error path. A successful aggregate platform state can coexist with the long tail is caused by uneven shuffle partition sizes, not only an insufficient total number of executors. This check never observes the affected `account_statements` records or their causal path.
Question 297
At a census statistics agency, a Spark SQL aggregation creates a few oversized shuffle partitions while most tasks finish quickly. The dataset `account_statements` is in scope, and downstream dashboards cannot accept fabricated values. Which implementation best resolves the stated defect without breaking the business constraint?
- Inspect skew and exchange metrics, then tune partitioning or salting/repartitioning for the actual key distribution and remeasure
- Increase driver memory without inspecting executor shuffle tasks; use that approach for `account_statements`.
- Raise auto-broadcast threshold for an oversized fact table; use that approach for `account_statements`.
- Reduce shuffle partitions drastically for every query; use that approach for `account_statements`.
Explanation
Correct Answer
A:Inspect skew and exchange metrics, then tune partitioning or salting/repartitioning for the actual key distribution and remeasure. This directly addresses the long tail is caused by uneven shuffle partition sizes, not only an insufficient total number of executors. The design should be accepted only after run the aggregation before and after the selected partitioning change and compare long-tail duration and spill; the related operational boundary is to change one evidence-backed distribution setting rather than increasing all compute knobs blindly.
Incorrect Answers
B:Increase driver memory without inspecting executor shuffle tasks; use that approach for `account_statements`. Driver heap growth does not rebalance skewed executor partitions. For `account_statements`, this cannot remove the causal limitation: the long tail is caused by uneven shuffle partition sizes, not only an insufficient total number of executors.
C:Raise auto-broadcast threshold for an oversized fact table; use that approach for `account_statements`. Broadcasting a large fact may cause memory pressure instead of balancing the aggregation. For `account_statements`, this cannot remove the causal limitation: the long tail is caused by uneven shuffle partition sizes, not only an insufficient total number of executors.
D:Reduce shuffle partitions drastically for every query; use that approach for `account_statements`. Fewer larger partitions can worsen spills and long straggling tasks. For `account_statements`, this cannot remove the causal limitation: the long tail is caused by uneven shuffle partition sizes, not only an insufficient total number of executors.
Question 298
At a census statistics agency, a Spark SQL aggregation creates a few oversized shuffle partitions while most tasks finish quickly. The dataset `account_statements` is in scope, and downstream dashboards cannot accept fabricated values. A teammate proposes: "Raise auto-broadcast threshold for an oversized fact table." Which technical limitation is the decisive reason not to approve that specific proposal?
- The dashboard cached the prior result; no DataFrame operation or underlying data requires correction.
- Fewer larger partitions can worsen spills and long straggling tasks. This would apply to another option for `account_statements`.
- Broadcasting a large fact may cause memory pressure instead of balancing the aggregation. The proposal does not repair `account_statements`.
- Driver heap growth does not rebalance skewed executor partitions. This would apply to an alternative change for `account_statements`.
Explanation
Correct Answer
C:Broadcasting a large fact may cause memory pressure instead of balancing the aggregation. The proposal does not repair `account_statements`. This identifies the known limitation of the exact proposed action. Broadcasting a large fact may cause memory pressure instead of balancing the aggregation. The underlying problem remains the long tail is caused by uneven shuffle partition sizes, not only an insufficient total number of executors; Inspect skew and exchange metrics, then tune partitioning or salting/repartitioning for the actual key distribution and remeasure instead targets that failure.
Incorrect Answers
A:The dashboard cached the prior result; no DataFrame operation or underlying data requires correction. The case provides a concrete failing data or control path; a mere alternate theory does not address the long tail is caused by uneven shuffle partition sizes, not only an insufficient total number of executors.
B:Fewer larger partitions can worsen spills and long straggling tasks. This would apply to another option for `account_statements`. This limitation belongs to the different alternative "Reduce shuffle partitions drastically for every query". The stem calls for the specific failure of "Raise auto-broadcast threshold for an oversized fact table".
D:Driver heap growth does not rebalance skewed executor partitions. This would apply to an alternative change for `account_statements`. This failure applies to the separate proposal "Increase driver memory without inspecting executor shuffle tasks".
Question 299
At a census statistics agency, a Spark SQL aggregation creates a few oversized shuffle partitions while most tasks finish quickly. The dataset `account_statements` is in scope, and downstream dashboards cannot accept fabricated values. Which bounded tradeoff would preserve the core outcome and control operational risk?
- Raise auto-broadcast threshold for an oversized fact table; monitor results only after launch of `account_statements`.
- Reduce shuffle partitions drastically for every query; monitor results only after launch of `account_statements`.
- Change one evidence-backed distribution setting rather than increasing all compute knobs blindly
- Increase driver memory without inspecting executor shuffle tasks; monitor results only after launch of `account_statements`.
Explanation
Correct Answer
C:Change one evidence-backed distribution setting rather than increasing all compute knobs blindly. This explicitly bounds the compromise while still addressing the long tail is caused by uneven shuffle partition sizes, not only an insufficient total number of executors. Acceptance relies on run the aggregation before and after the selected partitioning change and compare long-tail duration and spill rather than simply declaring the change complete.
Incorrect Answers
A:Raise auto-broadcast threshold for an oversized fact table; monitor results only after launch of `account_statements`. Broadcasting a large fact may cause memory pressure instead of balancing the aggregation. Monitoring after deployment does not make this approach a safe bounded tradeoff for `account_statements`.
B:Reduce shuffle partitions drastically for every query; monitor results only after launch of `account_statements`. Fewer larger partitions can worsen spills and long straggling tasks. Monitoring after deployment does not make this approach a safe bounded tradeoff for `account_statements`.
D:Increase driver memory without inspecting executor shuffle tasks; monitor results only after launch of `account_statements`. Driver heap growth does not rebalance skewed executor partitions. Monitoring after deployment does not make this approach a safe bounded tradeoff for `account_statements`.
Question 300
At a census statistics agency, a Spark SQL aggregation creates a few oversized shuffle partitions while most tasks finish quickly. The dataset `account_statements` is in scope, and downstream dashboards cannot accept fabricated values. Before release, which acceptance test would actually demonstrate that the root problem is controlled?
- Review a platform-level availability report for `account_statements` without testing whether the long tail is caused by uneven shuffle partition sizes, not only an insufficient total number of executors.
- Check that the intended configuration exists for `account_statements` and that a healthy run succeeded, without inducing the earlier failure.
- Run the aggregation before and after the selected partitioning change and compare long-tail duration and spill
- Run only a normal, unchanged `account_statements` sample and review Spark stage task-time spread.
Explanation
Correct Answer
C:Run the aggregation before and after the selected partitioning change and compare long-tail duration and spill. This reproduces the scenario-dependent condition, verifies Spark stage task-time spread, max/median shuffle bytes, spill metrics, and skewed key frequencies, and directly tests whether the long tail is caused by uneven shuffle partition sizes, not only an insufficient total number of executors is resolved. Observed results, not intent, determine approval.
Incorrect Answers
A:Review a platform-level availability report for `account_statements` without testing whether the long tail is caused by uneven shuffle partition sizes, not only an insufficient total number of executors. Aggregate availability does not establish the behavior of affected `account_statements` records. It cannot replace the direct exercise: run the aggregation before and after the selected partitioning change and compare long-tail duration and spill.
B:Check that the intended configuration exists for `account_statements` and that a healthy run succeeded, without inducing the earlier failure. Installed configuration and a healthy run are insufficient to prove the failure mode is gone. The original issue was the long tail is caused by uneven shuffle partition sizes, not only an insufficient total number of executors.
D:Run only a normal, unchanged `account_statements` sample and review Spark stage task-time spread. A normal sample does not challenge the long tail is caused by uneven shuffle partition sizes, not only an insufficient total number of executors.