Certified Data Engineer Associate Practice Test 29
Question 280
At a hospital analytics group, multiple deliveries of the same event ID inflate a daily fact table even though each ID represents one logical event. The dataset `payment_decisions` is in scope, and downstream dashboards cannot accept fabricated values. Before release, which acceptance test would actually demonstrate that the root problem is controlled?
- Check that the intended configuration exists for `payment_decisions` and that a healthy run succeeded, without inducing the earlier failure.
- Run only a normal, unchanged `payment_decisions` sample and review duplicate event ID frequency.
- Review a platform-level availability report for `payment_decisions` without testing whether raw row counts include replays and a blind distinct may retain conflicting versions for the same key.
- Replay an older and newer version of one ID plus exact duplicates and assert the latest logical event is counted once
Explanation
Correct Answer
D:Replay an older and newer version of one ID plus exact duplicates and assert the latest logical event is counted once. This reproduces the scenario-dependent condition, verifies duplicate event ID frequency, event version ordering, raw row counts, and curated unique ID counts, and directly tests whether raw row counts include replays and a blind distinct may retain conflicting versions for the same key is resolved. Observed results, not intent, determine approval.
Incorrect Answers
A:Check that the intended configuration exists for `payment_decisions` and that a healthy run succeeded, without inducing the earlier failure. Installed configuration and a healthy run are insufficient to prove the failure mode is gone. The original issue was raw row counts include replays and a blind distinct may retain conflicting versions for the same key.
B:Run only a normal, unchanged `payment_decisions` sample and review duplicate event ID frequency. A normal sample does not challenge raw row counts include replays and a blind distinct may retain conflicting versions for the same key.
C:Review a platform-level availability report for `payment_decisions` without testing whether raw row counts include replays and a blind distinct may retain conflicting versions for the same key. Aggregate availability does not establish the behavior of affected `payment_decisions` records. It cannot replace the direct exercise: replay an older and newer version of one ID plus exact duplicates and assert the latest logical event is counted once.
Question 281
At a consumer-lending business, an analyst must estimate unique visitor counts quickly on a very large clickstream, and exact distinct is unnecessarily expensive. The dataset `course_enrollments` is in scope, and raw records must remain available for replay. What evidence should the engineer collect to establish the cause rather than a coincidental symptom?
- Review aggregate cluster availability and job completion for `course_enrollments` without correlating the observed error path.
- Compare the saved configuration with the signed change ticket for `course_enrollments` while omitting source and output records.
- Correlate sample true distinct count, estimate error across affected and unaffected `R034-CASE-057` runs.
- Examine only a normal successful run of `course_enrollments` and trend sample true distinct count without recreating the fault.
Explanation
Correct Answer
C:Correlate sample true distinct count, estimate error across affected and unaffected `R034-CASE-057` runs.. By joining sample true distinct count, estimate error, shuffle cost, and query duration across the affected and unaffected conditions, the investigation directly tests the recorded mechanism: an exact distinct aggregation may require more shuffle/state than the approximate statistic needs. It also sets a measurable baseline for compare approximate and exact distinct on a representative sample against the accepted error bound.
Incorrect Answers
A:Review aggregate cluster availability and job completion for `course_enrollments` without correlating the observed error path. A successful aggregate platform state can coexist with an exact distinct aggregation may require more shuffle/state than the approximate statistic needs. This check never observes the affected `course_enrollments` records or their causal path.
B:Compare the saved configuration with the signed change ticket for `course_enrollments` while omitting source and output records. A desired configuration and approval ticket are not runtime proof. In this case they omit sample true distinct count, estimate error, shuffle cost, and query duration, the direct means of distinguishing an exact distinct aggregation may require more shuffle/state than the approximate statistic needs.
D:Examine only a normal successful run of `course_enrollments` and trend sample true distinct count without recreating the fault. A healthy-run trend can be useful, but it does not expose an exact distinct aggregation may require more shuffle/state than the approximate statistic needs. The failing-path sample true distinct count, estimate error, shuffle cost, and query duration has to be correlated before attributing cause.
Question 282
At a consumer-lending business, an analyst must estimate unique visitor counts quickly on a very large clickstream, and exact distinct is unnecessarily expensive. The dataset `course_enrollments` is in scope, and raw records must remain available for replay. Which platform configuration or data-engineering change is most defensible?
- Run exact count distinct on every exploratory refresh regardless of budget; use that approach for `course_enrollments`.
- Use approximate count distinct for exploratory distinct estimation and state its accuracy tradeoff
- Use count(*) as a unique visitor measure; use that approach for `course_enrollments`.
- Average the numeric visitor identifiers to estimate unique visitors; use that approach for `course_enrollments`.
Explanation
Correct Answer
B:Use approximate count distinct for exploratory distinct estimation and state its accuracy tradeoff. This directly addresses an exact distinct aggregation may require more shuffle/state than the approximate statistic needs. The design should be accepted only after compare approximate and exact distinct on a representative sample against the accepted error bound; the related operational boundary is to accept quantified estimation error to lower compute cost for nonfinancial exploratory metrics.
Incorrect Answers
A:Run exact count distinct on every exploratory refresh regardless of budget; use that approach for `course_enrollments`. The exact calculation ignores the explicit approximation and latency allowance. For `course_enrollments`, this cannot remove the causal limitation: an exact distinct aggregation may require more shuffle/state than the approximate statistic needs.
C:Use count(*) as a unique visitor measure; use that approach for `course_enrollments`. Event count counts repeat visits by the same visitor. For `course_enrollments`, this cannot remove the causal limitation: an exact distinct aggregation may require more shuffle/state than the approximate statistic needs.
D:Average the numeric visitor identifiers to estimate unique visitors; use that approach for `course_enrollments`. An arithmetic mean of identifiers has no relation to unique cardinality. For `course_enrollments`, this cannot remove the causal limitation: an exact distinct aggregation may require more shuffle/state than the approximate statistic needs.
Question 283
At a consumer-lending business, an analyst must estimate unique visitor counts quickly on a very large clickstream, and exact distinct is unnecessarily expensive. The dataset `course_enrollments` is in scope, and raw records must remain available for replay. A teammate proposes: "Use count(*) as a unique visitor measure." Which technical limitation is the decisive reason not to approve that specific proposal?
- Event count counts repeat visits by the same visitor. The proposal does not repair `course_enrollments`.
- The dashboard cached the prior result; no DataFrame operation or underlying data requires correction.
- An arithmetic mean of identifiers has no relation to unique cardinality. This would apply to an alternative change for `course_enrollments`.
- The exact calculation ignores the explicit approximation and latency allowance. This would apply to another option for `course_enrollments`.
Explanation
Correct Answer
A:Event count counts repeat visits by the same visitor. The proposal does not repair `course_enrollments`. This identifies the known limitation of the exact proposed action. Event count counts repeat visits by the same visitor. The underlying problem remains an exact distinct aggregation may require more shuffle/state than the approximate statistic needs; Use approximate count distinct for exploratory distinct estimation and state its accuracy tradeoff instead targets that failure.
Incorrect Answers
B:The dashboard cached the prior result; no DataFrame operation or underlying data requires correction. The case provides a concrete failing data or control path; a mere alternate theory does not address an exact distinct aggregation may require more shuffle/state than the approximate statistic needs.
C:An arithmetic mean of identifiers has no relation to unique cardinality. This would apply to an alternative change for `course_enrollments`. This failure applies to the separate proposal "Average the numeric visitor identifiers to estimate unique visitors".
D:The exact calculation ignores the explicit approximation and latency allowance. This would apply to another option for `course_enrollments`. This limitation belongs to the different alternative "Run exact count distinct on every exploratory refresh regardless of budget". The stem calls for the specific failure of "Use count(*) as a unique visitor measure".
Question 284
At a consumer-lending business, an analyst must estimate unique visitor counts quickly on a very large clickstream, and exact distinct is unnecessarily expensive. The dataset `course_enrollments` is in scope, and raw records must remain available for replay. What risk-and-cost compromise should the owner accept instead of the proposed shortcuts?
- Average the numeric visitor identifiers to estimate unique visitors; monitor results only after launch of `course_enrollments`.
- Accept quantified estimation error to lower compute cost for nonfinancial exploratory metrics
- Use count(*) as a unique visitor measure; monitor results only after launch of `course_enrollments`.
- Run exact count distinct on every exploratory refresh regardless of budget; monitor results only after launch of `course_enrollments`.
Explanation
Correct Answer
B:Accept quantified estimation error to lower compute cost for nonfinancial exploratory metrics. This explicitly bounds the compromise while still addressing an exact distinct aggregation may require more shuffle/state than the approximate statistic needs. Acceptance relies on compare approximate and exact distinct on a representative sample against the accepted error bound rather than simply declaring the change complete.
Incorrect Answers
A:Average the numeric visitor identifiers to estimate unique visitors; monitor results only after launch of `course_enrollments`. An arithmetic mean of identifiers has no relation to unique cardinality. Monitoring after deployment does not make this approach a safe bounded tradeoff for `course_enrollments`.
C:Use count(*) as a unique visitor measure; monitor results only after launch of `course_enrollments`. Event count counts repeat visits by the same visitor. Monitoring after deployment does not make this approach a safe bounded tradeoff for `course_enrollments`.
D:Run exact count distinct on every exploratory refresh regardless of budget; monitor results only after launch of `course_enrollments`. The exact calculation ignores the explicit approximation and latency allowance. Monitoring after deployment does not make this approach a safe bounded tradeoff for `course_enrollments`.
Question 285
At a consumer-lending business, an analyst must estimate unique visitor counts quickly on a very large clickstream, and exact distinct is unnecessarily expensive. The dataset `course_enrollments` is in scope, and raw records must remain available for replay. Which direct test could falsify the proposed fix instead of only confirming deployment?
- Run only a normal, unchanged `course_enrollments` sample and review sample true distinct count.
- Check that the intended configuration exists for `course_enrollments` and that a healthy run succeeded, without inducing the earlier failure.
- Review a platform-level availability report for `course_enrollments` without testing whether an exact distinct aggregation may require more shuffle/state than the approximate statistic needs.
- Compare approximate and exact distinct on a representative sample against the accepted error bound
Explanation
Correct Answer
D:Compare approximate and exact distinct on a representative sample against the accepted error bound. This reproduces the scenario-dependent condition, verifies sample true distinct count, estimate error, shuffle cost, and query duration, and directly tests whether an exact distinct aggregation may require more shuffle/state than the approximate statistic needs is resolved. Observed results, not intent, determine approval.
Incorrect Answers
A:Run only a normal, unchanged `course_enrollments` sample and review sample true distinct count. A normal sample does not challenge an exact distinct aggregation may require more shuffle/state than the approximate statistic needs.
B:Check that the intended configuration exists for `course_enrollments` and that a healthy run succeeded, without inducing the earlier failure. Installed configuration and a healthy run are insufficient to prove the failure mode is gone. The original issue was an exact distinct aggregation may require more shuffle/state than the approximate statistic needs.
C:Review a platform-level availability report for `course_enrollments` without testing whether an exact distinct aggregation may require more shuffle/state than the approximate statistic needs. Aggregate availability does not establish the behavior of affected `course_enrollments` records. It cannot replace the direct exercise: compare approximate and exact distinct on a representative sample against the accepted error bound.
Question 286
At a subscription media service, a Spark SQL aggregation creates a few oversized shuffle partitions while most tasks finish quickly. The dataset `fraud_signals` is in scope, and downstream dashboards cannot accept fabricated values. Which correlated observation is the strongest basis for the change decision?
- Review completed run status and aggregate cluster health without record-level correlation
- Compare the current configuration and the signed change approval, without source-row evidence
- Examine only a normal successful run of `fraud_signals` and trend Spark stage task-time spread without recreating the fault.
- Correlate Spark stage task-time spread, max/median shuffle bytes across affected and unaffected `R034-CASE-058` runs.
Explanation
Correct Answer
D:Correlate Spark stage task-time spread, max/median shuffle bytes across affected and unaffected `R034-CASE-058` runs.. By joining Spark stage task-time spread, max/median shuffle bytes, spill metrics, and skewed key frequencies across the affected and unaffected conditions, the investigation directly tests the recorded mechanism: the long tail is caused by uneven shuffle partition sizes, not only an insufficient total number of executors. It also sets a measurable baseline for run the aggregation before and after the selected partitioning change and compare long-tail duration and spill.
Incorrect Answers
A:Review completed run status and aggregate cluster health without record-level correlation. A successful aggregate platform state can coexist with the long tail is caused by uneven shuffle partition sizes, not only an insufficient total number of executors. This check never observes the affected `fraud_signals` records or their causal path.
B:Compare the current configuration and the signed change approval, without source-row evidence. A desired configuration and approval ticket are not runtime proof. In this case they omit Spark stage task-time spread, max/median shuffle bytes, spill metrics, and skewed key frequencies, the direct means of distinguishing the long tail is caused by uneven shuffle partition sizes, not only an insufficient total number of executors.
C:Examine only a normal successful run of `fraud_signals` and trend Spark stage task-time spread without recreating the fault. A healthy-run trend can be useful, but it does not expose the long tail is caused by uneven shuffle partition sizes, not only an insufficient total number of executors. The failing-path Spark stage task-time spread, max/median shuffle bytes, spill metrics, and skewed key frequencies has to be correlated before attributing cause.
Question 287
At a subscription media service, a Spark SQL aggregation creates a few oversized shuffle partitions while most tasks finish quickly. The dataset `fraud_signals` is in scope, and downstream dashboards cannot accept fabricated values. What implementation should the engineer select for the failing workflow?
- Reduce shuffle partitions drastically for every query; use that approach for `fraud_signals`.
- Increase driver memory without inspecting executor shuffle tasks; use that approach for `fraud_signals`.
- Raise auto-broadcast threshold for an oversized fact table; use that approach for `fraud_signals`.
- Inspect skew and exchange metrics, then tune partitioning or salting/repartitioning for the actual key distribution and remeasure
Explanation
Correct Answer
D:Inspect skew and exchange metrics, then tune partitioning or salting/repartitioning for the actual key distribution and remeasure. This directly addresses the long tail is caused by uneven shuffle partition sizes, not only an insufficient total number of executors. The design should be accepted only after run the aggregation before and after the selected partitioning change and compare long-tail duration and spill; the related operational boundary is to change one evidence-backed distribution setting rather than increasing all compute knobs blindly.
Incorrect Answers
A:Reduce shuffle partitions drastically for every query; use that approach for `fraud_signals`. Fewer larger partitions can worsen spills and long straggling tasks. For `fraud_signals`, this cannot remove the causal limitation: the long tail is caused by uneven shuffle partition sizes, not only an insufficient total number of executors.
B:Increase driver memory without inspecting executor shuffle tasks; use that approach for `fraud_signals`. Driver heap growth does not rebalance skewed executor partitions. For `fraud_signals`, this cannot remove the causal limitation: the long tail is caused by uneven shuffle partition sizes, not only an insufficient total number of executors.
C:Raise auto-broadcast threshold for an oversized fact table; use that approach for `fraud_signals`. Broadcasting a large fact may cause memory pressure instead of balancing the aggregation. For `fraud_signals`, this cannot remove the causal limitation: the long tail is caused by uneven shuffle partition sizes, not only an insufficient total number of executors.
Question 288
At a subscription media service, a Spark SQL aggregation creates a few oversized shuffle partitions while most tasks finish quickly. The dataset `fraud_signals` is in scope, and downstream dashboards cannot accept fabricated values. A teammate proposes: "Reduce shuffle partitions drastically for every query." Which technical limitation is the decisive reason not to approve that specific proposal?
- Fewer larger partitions can worsen spills and long straggling tasks. The proposal does not repair `fraud_signals`.
- Broadcasting a large fact may cause memory pressure instead of balancing the aggregation. This would apply to an alternative change for `fraud_signals`.
- Driver heap growth does not rebalance skewed executor partitions. This would apply to another option for `fraud_signals`.
- The dashboard cached the prior result; no DataFrame operation or underlying data requires correction.
Explanation
Correct Answer
A:Fewer larger partitions can worsen spills and long straggling tasks. The proposal does not repair `fraud_signals`. This identifies the known limitation of the exact proposed action. Fewer larger partitions can worsen spills and long straggling tasks. The underlying problem remains the long tail is caused by uneven shuffle partition sizes, not only an insufficient total number of executors; Inspect skew and exchange metrics, then tune partitioning or salting/repartitioning for the actual key distribution and remeasure instead targets that failure.
Incorrect Answers
B:Broadcasting a large fact may cause memory pressure instead of balancing the aggregation. This would apply to an alternative change for `fraud_signals`. This failure applies to the separate proposal "Raise auto-broadcast threshold for an oversized fact table".
C:Driver heap growth does not rebalance skewed executor partitions. This would apply to another option for `fraud_signals`. This limitation belongs to the different alternative "Increase driver memory without inspecting executor shuffle tasks". The stem calls for the specific failure of "Reduce shuffle partitions drastically for every query".
D:The dashboard cached the prior result; no DataFrame operation or underlying data requires correction. The case provides a concrete failing data or control path; a mere alternate theory does not address the long tail is caused by uneven shuffle partition sizes, not only an insufficient total number of executors.
Question 289
At a subscription media service, a Spark SQL aggregation creates a few oversized shuffle partitions while most tasks finish quickly. The dataset `fraud_signals` is in scope, and downstream dashboards cannot accept fabricated values. Which decision preserves the stated data contract while keeping the change scope proportionate?
- Raise auto-broadcast threshold for an oversized fact table
- Reduce shuffle partitions drastically for every query
- Increase driver memory without inspecting executor shuffle tasks; monitor results only after launch of `fraud_signals`.
- Change one evidence-backed distribution setting rather than increasing all compute knobs blindly
Explanation
Correct Answer
D:Change one evidence-backed distribution setting rather than increasing all compute knobs blindly. This explicitly bounds the compromise while still addressing the long tail is caused by uneven shuffle partition sizes, not only an insufficient total number of executors. Acceptance relies on run the aggregation before and after the selected partitioning change and compare long-tail duration and spill rather than simply declaring the change complete.
Incorrect Answers
A:Raise auto-broadcast threshold for an oversized fact table. Broadcasting a large fact may cause memory pressure instead of balancing the aggregation. Monitoring after deployment does not make this approach a safe bounded tradeoff for `fraud_signals`.
B:Reduce shuffle partitions drastically for every query. Fewer larger partitions can worsen spills and long straggling tasks. Monitoring after deployment does not make this approach a safe bounded tradeoff for `fraud_signals`.
C:Increase driver memory without inspecting executor shuffle tasks; monitor results only after launch of `fraud_signals`. Driver heap growth does not rebalance skewed executor partitions. Monitoring after deployment does not make this approach a safe bounded tradeoff for `fraud_signals`.
Question 290
At a subscription media service, a Spark SQL aggregation creates a few oversized shuffle partitions while most tasks finish quickly. The dataset `fraud_signals` is in scope, and downstream dashboards cannot accept fabricated values. What verification provides evidence of correct behavior under the original adverse condition?
- Check that the intended configuration exists for `fraud_signals` and that a healthy run succeeded, without inducing the earlier failure.
- Run only a normal, unchanged `fraud_signals` sample and review Spark stage task-time spread.
- Run the aggregation before and after the selected partitioning change and compare long-tail duration and spill
- Review a platform-level availability report for `fraud_signals` without testing whether the long tail is caused by uneven shuffle partition sizes, not only an insufficient total number of executors.
Explanation
Correct Answer
C:Run the aggregation before and after the selected partitioning change and compare long-tail duration and spill. This reproduces the scenario-dependent condition, verifies Spark stage task-time spread, max/median shuffle bytes, spill metrics, and skewed key frequencies, and directly tests whether the long tail is caused by uneven shuffle partition sizes, not only an insufficient total number of executors is resolved. Observed results, not intent, determine approval.
Incorrect Answers
A:Check that the intended configuration exists for `fraud_signals` and that a healthy run succeeded, without inducing the earlier failure. Installed configuration and a healthy run are insufficient to prove the failure mode is gone. The original issue was the long tail is caused by uneven shuffle partition sizes, not only an insufficient total number of executors.
B:Run only a normal, unchanged `fraud_signals` sample and review Spark stage task-time spread. A normal sample does not challenge the long tail is caused by uneven shuffle partition sizes, not only an insufficient total number of executors.
D:Review a platform-level availability report for `fraud_signals` without testing whether the long tail is caused by uneven shuffle partition sizes, not only an insufficient total number of executors. Aggregate availability does not establish the behavior of affected `fraud_signals` records. It cannot replace the direct exercise: run the aggregation before and after the selected partitioning change and compare long-tail duration and spill.