Visit here for our full Databricks Certified Associate Developer for Apache Spark exam dumps and practice test questions.
Question 121:
What is the purpose of the explode_outer() function?
A) To flatten arrays
B) To explode arrays preserving rows with null or empty arrays
C) To remove outer elements
D) To expand maps
Answer: B
Explanation:
The explode_outer() function flattens array or map columns into multiple rows, similar to explode(), but with a crucial difference: it preserves rows where the array or map column is null or empty by creating a single row with null values. Regular explode() drops these rows entirely, which can cause data loss when you need to track all original rows including those without array/map values.
Usage example: df.select(col(“id”), explode_outer(col(“items”))) creates one row per item in the items array, but for rows where items is null or empty, it creates one row with null in the exploded column. This behavior is similar to SQL’s LEFT OUTER JOIN semantics, ensuring no original rows are lost. The function is particularly valuable when the presence or absence of array elements has semantic meaning that must be preserved.
The distinction between explode() and explode_outer() is critical for maintaining data integrity. If you’re analyzing customer purchases and some customers have empty purchase arrays, explode() would remove those customers from results, potentially skewing analysis. Explode_outer() keeps them, allowing accurate customer counts and proper representation of customers with no purchases. This is essential for complete data analysis and accurate metrics.
Common use cases include preserving all entities when exploding associated collections, maintaining referential integrity during array flattening, analyzing populations that may have zero occurrences of array elements, and ensuring complete data representation in reports. Always consider whether rows with empty/null arrays have business meaning that should be preserved. If so, use explode_outer() instead of explode(). The function works with both arrays and maps, creating null values for the key and value columns when maps are null or empty. Understanding this distinction prevents unintentional data loss during array processing.
Question 122:
Which method is used to get column names of a DataFrame?
A) getColumns()
B) columns
C) columnNames()
D) listColumns()
Answer: B
Explanation:
The columns property returns a list of column names from a DataFrame as an array of strings. This property provides programmatic access to schema information, enabling dynamic operations based on available columns, validation of expected columns, and flexible data processing logic that adapts to different schemas. Unlike printSchema() which displays formatted schema information, the columns property returns data that can be used in code.
Usage example: column_list = df.columns returns something like [‘id’, ‘name’, ‘age’, ‘salary’]. This is useful for dynamic column selection, validation, iteration over columns, or generating operations programmatically. You can check if specific columns exist: if ‘age’ in df.columns, select subsets dynamically: df.select([col(c) for c in df.columns if c.startswith(‘sales_’)]), or perform operations on all columns programmatically.
The columns property is particularly valuable when writing generic data processing functions that work with varying schemas, when validating data against expected schemas, or when implementing dynamic transformations based on column presence. It’s often combined with schema property which provides full type information, not just names. Columns returns names in the order they appear in the DataFrame.
Common use cases include schema validation before processing, dynamically selecting columns based on naming patterns, iterating over columns to apply transformations, generating documentation or metadata about datasets, and implementing generic processing functions. For type information, use df.schema which returns a StructType with complete schema details. The columns property is a simple list operation with no computation overhead. Understanding how to access and manipulate schema information programmatically is essential for building flexible, maintainable data pipelines that handle schema variations gracefully. Combine with drop(), select(), and withColumn() for dynamic schema management.
Question 123:
What is the purpose of the countDistinct() function?
A) To count unique rows
B) To count distinct values in specified columns
C) To count total rows
D) To estimate unique counts
Answer: B
Explanation:
The countDistinct() function computes the exact count of distinct values in one or more specified columns, typically used within aggregations. This function is essential for understanding data cardinality, identifying unique entities, and measuring diversity in datasets. Unlike approx_count_distinct() which provides fast approximate counts, countDistinct() calculates exact results by identifying and counting all unique values.
Usage example: df.agg(countDistinct(“user_id”)) counts unique users in the dataset. You can count distinct combinations across multiple columns: df.agg(countDistinct(“user_id”, “product_id”)) counts unique user-product pairs. The function is often combined with groupBy() for per-group distinct counts: df.groupBy(“category”).agg(countDistinct(“product_id”)) counts unique products per category.
CountDistinct() requires shuffling data to bring duplicate values together for identification, making it potentially expensive on large datasets with high cardinality. The operation must track all unique values, requiring memory proportional to the number of distinct values. For very large cardinalities or when approximate counts suffice, consider approx_count_distinct() which uses less memory and computes faster.
Common use cases include counting unique customers, products, or transactions, measuring diversity in categorical variables, validating uniqueness constraints, and computing metrics for dashboards or reports. Distinct counting is fundamental to many analytical queries and key performance indicators. The function handles null values by treating them as a distinct value to count. For finding actual distinct values rather than just counting them, use distinct() or dropDuplicates(). Understanding the performance implications of exact distinct counts helps in choosing between exact and approximate methods. For exploratory analysis or monitoring where perfect accuracy isn’t critical, approximate counts offer significant performance benefits.
Question 124:
Which function is used to add days to a date column?
A) addDays()
B) date_add()
C) add_days()
D) plusDays()
Answer: B
Explanation:
The date_add() function adds a specified number of days to date or timestamp columns, returning a new date. This function is fundamental for date arithmetic, enabling calculations like future dates, date ranges, and time-based operations. It accepts two parameters: the date column and the number of days to add, which can be positive for future dates or negative for past dates.
Usage example: df.withColumn(“due_date”, date_add(col(“order_date”), 30)) calculates a date 30 days after the order date. You can use negative values for subtraction: date_add(col(“date”), -7) goes back 7 days. The function handles month boundaries, leap years, and all calendar complexities automatically, ensuring correct date calculations without manual logic.
Date_add() is essential for business logic involving timeframes, such as calculating expiration dates, due dates, subscription renewals, or aging analysis. It’s commonly combined with other date functions for complex temporal logic. For example, comparing whether events occurred within specific timeframes, filtering data by date ranges calculated from reference dates, or creating time-based features for analysis. The function works seamlessly with date and timestamp types.
Common use cases include calculating subscription end dates, determining payment due dates, analyzing data freshness by comparing dates to current date plus/minus days, implementing retention analysis, and creating time-based cohorts. Related functions include date_sub() which subtracts days (though date_add with negative values achieves the same), months_between() for month calculations, and add_months() for month arithmetic. For more complex date calculations involving specific business calendars or excluding weekends, you may need additional logic or UDFs. Understanding date arithmetic functions is crucial for temporal data processing and business logic implementation in data pipelines.
Question 125:
What is the purpose of the from_json() function?
A) To read JSON files
B) To parse JSON strings into structured columns
C) To convert DataFrames to JSON
D) To validate JSON format
Answer: B
Explanation:
The from_json() function parses JSON-formatted string columns into structured columns (StructType, ArrayType, or MapType), enabling extraction of nested data from JSON strings stored in DataFrame columns. This function is essential when working with JSON data embedded in string fields, such as API responses, log entries, or semi-structured data stored as text. It transforms flat string columns into rich nested structures that can be queried and manipulated.
Usage example: df.withColumn(“parsed”, from_json(col(“json_string”), schema)) parses the json_string column using the provided schema, creating a structured column with nested fields. You must specify the schema either as a DDL string (“structname:string,age:int”) or as a StructType object. The schema defines expected structure and types, ensuring proper parsing. Fields not matching the schema become null.
From_json() is the complement to to_json() which converts structured columns to JSON strings. Together, they enable bidirectional transformation between structured and string-based JSON representations. The function handles complex nested structures including arrays of structs, maps, and deeply nested objects. After parsing, you can access nested fields using dot notation or getField() methods.
Common use cases include parsing JSON stored in database text columns, extracting structured data from API response strings, processing log entries containing JSON payloads, and working with semi-structured data from various sources. When dealing with variable JSON schemas, you can use schema inference by reading sample data, though explicit schemas are more reliable for production. The function integrates with Spark’s nested data handling capabilities, enabling complex queries on parsed JSON structures. Understanding JSON parsing is critical for modern data engineering where JSON is ubiquitous in APIs, logs, and data interchange formats.
Question 126:
Which method is used to compute the sum of a column?
A) total()
B) sum()
C) aggregate()
D) calculate()
Answer: B
Explanation:
The sum() function computes the sum of values in a numeric column, typically used within aggregations. This fundamental aggregation function is essential for calculating totals, metrics, and summaries across datasets. It can be used with agg() after groupBy() for per-group sums, or directly on entire DataFrames for overall totals. The function handles null values by ignoring them in calculations.
Usage example: df.agg(sum(“amount”)) calculates the total of the amount column across all rows. For grouped aggregations: df.groupBy(“category”).agg(sum(“sales”).alias(“total_sales”)) computes sales totals per category. You can sum multiple columns simultaneously: df.agg(sum(“sales”), sum(“profit”)). The function works with all numeric types including integers, decimals, and floating-point numbers.
Sum() is one of several built-in aggregation functions alongside avg(), count(), min(), max(), and others. These functions are optimized by Spark’s Catalyst optimizer and execute efficiently in distributed fashion across cluster partitions. Each partition computes partial sums, which are then combined for the final result. This distributed approach enables summing massive datasets that don’t fit on single machines.
Common use cases include calculating revenue totals, aggregating quantities, computing financial metrics, generating summary statistics, and preparing data for reporting or dashboards. Sum() handles edge cases like empty groups (returning null) and overflow for very large sums. For more complex aggregations requiring custom logic, use aggregate functions or window functions. Understanding aggregation functions is fundamental to data analysis in Spark. Combine sum() with groupBy(), window functions, and filters for sophisticated analytical queries. The function integrates seamlessly with other DataFrame operations, enabling complex calculations in single queries.
Question 127:
What is the purpose of the lag() window function?
A) To slow down processing
B) To access values from previous rows within partitions
C) To delay execution
D) To measure latency
Answer: B
Explanation:
The lag() window function accesses values from previous rows within a window partition, enabling row-to-row comparisons and sequential analysis. This function is essential for time-series analysis, calculating changes between consecutive rows, and implementing business logic that depends on previous values. It returns the value from a specified number of rows before the current row, or null if no such row exists.
Usage example: df.withColumn(“prev_value”, lag(“value”, 1).over(Window.partitionBy(“id”).orderBy(“date”))) accesses the value from the previous row within each id partition, ordered by date. The second parameter specifies offset (default 1), and an optional third parameter provides a default value instead of null when no previous row exists. Lag() requires a window specification defining partitioning and ordering.
The function is particularly useful for calculating differences between consecutive measurements, identifying changes or trends, comparing current values with previous periods, and implementing complex sequential logic. Common applications include computing day-over-day changes, calculating moving differences, detecting value changes, and analyzing sequential patterns. The complementary lead() function accesses future rows instead of previous ones.
Common use cases include financial analysis comparing period-over-period metrics, IoT sensor data analyzing value changes over time, customer behavior tracking session sequences, and time-series forecasting using historical patterns. Window functions like lag() maintain DataFrame row count while adding computed columns, unlike aggregations that reduce rows. Understanding window functions is crucial for advanced analytics requiring row-context awareness. Combine lag() with conditional logic to implement complex business rules like detecting increases, flagging anomalies, or computing cumulative changes. The function requires careful window specification with appropriate partitioning and ordering for correct results.
Question 128:
Which method is used to show DataFrame contents?
A) display()
B) show()
C) print()
D) view()
Answer: B
Explanation:
The show() method displays DataFrame contents in a formatted table in the console, making it invaluable for data inspection during development and debugging. This action triggers computation and prints a specified number of rows in human-readable tabular format. By default, show() displays 20 rows and truncates long strings to 20 characters, though both parameters are configurable. Unlike collect() which returns data to the driver as objects, show() only prints formatted output.
Usage example: df.show() displays the first 20 rows. You can specify row count: df.show(50) shows 50 rows. To display full column values without truncation: df.show(truncate=False). For vertical display of rows: df.show(vertical=True), useful for wide tables with many columns. The method handles complex nested structures by formatting them readably.
Show() is designed for interactive development and debugging, providing quick visibility into data without programming collection and iteration logic. It’s efficient because it only computes and retrieves the specified number of rows, not the entire dataset. However, it’s an action that triggers execution, so calling show() multiple times recomputes results unless data is cached. The formatted output makes it easy to spot data quality issues, verify transformations, and understand data structure.
Common use cases include verifying data loads after reading files, checking transformation results during development, inspecting sample data for exploration, debugging data quality issues, and validating filter or aggregation logic. For programmatic access to data, use collect(), take(), or head() instead. Show() is not suitable for production data processing but essential for development workflows. Understanding when to use show() versus other data retrieval methods helps balance interactive exploration with efficient data processing. The method is particularly useful in notebook environments where visualizing intermediate results aids iterative development.
Question 129:
What is the purpose of the array() function?
A) To create array columns from multiple columns
B) To check array types
C) To sort arrays
D) To count array elements
Answer: A
Explanation:
The array() function creates array columns by combining multiple columns or literal values into a single array-typed column. This function enables creating structured collections within DataFrames, useful for grouping related values, preparing data for array operations, or creating nested structures. Each row’s array contains values from the specified columns for that row, creating arrays of consistent structure across all rows.
Usage example: df.withColumn(“coordinates”, array(col(“latitude”), col(“longitude”))) creates an array column containing both coordinate values. You can combine multiple columns: array(col(“q1_sales”), col(“q2_sales”), col(“q3_sales”), col(“q4_sales”)) creates quarterly sales arrays. The function also accepts literal values: array(lit(1), lit(2), lit(3)) creates an array with constant values. All elements must have compatible types.
Array columns enable powerful operations through array functions like array_contains(), array_intersect(), explode(), and aggregate operations on array elements. Creating arrays is often a preprocessing step for complex analytics, feature engineering for machine learning, or preparing data for systems expecting array-formatted inputs. Arrays can contain any data type including structs, enabling arrays of complex objects.
Common use cases include grouping related measurements or attributes, creating feature vectors for machine learning, preparing data for JSON output with nested arrays, implementing multi-valued attributes, and organizing time-series data points. The inverse operation is selecting individual array elements using getItem() or bracket notation, or exploding arrays into separate rows. Understanding array creation and manipulation is important for working with nested data structures and semi-structured schemas. Combine array() with other array functions for comprehensive collection operations within DataFrames. Array columns are efficiently stored in Parquet and other columnar formats.
Question 130:
Which function is used to replace null values with specific values?
A) replaceNull()
B) coalesce()
C) nvl()
D) Both B and C
Answer: D
Explanation:
Both coalesce() and nvl() functions can replace null values with specific values in Spark SQL. The coalesce() function returns the first non-null value from a list of columns or expressions, while nvl() specifically handles two arguments, returning the second if the first is null. Both are valuable for null handling, with coalesce() being more flexible for multiple fallback options.
Usage example: df.withColumn(“value”, coalesce(col(“primary_value”), col(“backup_value”), lit(0))) uses primary_value if not null, otherwise backup_value, otherwise 0. The nvl() function is simpler for two-value scenarios: nvl(col(“value”), lit(0)) returns value if not null, otherwise 0. Coalesce() can chain multiple columns, checking each in order until finding a non-null value.
These functions differ from fillna() method which operates on entire DataFrames or column subsets, replacing nulls based on column names. Coalesce() and nvl() work at the expression level within transformations, enabling column-specific logic and combinations. They’re particularly useful in select() statements or complex expressions where you need inline null handling. The functions preserve data types and work with any comparable types.
Common use cases include providing default values for missing data, implementing fallback logic when primary data sources lack values, consolidating data from multiple sources with overlapping coverage, and ensuring calculations don’t fail due to nulls. These functions are more flexible than fillna() for complex scenarios requiring different defaults per row based on conditions. Understanding multiple null handling approaches enables choosing the right tool for each scenario. Coalesce() is more powerful for complex fallback chains, while nvl() is simpler for basic null replacement. Both are optimized by Catalyst and perform efficiently in large-scale data processing.
Question 131:
What is the purpose of the grouping() function in cube and rollup operations?
A) To create groups
B) To identify aggregation levels by distinguishing nulls
C) To count groups
D) To validate grouping
Answer: B
Explanation:
The grouping() function identifies which aggregation level a row represents in cube() or rollup() operations by distinguishing between actual null values in data and nulls representing aggregation levels. When cube() or rollup() creates subtotal rows, they use null to indicate aggregated dimensions. Grouping() returns 1 if the column is aggregated (null represents aggregation level) and 0 if it contains actual data, enabling proper interpretation of results.
Usage example: df.cube(“region”, “product”).agg(sum(“sales”), grouping(“region”), grouping(“product”)) adds indicators showing which columns are aggregated in each result row. A grand total row where both columns are aggregated returns 1 for both grouping calls. Partial aggregations return mixed values. This distinction is crucial because cube/rollup results contain nulls with different meanings: actual null data versus aggregation indicators.
Without grouping(), it’s impossible to distinguish subtotal rows from rows with legitimate null dimension values. This ambiguity makes results difficult to interpret and process. Grouping() provides metadata about aggregation structure, enabling correct filtering, labeling, and processing of multi-dimensional aggregation results. The function only makes sense within cube() or rollup() contexts where aggregation levels exist.
Common use cases include creating reports with subtotal labels, filtering for specific aggregation levels, implementing OLAP cube queries, and processing hierarchical aggregations. Combine grouping() with CASE expressions to label aggregation levels: CASE WHEN grouping(region)=1 THEN ‘All Regions’ ELSE region END creates readable labels. Understanding grouping() is essential for working with OLAP-style aggregations where multiple aggregation levels coexist in results. The function enables building sophisticated analytical reports and dashboards that present data at multiple granularities. Use grouping() whenever implementing cube or rollup to ensure result interpretability.
Question 132:
Which method is used to read data from a CSV file with custom options?
A) spark.read.csv()
B) spark.load.csv()
C) spark.import.csv()
D) spark.readCSV()
Answer: A
Explanation:
The spark.read.csv() method reads CSV files into DataFrames with extensive options for customizing parsing behavior. This method is part of the DataFrameReader API and supports options for delimiters, headers, schema inference, null handling, date formats, and more. Proper configuration ensures accurate data loading from diverse CSV formats, handling edge cases like quoted fields, escape characters, and multiline records.
Usage example: spark.read.csv(path, header=True, inferSchema=True, sep=”,”) reads with headers and automatic schema inference. You can specify options explicitly: spark.read.option(“header”, “true”).option(“delimiter”, “|”).option(“quote”, ‘”‘).csv(path). Common options include header (whether first row contains column names), inferSchema (automatically detect types), sep (field delimiter), quote (quote character), escape (escape character), nullValue (string representing nulls), and dateFormat (date parsing format).
For production code, explicitly defining schemas is recommended over inference to ensure consistency and avoid inference overhead: spark.read.schema(schema).csv(path). Custom schemas guarantee correct types and handle type inference limitations. Options like mode control error handling: “PERMISSIVE” (default) handles malformed records by setting fields to null, “DROPMALFORMED” drops bad records, and “FAILFAST” throws exceptions on malformed data.
Common use cases include loading data from various CSV sources with different formats, handling CSVs with special characters requiring custom delimiters or quotes, parsing dates with non-standard formats, and managing data quality through appropriate mode settings. Understanding CSV options is crucial for robust data ingestion pipelines. CSV is common but tricky due to format variations, escaping requirements, and encoding issues. Proper configuration prevents data loss and quality problems. Always validate loaded data and monitor for parsing issues using appropriate mode settings.
Question 133:
What is the purpose of the rand() function?
A) To generate random samples
B) To create random values between 0 and 1
C) To randomize data order
D) To select random rows
Answer: B
Explanation:
The rand() function generates random floating-point values between 0.0 (inclusive) and 1.0 (exclusive) for each row in a DataFrame. This function is useful for creating random features, implementing sampling logic, assigning random values, or generating test data. Each row receives an independently generated random value from a uniform distribution. An optional seed parameter enables reproducible random generation.
Usage example: df.withColumn(“random_value”, rand()) adds a column with random values. For reproducibility: df.withColumn(“random_value”, rand(seed=42)) generates the same random sequence across runs when using the same seed. Random values can drive various logic like random splits: df.withColumn(“split”, when(rand() < 0.8, “train”).otherwise(“test”)) creates randomized train/test splits. The function generates different values for each row but the same distribution.
Rand() is commonly combined with other operations for probabilistic logic, random sampling implementations, A/B test assignments, and data shuffling. Unlike sample() which filters rows, rand() adds random values as a column that can drive complex random logic. For random integers instead of floats, combine with other functions: (rand() * 100).cast(“integer”) generates random integers 0-99.
Common use cases include creating train/test/validation splits for machine learning, implementing random sampling without sample() method, generating random features or noise for testing, assigning users randomly to experiment groups, and shuffling data using random ordering. For random selection from predefined values, combine rand() with when() conditionals. Understanding random generation is valuable for statistical sampling, experimentation, and testing workflows. Always use seeds in production for reproducible results when determinism is needed. Random functions introduce non-determinism that can complicate debugging if not properly controlled.
Question 134:
Which aggregation function calculates the average of a column?
A) mean()
B) avg()
C) average()
D) Both A and B
Answer: D
Explanation:
Both mean() and avg() calculate the average (arithmetic mean) of numeric column values, and they are functionally identical in Spark. These aggregation functions sum all values and divide by count, ignoring null values in the calculation. Either function can be used within agg() after groupBy() for grouped averages, or on entire DataFrames for overall averages. The availability of both names accommodates different naming conventions from SQL and statistical terminology.
Usage example: df.agg(avg(“salary”)) or df.agg(mean(“salary”)) both calculate average salary across all rows. For grouped calculations: df.groupBy(“department”).agg(avg(“salary”).alias(“avg_salary”)) computes average salary per department. The functions handle nulls by excluding them from both numerator and denominator, so average calculation is based only on non-null values. Empty groups or columns with all nulls return null.
Average calculations are fundamental to statistical analysis and reporting. They provide central tendency measures essential for understanding data distributions, comparing groups, and summarizing numeric characteristics. The functions work with any numeric data type and return floating-point results. For counts without nulls affecting results, averages provide better insights than sums for normalized comparisons across different group sizes.
Common use cases include calculating average sales per region, mean response times in performance analysis, average customer age or tenure, typical transaction amounts, and benchmark metrics for comparisons. Averages are sensitive to outliers, so complement them with median calculations or percentiles for robust analysis. Understanding aggregation functions is fundamental to data analysis in Spark. Combine avg() with groupBy(), window functions, and having clauses for sophisticated analytical queries. The functions integrate seamlessly with Spark’s distributed execution, computing partial averages per partition before combining for final results.
Question 135:
What is the purpose of the from_unixtime() function?
A) To get current Unix time
B) To convert Unix timestamp to readable datetime string
C) To convert datetime to Unix timestamp
D) To validate timestamps
Answer: B
Explanation:
The from_unixtime() function converts Unix timestamp values (seconds since epoch: January 1, 1970) into human-readable datetime strings. This function is essential when working with systems that store timestamps as Unix epoch seconds, enabling conversion to formatted date strings for display or further date operations. It accepts an optional format parameter to customize the output string format using Java SimpleDateFormat patterns.
Usage example: df.withColumn(“datetime”, from_unixtime(col(“timestamp_column”))) converts Unix timestamps to default format “yyyy-MM-dd HH:mm:ss”. Custom format: from_unixtime(col(“timestamp”), “MM/dd/yyyy HH:mm”) produces different formatting. The function handles timezone based on Spark’s timezone configuration. Input must be numeric seconds since epoch; millisecond timestamps need division by 1000 first.
Unix timestamps are common in systems, logs, and APIs because they’re timezone-independent, easy to compare, and efficient to store. Converting them to readable formats is necessary for reporting, display, and date-based operations. The inverse function is unix_timestamp() which converts datetime strings or columns to Unix timestamps. Together they enable bidirectional conversion between human-readable and epoch representations.
Common use cases include processing log data with Unix timestamps, converting API responses with epoch times, preparing data for display in reports, integrating data from systems using Unix time, and enabling date arithmetic after conversion. After converting to readable format, you can use date functions like year(), month(), or date arithmetic operations. Understanding timestamp conversions is crucial for working with temporal data from diverse sources. Different systems use seconds, milliseconds, or microseconds since epoch, so verify units before conversion. Timezone handling requires careful consideration as Unix timestamps are UTC-based but displayed times depend on session timezone settings.
Question 136:
Which method is used to apply a function to each element in a column?
A) apply()
B) map()
C) transform()
D) withColumn()
Answer: D
Explanation:
The withColumn() method combined with Column expressions or UDFs applies functions to each element in a column, transforming values row by row. While RDDs have a map() method, DataFrames use withColumn() with built-in functions or user-defined functions to achieve element-wise transformations. This approach integrates with Spark’s optimization engine for better performance than RDD operations while maintaining expressive transformation capabilities.
Usage example: df.withColumn(“doubled”, col(“value”) * 2) applies multiplication to each value. For complex logic: df.withColumn(“category”, when(col(“age”) < 18, “minor”).otherwise(“adult”)) applies conditional logic per row. For custom functions: first create a UDF: custom_udf = udf(lambda x: x.upper(), StringType()), then apply: df.withColumn(“upper_name”, custom_udf(col(“name”))). Built-in functions are preferred over UDFs for performance.
DataFrames don’t have a direct apply() method like pandas, but withColumn() achieves similar results more efficiently by leveraging Catalyst optimization. The transform() method exists but serves different purposes (chaining transformation functions, not element-wise operations). Understanding DataFrame transformation patterns is crucial for efficient data processing.
Common use cases include calculating derived columns from existing values, applying business logic transformations, cleaning or normalizing data, type conversions, and implementing feature engineering for machine learning. Always prefer built-in functions over UDFs when possible, as they’re optimized by Catalyst and avoid serialization overhead. For complex transformations requiring UDFs, consider Pandas UDFs which vectorize operations for better performance. WithColumn() returns new DataFrames due to immutability, enabling method chaining for sequential transformations. Understanding transformation patterns in DataFrames versus RDDs is essential for writing efficient Spark code leveraging modern optimization capabilities.
Question 137:
What is the purpose of the percentile_approx() function?
A) To calculate exact percentiles
B) To compute approximate percentiles efficiently
C) To find percentage values
D) To validate percentages
Answer: B
Explanation:
The percentile_approx() function computes approximate percentiles of numeric columns efficiently using probabilistic algorithms, providing fast estimations without expensive exact sorting. This aggregation function is particularly valuable for large datasets where exact percentile calculation would be prohibitively costly. It uses a quantile approximation algorithm that balances accuracy against computation time and memory usage through configurable accuracy parameters.
Usage example: df.agg(percentile_approx(“salary”, 0.5)) calculates approximate median salary. For multiple percentiles: percentile_approx(“value”, array(lit(0.25), lit(0.5), lit(0.75))) returns quartiles. The optional third parameter controls accuracy: percentile_approx(“salary”, 0.5, 10000) uses higher accuracy. Larger accuracy parameters provide better precision but require more memory and computation. The function is designed for grouped aggregations and overall dataset percentiles.
Approximate percentiles enable statistical analysis on massive datasets that would be impractical with exact methods. The algorithm provides reasonable accuracy sufficient for most analytical purposes while executing orders of magnitude faster. This makes percentile analysis feasible in interactive queries, dashboards, and exploratory analysis where speed matters more than perfect precision.
Common use cases include calculating median and quartile statistics, identifying outlier thresholds, creating histogram bins, analyzing value distributions, and generating summary statistics for monitoring or reporting. The function is particularly valuable in performance analysis (response time percentiles), financial analysis (risk metrics), and any scenario requiring percentile statistics at scale. Understanding when approximate algorithms suffice versus when exact calculations are necessary helps optimize analytical workloads. For small datasets, exact percentiles might be preferable, but for big data scenarios, percentile_approx() provides practical solutions for otherwise intractable computations.
Question 138:
Which function is used to convert a column to JSON string format?
A) toJson()
B) to_json()
C) asJson()
D) jsonify()
Answer: B
Explanation:
The to_json() function converts structured columns (StructType, ArrayType, or MapType) into JSON-formatted string columns. This function is essential for serializing complex nested structures into JSON strings for storage, transmission, or integration with systems expecting JSON input. It’s the inverse of from_json(), together enabling bidirectional conversion between structured and JSON string representations.
Usage example: df.withColumn(“json_string”, to_json(struct(col(“name”), col(“age”), col(“address”)))) converts multiple columns wrapped in a struct into a JSON string. You can convert existing struct columns: to_json(col(“address_struct”)) serializes the struct to JSON. Options can customize output format, such as ignoring null values or controlling date formats. The resulting string is valid JSON that can be parsed by any JSON parser.
To_json() is particularly useful when preparing data for APIs, message queues, or storage systems that handle JSON. It enables storing complex nested structures in systems that only support string types. After conversion, JSON strings can be written to databases, sent through messaging systems, or exposed through APIs. The function properly handles escaping, quoting, and JSON syntax requirements automatically.
Common use cases include preparing data for REST API responses, serializing complex objects for Kafka messages, storing structured data in text fields, creating JSON exports, and integrating with systems expecting JSON format. The function handles nested structures of arbitrary complexity, converting arrays of structs, maps, and deeply nested objects into proper JSON. Understanding JSON conversion functions is critical for modern data engineering where JSON is ubiquitous in inter-system communication. Combine to_json() with from_json() for round-trip conversions when processing data through JSON intermediaries.
Question 139:
What is the purpose of the ntile() window function?
A) To create tiles
B) To divide ordered data into N buckets
C) To filter data
D) To sort data
Answer: B
Explanation:
The ntile() window function divides ordered data within window partitions into approximately equal-sized buckets, assigning each row a bucket number from 1 to N. This function is essential for creating quantile-based groups, implementing percentile rankings, or dividing data into equal segments for analysis or sampling. It requires window specification with ordering to determine how rows are distributed into buckets.
Usage example: df.withColumn(“quartile”, ntile(4).over(Window.orderBy(“salary”))) divides rows into 4 quartiles based on salary, assigning values 1-4. For partitioned buckets: ntile(10).over(Window.partitionBy(“region”).orderBy(“sales”)) creates deciles within each region. If row count isn’t perfectly divisible by N, ntile() distributes remainder rows evenly, with earlier buckets getting extra rows.
Ntile() enables quantile analysis, creating groups for comparative analysis, and implementing statistical segmentation. It’s particularly useful for creating training/validation/test splits in machine learning, segmenting customers into groups, analyzing performance across percentile bands, and implementing any analysis requiring equal-sized ordered groups. The function maintains all rows while adding group assignment, unlike limit() which filters rows.
Common use cases include creating percentile groups for analysis, dividing data into equal segments for processing, implementing stratified sampling, creating customer segments based on value or behavior metrics, and analyzing patterns across quantile groups. Combine ntile() with groupBy() to compute statistics per bucket. Understanding ntile() enables sophisticated segmentation and comparative analysis across data distribution. It’s more flexible than simple bucketing as it ensures equal group sizes regardless of value distribution. The function is particularly valuable in scenarios where you need to compare performance across equal-sized groups or implement portfolio analysis.
Question 140:
Which method is used to write data with specific number of files?
A) coalesce() before writing
B) repartition() before writing
C) numFiles() option
D) Both A and B
Answer: D
Explanation:
Both coalesce() and repartition() can control the number of output files when writing data, though they work differently. Coalesce() reduces partitions without shuffling, ideal for decreasing file count, while repartition() can increase or decrease partitions with full shuffle for balanced files. Using these methods before write operations directly controls output file count since Spark writes one file per partition.
Usage example: df.coalesce(1).write.parquet(path) writes a single output file. For balanced multiple files: df.repartition(10).write.parquet(path) creates 10 files with evenly distributed data. Coalesce() is more efficient when reducing partitions as it minimizes data movement, while repartition() ensures even distribution across all new partitions through full shuffle. Choose based on whether you need balanced file sizes.
Controlling output file count is important for several reasons. Too many small files create overhead in storage systems and slow down subsequent reads. Too few large files limit read parallelism and may cause memory issues. Optimal file sizes typically range from 100MB to 1GB depending on use case. Before writing, analyze data volume and partition count to determine appropriate target file count.
Common use cases include consolidating many small partitions into fewer larger files after filtering operations, creating single output files for small result sets, balancing file sizes for optimal read performance, and meeting external system requirements for specific file counts. Consider tradeoffs: fewer files mean less parallelism but better small file problem mitigation. More files enable parallel processing but create overhead. Understanding partition management before writes is crucial for optimizing storage and read performance. Monitor actual output file sizes and counts to validate partition settings.
