Databricks Certified Associate Developer for Apache Spark 3.5 Exam - Topic 1 Question 5 Discussion

By default, the spark.read.text() method reads a text file one line per record. This means that each line in a text file becomes one row in the resulting DataFrame.

To read each file as a single record, Apache Spark provides the option wholetext, which, when set to True, causes Spark to treat the entire file contents as one single string per row.

Correct usage:

df = spark.read.option('wholetext', True).text(txt_path)

This way, each record in the DataFrame will contain the full content of one file instead of one line per record.

To also include the file path, the function input_file_name() can be used to create an additional column that stores the complete path of the file being read:

from pyspark.sql.functions import input_file_name

df = spark.read.option('wholetext', True).text(txt_path)

.withColumn('file_path', input_file_name())

This approach satisfies both requirements from the question:

Each record holds the entire contents of a file.

Each record also contains the file path from which the text was read.

Why the other options are incorrect:

B or D (lineSep) -- The lineSep option only defines the delimiter between lines. It does not combine the entire file content into a single record.

C (wholetext=False) -- This is the default behavior, which still reads one record per line rather than per file.

Reference (Databricks Apache Spark 3.5 -- Python / Study Guide):

PySpark API Reference: DataFrameReader.text --- describes the wholetext option.

PySpark Functions: input_file_name() --- adds a column with the source file path.

Databricks Certified Associate Developer for Apache Spark Exam Guide (June 2025): Section ''Using Spark DataFrame APIs'' --- covers reading files and handling DataFrames.