MENU
A data scientist wants each record in the DataFrame to contain:
The first attempt at the code does read the text files but each record contains a single line. This code is shown below:
The entire contents of a file
The full file path
The issue: reading line-by-line rather than full text per file.
Code:
corpus = spark.read.text("/datasets/raw_txt/*") \
.select('*', '_metadata.file_path')
Which change will ensure one record per file?
Options:
Suggested Answer: A
Get Premium Databricks Certified Associate Developer for Apache Spark 3.5 Questions
as Interactive Web-Based Practice Test or PDF
Real Exam Format
Detailed Explanations
Instant Access
Get Your Practice Test
To read each file as a single record, use:
spark.read.text(path, wholetext=True)
This ensures that Spark reads the entire file contents into one row.
12 comments · newest first
AN
Andra
3 months ago
Option B is tempting, but it won't solve the issue. We need whole text.
FL
Floyd
3 months ago
Agreed! We need one record per file, not line by line.
MY
Myrtie
3 months ago
I think option A is the best choice. Wholetext=True will read the entire file.
TA
Tashia
3 months ago
Not sure if A will work, seems too simple.
LO
Long
3 months ago
Wait, can you really read whole files like that?
LI
Lina
4 months ago
Definitely A, it makes the most sense.
LA
Lawrence
5 months ago
I think B is better for line separation.
DO
Dorethea
5 months ago
Option A is the way to go!
MI
Miesha
6 months ago
I’m confused about the options. I feel like `wholetext=True` makes sense, but I’m not 100% sure if it’s the only way to achieve that.
HA
Halina
6 months ago
This reminds me of a practice question where we had to adjust file reading options. I think `wholetext=False` would just keep it as is, right?
TE
Tequila
6 months ago
I'm not entirely sure, but I feel like the `lineSep` options are more about how to split lines rather than reading whole files.
RO
Roselle
7 months ago
I think I remember that the `wholetext=True` option is what we need to read the entire file as one record.