MENU
A data scientist has a Spark DataFrame spark_df. They want to create a new Spark DataFrame that contains only the rows from spark_df where the value in column discount is less than or equal 0.
Which of the following code blocks will accomplish this task?
Suggested Answer: C
Get Premium Databricks Machine Learning Associate Questions
as Interactive Web-Based Practice Test or PDF
Real Exam Format
Detailed Explanations
Instant Access
Get Your Practice Test
To filter rows in a Spark DataFrame based on a condition, the filter method is used. In this case, the condition is that the value in the 'discount' column should be less than or equal to 0. The correct syntax uses the filter method along with the col function from pyspark.sql.functions.
Correct code:
from pyspark.sql.functions import col filtered_df = spark_df.filter(col('discount') <= 0)
Option A and D use Pandas syntax, which is not applicable in PySpark. Option B is closer but misses the use of the col function.
39 comments · newest first
IN
In
10 months ago
D has a syntax error, not sure how that got in there.
CH
Christoper
11 months ago
Definitely going with C, it’s the cleanest!
YA
Yaeko
11 months ago
Wait, is A even valid for Spark? Seems off.
TA
Tamar
11 months ago
I think B looks good too, but it's not Spark syntax.
GA
Garry
11 months ago
Option C is the correct way to filter in Spark.
DO
Dottie
12 months ago
I think option A is definitely wrong because loc is not used in Spark, but I’m not completely confident about the others.
CA
Carmen
12 months ago
I feel like option B might be a common way to filter data, but I can't recall if it works with Spark DataFrames.
AL
Aliza
12 months ago
I'm not entirely sure, but I remember something about using loc in Pandas. Is that applicable here, or is it just for Pandas DataFrames?
RE
Reena
1 year ago
I think option C looks familiar because we practiced using the filter method in Spark. It seems like the right approach for this task.
DI
Dick
1 year ago
This is a great question to test our understanding of Spark DataFrame operations. I think option C is the best choice here, as it directly uses the Spark-specific `filter()` method to apply the condition on the 'discount' column.
MI
Ming
1 year ago
I'm a little confused by the different options presented here. They all look like they could potentially work, but I'm not sure which one is the most efficient or Spark-idiomatic approach. I'll have to think this through carefully.
ER
Erick
1 year ago
Okay, I've got this! Option B is the way to go - it's the classic way to filter a DataFrame in Pandas, and Spark has a similar syntax, so I'm confident that will work here.
RA
Rashida
1 year ago
Hmm, I'm a bit unsure about this one. I know we need to filter the DataFrame, but I'm not totally familiar with the Spark syntax. I might have to double-check the documentation to make sure I'm using the right method.
TH
Thad
1 year ago
This looks like a straightforward Spark DataFrame filtering task. I think I'll go with option C - it seems the most direct way to filter the DataFrame based on the condition in the question.
TI
Tijuana
1 year ago
This seems like a straightforward memory leak issue. I'd start by looking for OutOfMemoryError messages in the logs, and then create a heap dump to analyze.
SO
Son
1 year ago
I'm pretty confident the answer is option D. The parameter to enable YARN log aggregation is "yarn.log-aggregation-enable".
DA
Darci
1 year ago
Okay, I've got this. Running services as unprivileged accounts and using the principle of least privilege are definitely the way to go. That should help prevent unauthorized access.
LO
Louvenia
1 year ago
I feel like both Archive and Infrequent Access match what I've studied, but I might need to double-check my notes.
TH
Theron
1 year ago
Yeah, I'm leaning towards A, B, and E as being suitable. I just hope I don't mix them up in the exam!
CH
Chantell
1 year ago
Hmm, I'm not sure about this one. I'll have to think it through carefully, the options seem pretty similar.
MI
Miesha
2 years ago
I agree with Sharen, option B seems like the most straightforward solution.
AN
Annalee
2 years ago
Hmm, I'm torn between B and C. Maybe I'll flip a coin to decide. Or, you know, use a random number generator. Data scientists love those, right?
MI
Mitsue
2 years ago
Tegan: Good idea. Let's test them out and compare the results.
TE
Tegan
2 years ago
Alisha: Maybe we should both try out our options and see which one works better.
AL
Alisha
2 years ago
I'm leaning towards C) spark_df.filter (col(\discount\) <= 0) actually.
TH
Theodora
2 years ago
I think B) spark_df[spark_df[\discount\] <= 0] is the correct option.
SH
Sharen
2 years ago
But with option B, we can directly filter the DataFrame based on the condition.
LA
Lashon
2 years ago
I disagree, I believe the correct answer is C.
KA
Karl
2 years ago
As a data scientist, I'd choose C. It's more readable and maintainable than the other options.
MA
Mayra
2 years ago
I agree with User1, B looks like the right choice.
LA
Lakeesha
2 years ago
I think B is the correct option.
WH
Whitley
2 years ago
I think B is the way to go. It's a simple and straightforward indexing operation on the DataFrame.
EL
Eleonora
2 years ago
Agreed, it's a simple and straightforward indexing operation on the DataFrame.
AL
Alida
2 years ago
I think B is the way to go.
DA
Dalene
2 years ago
Option C looks good to me. It's a direct Spark DataFrame filter operation on the 'discount' column.
CH
Cherry
2 years ago
I would go with Option B as well. It seems like a straightforward way to filter the DataFrame.
LO
Lorrie
2 years ago
I think Option B would work too. It filters the DataFrame based on the condition in the 'discount' column.
AM
Ammie
2 years ago
Option C looks good to me. It's a direct Spark DataFrame filter operation on the 'discount' column.
SH
Sharen
2 years ago
I think the correct answer is B.