Discuss Databricks Databricks Machine Learning Associate Exam Topic 1 Question 40

A data scientist has a Spark DataFrame spark_df. They want to create a new Spark DataFrame that contains only the rows from spark_df where the value in column price is greater than 0.

Which of the following code blocks will accomplish this task?

Aspark_df[spark_df['price'] > 0]

Bspark_df.filter(col('price') > 0)

CSELECT * FROM spark_df WHERE price > 0

Dspark_df.loc[spark_df['price'] > 0,:]

Espark_df.loc[:,spark_df['price'] > 0]

Suggested Answer: B

To filter rows in a Spark DataFrame based on a condition, you use the filter method along with a column condition. The correct syntax in PySpark to accomplish this task is spark_df.filter(col('price') > 0), which filters the DataFrame to include only those rows where the value in the 'price' column is greater than 0. The col function is used to specify column-based operations. The other options provided either do not use correct Spark DataFrame syntax or are intended for different types of data manipulation frameworks like pandas. Reference:

PySpark DataFrame API documentation (Filtering DataFrames).

Get Premium Databricks Machine Learning Associate Questions as Interactive Web-Based Practice Test or PDF Real Exam Format Detailed Explanations Instant Access Get Your Practice Test

12 comments · newest first

Lilli 3 months ago

A looks tempting, but it's not the right syntax for Spark.

Whitney 3 months ago

I agree, B makes sense. It's clear and efficient.

Willetta 3 months ago

I think option B is the best choice. It uses the filter method correctly.

Carmelina 3 months ago

D and E are for pandas, not Spark!

Gail 3 months ago

Wait, is C even valid for Spark? Seems off.

Lennie 4 months ago

Definitely going with B, it’s the standard method.

Kati 5 months ago

I think A looks right too, but not sure.

Laine 5 months ago

Option B is the correct way to filter in Spark!

Jin 6 months ago

I think option A is incorrect because it looks more like Pandas syntax, not Spark.

Blondell 6 months ago

I feel like I've seen something similar before, but I can't remember if `loc` is even applicable in Spark.

Tom 6 months ago

I think option B looks right because I recall using `filter` for conditions in Spark DataFrames.

Luther 7 months ago

I remember practicing filtering DataFrames, but I'm not sure if it's `filter` or `where` in Spark.

Databricks Machine Learning Associate Exam - Topic 1 Question 40 Discussion