Discuss Databricks Databricks Machine Learning Associate Exam Topic 3 Question 26

A data scientist wants to use Spark ML to one-hot encode the categorical features in their PySpark DataFrame features_df. A list of the names of the string columns is assigned to the input_columns variable.

They have developed this code block to accomplish this task:

The code block is returning an error.

Which of the following adjustments does the data scientist need to make to accomplish this task?

AThey need to specify the method parameter to the OneHotEncoder.

BThey need to remove the line with the fit operation.

CThey need to use Stringlndexer prior to one-hot encodinq the features.

DThey need to use VectorAssembler prior to one-hot encoding the features.

Suggested Answer: C

For large datasets, Spark ML uses iterative optimization methods to distribute the training of a linear regression model. Specifically, Spark MLlib employs techniques like Stochastic Gradient Descent (SGD) and Limited-memory Broyden--Fletcher--Goldfarb--Shanno (L-BFGS) optimization to iteratively update the model parameters. These methods are well-suited for distributed computing environments because they can handle large-scale data efficiently by processing mini-batches of data and updating the model incrementally.

Databricks documentation on linear regression: Linear Regression in Spark ML

Get Premium Databricks Machine Learning Associate Questions as Interactive Web-Based Practice Test or PDF Real Exam Format Detailed Explanations Instant Access Get Your Practice Test

45 comments · newest first

Ivette 10 months ago

I thought VectorAssembler was needed for this too!

Francine 10 months ago

Removing the fit line? That doesn't seem right.

Arminda 11 months ago

Wait, can you skip the fit operation? Sounds off.

Luz 11 months ago

Totally agree with that! StringIndexer is essential.

Hailey 11 months ago

They need to use StringIndexer prior to one-hot encoding.

Rebbecca 11 months ago

I feel like VectorAssembler is more for combining features rather than one-hot encoding, so that option seems off to me.

Bonita 12 months ago

I practiced a similar question, and I believe using StringIndexer is crucial for categorical features.

Trina 12 months ago

I think the OneHotEncoder might need a method parameter, but I can't recall the exact syntax.

Tonja 12 months ago

I remember something about needing to use StringIndexer before one-hot encoding, but I'm not entirely sure if that's the only step needed.

Simona 1 year ago

I'm feeling pretty confident about this one. I think the solution is to specify the method parameter in the OneHotEncoder to handle the string columns properly.

Cassi 1 year ago

Based on the options, it seems like we might need to use StringIndexer first to convert the string columns to numeric indices before one-hot encoding. That could be the issue.

Melissia 1 year ago

I'm a bit confused by the OneHotEncoder setup. I wonder if we need to do some additional preprocessing on the data before applying it.

Annmarie 1 year ago

Okay, let me think this through step-by-step. I think the key is to understand what the error is telling us and how we can fix it.

Derick 1 year ago

Hmm, this looks like a tricky one. I'll need to carefully review the code and error message to figure out what's going on.

Jill 1 year ago

Interesting, I wonder if we need to use VectorAssembler to combine the features into a single vector before one-hot encoding. That might be the key step we're missing.

Mammie 1 year ago

Ah, I see. We might need to use StringIndexer first to convert the string columns to numeric indices before one-hot encoding them. That could be the issue.

Eladia 1 year ago

Hmm, the fit operation seems unnecessary here since we're just trying to transform the data, not train a model. I'd try removing that line and see if that fixes the problem.

Mel 1 year ago

I'm not entirely sure what the issue is here, but I think I might need to specify the method parameter to the OneHotEncoder to get this working.

Thurman 1 year ago

Maybe the error is because they forgot to add the 'sparkly' parameter to the OneHotEncoder. You know, to make it extra fabulous.

Shannan 1 year ago

I heard the data scientist tried to one-hot encode their socks. Turns out they were just a bunch of ones and zeros!

Kenny 1 year ago

↳ replying to Shannan

C: They need to use StringIndexer prior to one-hot encoding the features.

Martha 1 year ago

↳ replying to Shannan

B: They need to use VectorAssembler prior to one-hot encoding the features.

Daniela 1 year ago

↳ replying to Shannan

C: They need to use VectorAssembler prior to one-hot encoding the features.

Delisa 1 year ago

↳ replying to Shannan

B: They need to use StringIndexer prior to one-hot encoding the features.

Hannah 1 year ago

↳ replying to Shannan

A: They need to specify the method parameter to the OneHotEncoder.

Wilburn 1 year ago

↳ replying to Shannan

A: They need to specify the method parameter to the OneHotEncoder.

Alfred 1 year ago

VectorAssembler? Sounds like a superhero name. Maybe that's the solution, but I'm not sure.

Pura 1 year ago

↳ replying to Alfred

User3: Maybe they need to use StringIndexer before one-hot encoding the features.

Lonny 1 year ago

↳ replying to Alfred

User2: No, they should specify the method parameter to the OneHotEncoder.

Dick 1 year ago

↳ replying to Alfred

User1: I think the data scientist needs to use VectorAssembler before one-hot encoding.

Rosendo 1 year ago

Ah, I see the issue. The method parameter is missing from the OneHotEncoder. We need to specify that.

Essie 1 year ago

↳ replying to Rosendo

User1: Let's add that parameter and see if it works.

Joaquin 1 year ago

↳ replying to Rosendo

User2: Yes, that's correct. That should fix the error.

Eden 1 year ago

↳ replying to Rosendo

User1: I think we need to specify the method parameter to the OneHotEncoder.

Jenelle 1 year ago

Wait, I think we need to use StringIndexer first to convert the string columns to numerical values. Then we can use OneHotEncoder.

Latrice 1 year ago

↳ replying to Jenelle

D: And OneHotEncoder will then encode those numerical values as binary vectors.

Osvaldo 1 year ago

↳ replying to Jenelle

C: That makes sense, StringIndexer will convert the strings to numerical values.

Whitley 1 year ago

↳ replying to Jenelle

B: Then we can use OneHotEncoder to encode the categorical features.

Reita 1 year ago

↳ replying to Jenelle

A: I think you're right, we should use StringIndexer first.

Ligia 1 year ago

I believe they should also use StringIndexer before one-hot encoding the features to properly encode the categorical values.

Quentin 1 year ago

Hmm, the error is probably due to the fit operation. Let's try removing that line and see if it works.

Lawrence 1 year ago

↳ replying to Quentin

User2: Yeah, let's try that and see if it fixes the error.

Adaline 1 year ago

↳ replying to Quentin

User1: I think we should remove the line with the fit operation.

Essie 2 years ago

I agree with Daisy. Without specifying the method parameter, the code won't work properly.

Daisy 2 years ago

I think the data scientist needs to specify the method parameter to the OneHotEncoder.

Databricks Machine Learning Associate Exam - Topic 3 Question 26 Discussion