MENU
A machine learning engineer is trying to scale a machine learning pipeline by distributing its feature engineering process.
Which of the following feature engineering tasks will be the least efficient to distribute?
Suggested Answer: E
Get Premium Databricks Machine Learning Associate Questions
as Interactive Web-Based Practice Test or PDF
Real Exam Format
Detailed Explanations
Instant Access
Get Your Practice Test
To display visual histograms and summaries of the numeric features in a Spark DataFrame, the Databricks utility function dbutils.data.summarize can be used. This function provides a comprehensive summary, including visual histograms.
Correct code:
dbutils.data.summarize(spark_df)
Other options like spark_df.describe() and spark_df.summary() provide textual statistical summaries but do not include visual histograms.
Databricks Utilities Documentation
36 comments · newest first
TY
Ty
10 months ago
I doubt that creating binary indicators is inefficient to distribute.
CO
Colene
11 months ago
Wait, why would one-hot encoding be less efficient?
CA
Casey
11 months ago
Imputing with the mean is quick, but not always accurate.
MI
Mitzie
11 months ago
I think target encoding is harder to distribute effectively.
GE
Georgene
11 months ago
One-hot encoding is pretty straightforward.
VA
Valentin
12 months ago
I practiced a similar question, and I think creating binary indicators for missing values is pretty efficient, so maybe it's not the least efficient task.
LI
Lindsey
12 months ago
Imputing missing values with the mean seems straightforward, but I feel like using the median might be less efficient to distribute due to its calculation.
SH
Shawnda
12 months ago
I think target encoding might be tricky to distribute because it relies on the target variable, which could create dependencies.
EZ
Ezekiel
1 year ago
I remember we discussed how one-hot encoding can be done in parallel, but I'm not sure if it's the least efficient.
RO
Roselle
1 year ago
Imputing with the mean or median seems like it would be the most efficient to distribute, since it's a simple calculation that can be done independently on each partition of the data.
RA
Raina
1 year ago
I'm pretty confident that creating binary indicator features for missing values would be the easiest to distribute. That's a pretty straightforward task that can be parallelized nicely.
LA
Lashanda
1 year ago
Okay, let's see. I think the target encoding might be the least efficient to distribute since it requires calculating statistics across the entire dataset.
FR
Francene
1 year ago
Hmm, I'm a bit unsure about this. I know one-hot encoding can get computationally expensive, but I'm not sure how the other options compare.
VA
Vanna
1 year ago
This seems like a tricky one. I'll need to think through the different feature engineering tasks and how they might scale.
DA
Daryl
1 year ago
Distributing feature engineering? Sounds like a job for the Avengers! Maybe we can get Thor to throw the one-hot encoders around the cluster for us. Or perhaps Hulk can just smash all the missing values into submission. Either way, this is gonna be an Endgame-level challenge.
TU
Tula
1 year ago
Ooh, true median imputation? Now we're talking! That's gotta be the most distributed-friendly option. Imagine all those servers just crunching away, finding the perfect median for each feature. It's like a mathematical orchestra!
LE
Lera
1 year ago
Yeah, it's like each server can independently calculate the median for different features efficiently.
CR
Cruz
1 year ago
I agree, it seems like a task that can easily be parallelized across multiple servers.
VE
Vernell
1 year ago
True median imputation is definitely the way to go for distributing feature engineering tasks.
DO
Dominga
1 year ago
Imputing missing values with the mean? That's a classic move, but I bet it's not the most efficient to distribute. Imagine all those little means flying around the cluster, colliding and causing mayhem. Nah, I'll go with the binary indicator features. Keeps things simple, you know?
HO
Honey
1 year ago
Creating binary indicator features does sound like a straightforward task to distribute.
JU
Julio
1 year ago
I think target encoding might be a bit tricky to distribute efficiently.
CY
Cyndy
1 year ago
I agree, one-hot encoding can be easily distributed across multiple nodes.
EV
Eveline
1 year ago
Target encoding? Really? That's going to be a nightmare to distribute. Imagine trying to coordinate all those little target values across the cluster. I'd rather just impute the missing values with the true median and call it a day.
EZ
Ezekiel
1 year ago
D: Definitely, it's a more efficient way to handle missing values in a distributed pipeline.
LE
Leslie
1 year ago
C: I think imputing missing values with the true median is a simpler option.
EV
Evangelina
1 year ago
B: Yeah, it would be a nightmare to coordinate all those target values.
OC
Octavio
1 year ago
A: I agree, target encoding seems like a headache to distribute.
MI
Michell
2 years ago
I disagree. I believe creating binary indicator features for missing values would be the least efficient task to distribute because it involves checking for missing values in each feature separately.
WA
Wai
2 years ago
Hmm, one-hot encoding seems like the obvious choice here. I mean, how hard can it be to distribute that process? It's not like we're training a neural network or anything. Just slap it on a few more servers and voila!
TR
Tresa
1 year ago
User 4: Imputing missing feature values with the true median
KR
Kris
1 year ago
User 3: Imputing missing feature values with the mean
SE
Sena
1 year ago
User 2: Target encoding categorical features
TH
Theresia
1 year ago
User 1: One-hot encoding categorical features
DO
Dortha
2 years ago
I agree with Lizbeth. Target encoding involves calculating statistics based on the target variable, which can be tricky to distribute efficiently.
LI
Lizbeth
2 years ago
I think target encoding categorical features will be the least efficient to distribute.