Free Preparation Discussions
MENU
Databricks Machine Learning Associate Exam - Topic 1 Question 37 Discussion
Topic #: 1
A data scientist has developed a machine learning pipeline with a static input data set using Spark ML, but the pipeline is taking too long to process. They increase the number of workers in the cluster to get the pipeline to run more efficiently. They notice that the number of rows in the training set after reconfiguring the cluster is different from the number of rows in the training set prior to reconfiguring the cluster.
Which of the following approaches will guarantee a reproducible training and test set for each model?
To ensure reproducible training and test sets, writing the split data sets to persistent storage is a reliable approach. This allows you to consistently load the same training and test data for each model run, regardless of cluster reconfiguration or other changes in the environment.
Correct approach:
Split the data.
Write the split data to persistent storage (e.g., HDFS, S3).
Load the data from storage for each model training session.
train_df, test_df = spark_df.randomSplit([0.8, 0.2], seed=42) train_df.write.parquet('path/to/train_df.parquet') test_df.write.parquet('path/to/test_df.parquet') # Later, load the data train_df = spark.read.parquet('path/to/train_df.parquet') test_df = spark.read.parquet('path/to/test_df.parquet')
Robt
9 hours agoHerman
6 days agoCatherin
11 days agoFrederic
16 days agoCheryl
21 days agoCatarina
26 days agoKenny
1 month agoCharlie
1 month agoShelton
1 month agoKiera
2 months agoKattie
2 months agoMargurite
2 months agoMozell
2 months agoMaddie
2 months agoAudria
3 months agoLynelle
3 months agoMichel
3 months agoRaylene
3 months agoStephaine
2 months ago