MENU
A data engineer writes the following code to join two DataFrames df1 and df2:
df1 = spark.read.csv("sales_data.csv") # ~10 GB
df2 = spark.read.csv("product_data.csv") # ~8 MB
result = df1.join(df2, df1.product_id == df2.product_id)
Which join strategy will Spark use?
The default broadcast join threshold in Spark is:
spark.sql.autoBroadcastJoinThreshold = 10MB
Since df2 is only 8 MB (less than 10 MB), Spark will automatically apply a broadcast join without requiring explicit hints.
From the Spark documentation:
''If one side of the join is smaller than the broadcast threshold, Spark will automatically broadcast it to all executors.''
A is incorrect because Spark does support auto broadcast even with static plans.
B is correct: Spark will automatically broadcast df2.
C and D are incorrect because Spark's default logic handles this optimization.
Final Answer: B