Free Preparation Discussions
MENU
Databricks Certified Associate Developer for Apache Spark 3.5 Exam - Topic 3 Question 14 Discussion
Topic #: 3
35 of 55.
A data engineer is building a Structured Streaming pipeline and wants it to recover from failures or intentional shutdowns by continuing where it left off.
How can this be achieved?
In Structured Streaming, checkpoints store state information (offsets, progress, and metadata) needed to resume a stream after a failure or restart.
Correct usage:
Set the checkpointLocation option when writing the streaming output:
streaming_df.writeStream
.format('delta')
.option('checkpointLocation', '/path/to/checkpoint/dir')
.start('/path/to/output')
Spark uses this checkpoint directory to recover progress automatically and maintain exactly-once semantics.
Why the other options are incorrect:
A/D: recoveryLocation is not a valid Spark configuration option.
B: Checkpointing must be configured in writeStream, not during readStream.
PySpark Structured Streaming Guide --- Checkpointing and recovery.
Databricks Exam Guide (June 2025): Section ''Structured Streaming'' --- explains checkpointing and fault-tolerant streaming recovery.
Denise
2 days agoFanny
7 days agoFiliberto
12 days agoOretha
18 days agoWillodean
23 days agoEva
28 days agoJoni
1 month agoRozella
3 months agoTomas
3 months agoMarg
3 months ago