A data engineering team needs to integrate two data sources into Databricks:
Clickstream events: 5,000 events per second from an Apache Kafka topic
Customer master data: Only changed records every four hours from a Snowflake database
The solution must process clickstream data with latency under 30 seconds and prevent reprocessing customer master data that has not changed.
Which ingestion approach meets these requirements?
Suggested Answer: A
Get Premium Databricks Certified Data Engineer Associate Questions
as Interactive Web-Based Practice Test or PDF
Real Exam Format
Detailed Explanations
Instant Access
Get Your Practice Test
18 comments · newest first
JO
Joseph
3 days ago
Right! A managed connector with incremental processing fits.
LU
Lucia
8 days ago
But what about Snowflake? Incremental is key here.
CA
Carry
14 days ago
Agreed! Structured Streaming is perfect for Kafka.
GE
German
19 days ago
I think option A is the best. It handles real-time data well.
JU
Justa
24 days ago
C) could work, but not as optimal as A) for real-time needs.
CO
Cordie
29 days ago
Wait, can we really process that much data in under 30 seconds?
AN
Annmarie
1 month ago
B) seems inefficient, fetching all data every time?
AL
Alecia
1 month ago
Totally agree, A) handles both requirements well!
RE
Regenia
1 month ago
A) is the best choice for low latency with Kafka.
FL
Flo
2 months ago
C) could work, but not sure about the hourly query part.
GL
Glen
2 months ago
B) seems inefficient, fetching all data every time? No thanks.
AL
Aleisha
2 months ago
Wait, can we really avoid reprocessing with A)? Sounds risky.
HA
Han
2 months ago
Totally agree, A) makes the most sense here!
AL
Alonso
2 months ago
A) is the best option for low latency with Kafka.
TA
Tarra
3 months ago
I think option D is not ideal because a full refresh for Snowflake could lead to unnecessary data processing, which we wanted to avoid.
BR
Brittni
3 months ago
Option C sounds familiar, but I’m concerned about the latency requirement. I feel like querying Snowflake hourly might not meet the 30-second limit.
CH
Christiane
3 months ago
I'm not entirely sure about the Snowflake connector; I think we practiced a similar question where we had to avoid fetching all data unnecessarily.
CA
Casie
3 months ago
I remember we discussed using Structured Streaming for real-time data, so option A seems like a good fit for the Kafka part.