Free Preparation Discussions
MENU
Amazon Exam Amazon-DEA-C01 Topic 2 Question 12 Discussion
Topic #: 2
A data engineer is building an automated extract, transform, and load (ETL) ingestion pipeline by using AWS Glue. The pipeline ingests compressed files that are in an Amazon S3 bucket. The ingestion pipeline must support incremental data processing.
Which AWS Glue feature should the data engineer use to meet this requirement?
Problem Analysis:
The pipeline processes compressed files in S3 and must support incremental data processing.
AWS Glue features must facilitate tracking progress to avoid reprocessing the same data.
Key Considerations:
Incremental data processing requires tracking which files or partitions have already been processed.
The solution must be automated and efficient for large-scale ETL jobs.
Solution Analysis:
Option A: Workflows
Workflows organize and orchestrate multiple Glue jobs but do not track progress for incremental data processing.
Option B: Triggers
Triggers initiate Glue jobs based on a schedule or events but do not track which data has been processed.
Option C: Job Bookmarks
Job bookmarks track the state of the data that has been processed, enabling incremental processing.
Automatically skip files or partitions that were previously processed in Glue jobs.
Option D: Classifiers
Classifiers determine the schema of incoming data but do not handle incremental processing.
Final Recommendation:
Job bookmarks are specifically designed to enable incremental data processing in AWS Glue ETL pipelines.
Gracia
2 months agoErnie
2 months agoMalcom
1 months agoErinn
2 months agoJohnetta
2 months agoCecilia
2 months agoMartina
2 months agoMauricio
2 months agoReyes
1 months agoJohnetta
2 months agoLorrine
2 months agoJulio
2 months agoTayna
2 months agoBenedict
2 months agoPura
2 months agoSimona
2 months ago