A company builds a new data pipeline to process data for business intelligence reports. Users have noticed that data is missing from the reports.
A data engineer needs to add a data quality check for columns that contain null values and for referential integrity at a stage before the data is added to storage.
Which solution will meet these requirements with the LEAST operational overhead?
AWS Glue Data Quality transforms allow you to define built-in rules like IsComplete for null validation and ReferentialIntegrity for relationship validation---all with minimal code and operational overhead.
''Use AWS Glue Data Quality rules such as IsComplete and ReferentialIntegrity within ETL jobs to automatically validate incoming data.''
A data engineer must ingest a source of structured data that is in .csv format into an Amazon S3 data lake. The .csv files contain 15 columns. Data analysts need to run Amazon Athena queries on one or two columns of the dataset. The data analysts rarely query the entire file.
Which solution will meet these requirements MOST cost-effectively?
Amazon Athena is a serverless interactive query service that allows you to analyze data in Amazon S3 using standard SQL. Athena supports various data formats, such as CSV, JSON, ORC, Avro, and Parquet. However, not all data formats are equally efficient for querying. Some data formats, such as CSV and JSON, are row-oriented, meaning that they store data as a sequence of records, each with the same fields. Row-oriented formats are suitable for loading and exporting data, but they are not optimal for analytical queries that often access only a subset of columns. Row-oriented formats also do not support compression or encoding techniques that can reduce the data size and improve the query performance.
On the other hand, some data formats, such as ORC and Parquet, are column-oriented, meaning that they store data as a collection of columns, each with a specific data type. Column-oriented formats are ideal for analytical queries that often filter, aggregate, or join data by columns. Column-oriented formats also support compression and encoding techniques that can reduce the data size and improve the query performance. For example, Parquet supports dictionary encoding, which replaces repeated values with numeric codes, and run-length encoding, which replaces consecutive identical values with a single value and a count. Parquet also supports various compression algorithms, such as Snappy, GZIP, and ZSTD, that can further reduce the data size and improve the query performance.
Therefore, creating an AWS Glue extract, transform, and load (ETL) job to read from the .csv structured data source and writing the data into the data lake in Apache Parquet format will meet the requirements most cost-effectively. AWS Glue is a fully managed service that provides a serverless data integration platform for data preparation, data cataloging, and data loading. AWS Glue ETL jobs allow you to transform and load data from various sources into various targets, using either a graphical interface (AWS Glue Studio) or a code-based interface (AWS Glue console or AWS Glue API). By using AWS Glue ETL jobs, you can easily convert the data from CSV to Parquet format, without having to write or manage any code. Parquet is a column-oriented format that allows Athena to scan only the relevant columns and skip the rest, reducing the amount of data read from S3. This solution will also reduce the cost of Athena queries, as Athena charges based on the amount of data scanned from S3.
The other options are not as cost-effective as creating an AWS Glue ETL job to write the data into the data lake in Parquet format. Using an AWS Glue PySpark job to ingest the source data into the data lake in .csv format will not improve the query performance or reduce the query cost, as .csv is a row-oriented format that does not support columnar access or compression. Creating an AWS Glue ETL job to ingest the data into the data lake in JSON format will not improve the query performance or reduce the query cost, as JSON is also a row-oriented format that does not support columnar access or compression. Using an AWS Glue PySpark job to ingest the source data into the data lake in Apache Avro format will improve the query performance, as Avro is a column-oriented format that supports compression and encoding, but it will require more operational effort, as you will need to write and maintain PySpark code to convert the data from CSV to Avro format.Reference:
Amazon Athena
Choosing the Right Data Format
AWS Glue
[AWS Certified Data Engineer - Associate DEA-C01 Complete Study Guide], Chapter 5: Data Analysis and Visualization, Section 5.1: Amazon Athena
A company has a data warehouse in Amazon Redshift. To comply with security regulations, the company needs to log and store all user activities and connection activities for the data warehouse.
Which solution will meet these requirements?
Problem Analysis:
The company must log all user activities and connection activities in Amazon Redshift for security compliance.
Key Considerations:
Redshift supports audit logging, which can be configured to write logs to an S3 bucket.
S3 provides durable, scalable, and cost-effective storage for logs.
Solution Analysis:
Option A: S3 for Logging
Standard approach for storing Redshift logs.
Easy to set up and manage with minimal cost.
Option B: Amazon EFS
EFS is unnecessary for this use case and less cost-efficient than S3.
Option C: Aurora MySQL
Using a database to store logs increases complexity and cost.
Option D: EBS Volume
EBS is not a scalable option for log storage compared to S3.
Final Recommendation:
Enable Redshift audit logging and specify an S3 bucket as the destination.
:
Amazon Redshift Audit Logging
Storing Logs in Amazon S3
A data engineer is using an AWS Glue ETL job to remove outdated customer records from a table that contains customer account information. The data engineer is using the following SQL command:
MERGE INTO accounts t USING monthly_accounts_update s
ON t.customer = s.customer
WHEN MATCHED THEN DELETE
What will happen when the data engineer runs the SQL command?
In AWS Glue's SQL implementation (Spark SQL-compatible), the MERGE INTO statement supports conditional actions.
The clause WHEN MATCHED THEN DELETE deletes matching records from the target table (accounts) where the join condition is true.
''A MERGE INTO statement can perform updates, inserts, or deletes based on the match condition between source and target tables.''
-- Ace the AWS Certified Data Engineer - Associate Certification - version 2 - apple.pdf
A company is planning to upgrade its Amazon Elastic Block Store (Amazon EBS) General Purpose SSD storage from gp2 to gp3. The company wants to prevent any interruptions in its Amazon EC2 instances that will cause data loss during the migration to the upgraded storage.
Which solution will meet these requirements with the LEAST operational overhead?
Changing the volume type of the existing gp2 volumes to gp3 is the easiest and fastest way to migrate to the new storage type without any downtime or data loss. You can use the AWS Management Console, the AWS CLI, or the Amazon EC2 API to modify the volume type, size, IOPS, and throughput of your gp2 volumes. The modification takes effect immediately, and you can monitor the progress of the modification using CloudWatch. The other options are either more complex or require additional steps, such as creating snapshots, transferring data, or attaching new volumes, which can increase the operational overhead and the risk of errors.Reference:
Migrating Amazon EBS volumes from gp2 to gp3 and save up to 20% on costs(Section: How to migrate from gp2 to gp3)
Switching from gp2 Volumes to gp3 Volumes to Lower AWS EBS Costs(Section: How to Switch from GP2 Volumes to GP3 Volumes)
Modifying the volume type, IOPS, or size of an EBS volume - Amazon Elastic Compute Cloud(Section: Modifying the volume type)
Angela White
11 days agoCarol Parker
24 days agoSharon Rogers
1 month agoBetty Rogers
2 months agoHarold Rodriguez
2 months agoStephen Sanchez
3 months agoBrian Taylor
3 months agoLinda Johnson
3 months agoJoseph Cook
2 months agoGeorge Nguyen
2 months agoDonna Lewis
2 months agoSharen
4 months agoEveline
4 months agoAbraham
4 months agoTelma
4 months agoLashawn
5 months agoLavelle
5 months agoSanjuana
5 months agoLatricia
5 months agoCorazon
6 months agoLaine
6 months agoCecily
6 months agoMarsha
6 months agoFelix
7 months agoCammy
7 months agoFloyd
7 months agoKarma
7 months agoDenise
8 months agoKenneth
8 months agoZona
8 months agoCyril
8 months agoRyann
9 months agoSue
9 months agoPamela
9 months agoTammara
9 months agoChantell
10 months agoJame
10 months agoStanton
10 months agoMargot
10 months agoStefanie
10 months agoNakisha
11 months agoCarlene
11 months agoBea
11 months agoTomoko
1 year agoCarlota
1 year agoJustine
1 year agoDallas
1 year agoHui
1 year agoLeonor
1 year agoRosenda
1 year agoDiego
1 year agoElbert
1 year agoJohnetta
1 year agoFletcher
1 year agoAndra
1 year agoKaitlyn
1 year agoCecilia
1 year agoMarquetta
1 year agoWade
1 year agoGlory
1 year agoTatum
2 years agoMelodie
2 years agoVicki
2 years agoGaston
2 years agoPedro
2 years agoTanesha
2 years agoFredric
2 years agoGlenn
2 years agoEliseo
2 years agoShawna
2 years agoEloisa
2 years agoDaron
2 years agoLashonda
2 years agoEdgar
2 years agoRessie
2 years agoIlene
2 years agoKarina
2 years ago