MENU
Your company produces 20,000 files every hour. Each data file is formatted as a comma separated values (CSV) file that is less than 4 KB. All files must be ingested on Google Cloud Platform before they can be processed. Your company site has a 200 ms latency to Google Cloud, and your Internet connection bandwidth is limited as 50 Mbps. You currently deploy a secure FTP (SFTP) server on a virtual machine in Google Compute Engine as the data ingestion point. A local SFTP client runs on a dedicated machine to transmit the CSV files as is. The goal is to make reports with data from the previous day available to the executives by 10:00 a.m. each day. This design is barely able to keep up with the current volume, even though the bandwidth utilization is rather low.
You are told that due to seasonality, your company expects the number of files to double for the next three months. Which two actions should you take? (choose two.)
Suggested Answer: C, E
Get Premium Professional Data Engineer Questions
as Interactive Web-Based Practice Test or PDF
Real Exam Format
Detailed Explanations
Instant Access
Get Your Practice Test
29 comments · newest first
GE
Gene
10 months ago
TAR files could work, but is it worth the extra steps?
RO
Rory
10 months ago
Wait, can we really compress those files enough to make a difference?
HA
Hannah
10 months ago
I think increasing bandwidth is a must.
MA
Marylyn
10 months ago
Doubling files means we need a better solution fast!
GL
Glennis
11 months ago
gsutil for parallel uploads sounds like a solid plan!
DE
Denny
11 months ago
I’m a bit hesitant about the TAR file approach. It could save bandwidth, but what if the disassembly process in the cloud introduces delays?
FA
Fatima
11 months ago
I think using gsutil for parallel uploads could really help with the ingestion process. We practiced a similar question about optimizing data transfers last week.
GA
Gail
11 months ago
Increasing the bandwidth sounds like a straightforward solution, but I wonder if it would actually solve the latency issue we talked about.
RO
Rosalind
12 months ago
I remember we discussed data compression in class, and it seems like a good way to speed up transfers. But I'm not sure if it would be enough on its own.
FR
Freeman
12 months ago
This is an interesting challenge! I like the idea of using the Google Cloud Storage Transfer Service to move the data from on-premises to the cloud. That could be a really efficient way to handle the increased volume without having to mess with the local infrastructure.
GE
German
12 months ago
I'm a bit confused by all the options here. I think I'd need to do some more research on the different tools and techniques before deciding on the best approach. Maybe I'll start by looking into the pros and cons of each solution.
NA
Nadine
1 year ago
Okay, let's think this through step-by-step. First, I'd look at the compression option to reduce the file size. Then, I'd explore the parallel processing idea using the gsutil tool. That should help us meet the tight deadline.
JE
Jeannine
1 year ago
Hmm, this is a tricky one. I'm not sure if increasing the bandwidth alone will be enough, since the latency is also a factor. Maybe I should look into some cloud-based solutions to speed up the data transfer.
AL
Alida
1 year ago
This looks like a classic data ingestion problem. I'd start by looking at the bandwidth and latency constraints, and see if I can improve the transfer speed through compression or parallel processing.
MI
Mitsue
1 year ago
D is just weird. Taping files? What is this, the 90s? C and E all the way.
AV
Avery
1 year ago
A and B might work in the short term, but they won't be sustainable long-term. Gotta rethink the whole process.
EL
Eladia
9 months ago
Let's not forget about the TAR option too!
NI
Nieves
9 months ago
gsutil could really speed things up.
MA
Malinda
10 months ago
A and B might work for now, but we need a long-term solution.
KR
Kristofer
11 months ago
Agreed! Redesigning the ingestion process is key.
YA
Yan
1 year ago
That's true, but I think prioritizing parallel transfer and using a dedicated storage endpoint might be more efficient in the long run.
DO
Dorethea
1 year ago
But what about option A? Introducing data compression could also help with faster file transfer.
AR
Arthur
1 year ago
I agree, using gsutil tool and Google Cloud Storage Transfer Service can help with the increased volume of files.
YA
Yan
1 year ago
I think we should consider option C and E.
CA
Carissa
1 year ago
C and E seem like the best options here. Parallelizing the data transfer and using a storage service could really help handle the increased load.
ST
Stefany
1 year ago
E) Create an S3-compatible storage endpoint in your network, and use Google Cloud Storage Transfer Service to transfer on-premices data to the designated storage bucket.
TO
Tomas
1 year ago
E) Create an S3-compatible storage endpoint in your network, and use Google Cloud Storage Transfer Service to transfer on-premices data to the designated storage bucket.
PA
Paris
1 year ago
C) Redesign the data ingestion process to use gsutil tool to send the CSV files to a storage bucket in parallel.
BR
Brianne
1 year ago
C) Redesign the data ingestion process to use gsutil tool to send the CSV files to a storage bucket in parallel.