MENU
A data architect needs to load large amounts of data from a database that is continuously updated.
* New records are added, and existing records get updated and deleted.
* Each record has a LastModified field.
* All existing records are exported into a QVD file.
* The data architect wants to load the records into Qlik Sense efficiently.
Which steps should the data architect take to meet these requirements?
When dealing with a database that is continuously updated with new records, updates, and deletions, an efficient data load strategy is necessary to minimize the load time and keep the Qlik Sense data model up-to-date.
Explanation of Steps:
Load the existing data from the QVD:
This step retrieves the already loaded and processed data from a previous session. It acts as a base to which new or updated records will be added.
Load new and updated data from the database. Concatenate with the table loaded from the QVD:
The next step is to load only the new and updated records from the database. This minimizes the amount of data being loaded and focuses on just the changes.
The new and updated records are then concatenated with the existing data from the QVD, creating a combined dataset that includes all relevant information.
Create a separate table for the deleted rows and use a WHERE NOT EXISTS to remove these records:
A separate table is created to handle deletions. The WHERE NOT EXISTS clause is used to identify and remove records from the combined dataset that have been deleted in the source database.