MENU
A data engineer needs to apply custom logic to identify employees with more than 5 years of experience in array column employees in table stores. The custom logic should create a new column exp_employees that is an array of all of the employees with more than 5 years of experience for each row. In order to apply this custom logic at scale, the data engineer wants to use the FILTER higher-order function.
Which of the following code blocks successfully completes this task?
Option A is the correct answer because it uses the FILTER higher-order function correctly to filter out employees with more than 5 years of experience from the array column ''employees''. It applies a lambda functioni -> i.years_exp > 5that checks if the years of experience of each employee in the array is greater than 5. If this condition is met, the employee is included in the new array column ''exp_employees''.
: The use of higher-order functions like FILTER can be referenced from Databricks documentation onHigher-Order Functions.