GDPR: Converting SAS jobs to data movement operations in Azure
Challenge
The General Data Protection Regulation (GDPR) is a European data privacy law with stringent requirements designed to protect the privacy of European citizens living in Europe and abroad.
In order to comply with GDPR, a major insurance provider needed to update the way they moved and processed data. In the past, this insurance provider wrote code in SAS to move data from place to place. SAS code executions were inefficient and resulted in long wait periods. Worse still, the SAS platform made it impossible to trace each step SAS code performs, making it challenging to document all changes to the data for GDPR and data tracing requirements. To make matters worse, not only was moving data and documenting changes for GDPR compliance a time-consuming and inefficient process, but licensing costs for SAS proved exorbitant for the provider.
Solution
Neal Analytics and Microsoft worked with the insurance provider to understand their exact data movement and transformation needs. As a result, Neal Analytics architected an environment leveraging Azure Data Factory & Azure Data Lake in a way that the transition state of the data was traceable through a series of incremental loads and files saved in the data lake.
Using the Azure environment allowed for automated data movement tasks that were previously handled by SAS to instead be handled by Azure Data Factory & Azure Functions, helping reduce the need for individuals to manually write code to move data from place to place. Moving to Azure also meant a reduced need for dedicated server resources. Additionally, queries were able to be run in a manner of minutes rather than hours or days thanks to the increased compute power granted by Azure and its ability to run several queries in parallel.
Result
The traceability granted by the data movement solution built by Neal Analytics on Microsoft Azure enabled the insurance provider to successfully meet and maintain GDPR compliance. They also saw a sharp boost in efficiency and enabled individuals who wrote code in SAS to refocus their attention on tasks that drove additional value for the insurance provider.
The insurance provider also realized a significant drop in SAS-related licensing and dedicated server expenses, instead leveraging scalable, quick, and cost-efficient Azure resources to hand data movement.
