A leading US-based life sciences and instruments company achieves a single source of truth by implementing a secure and compliant data lake on AWS.
Challenges
The proliferation of data silos resulting from dozens of ERP systems across the organization limited data accessibility ability to have a single source of truth. Additionally, the lack of analytical data across numerous data silos made it difficult for the IT team to manage them. They faced challenges with schema change tracking, data cataloging, and data access control. Additionally, the Change Data Capture (CDC) was slow and infrequent making it difficult to track data from various sources.
Solutions
Integrate data from its 20+ ERP systems to a single data lake on the cloud.
- Built a data lake on AWS to integrate data sources from different silos
- Built a data ingestion pipeline to collect data from disparate systems
- Designed a framework to manage high volumes of data at high frequency using AWS Lamda and Spark
- Implement security best practices and data encryption
- Establish IAM roles and policies to control data access
Outcomes
A single source of truth by implementing a secure and compliant data lake on AWS.
- Consolidated data from over 20+ ERP systems to a single data lake
- Provided flexible of access for data analysts and business users
- Enabled the rapid movement of data from all sources in matter of minutes
- Delivered a platform that provides actionable insights from a cost-effective portfolio of data services delivered with the agility on the cloud
