Challenges
- A multinational manufacturer and supplier of networking hardware, software, telecommunications equipment and other high-technology services and products.
- Marketing Data Platform sources data from variety of sources like marketing data base, web data, partner data hosted on HDFS, Cassandra. Entire platform was hosted on premise infrastructure.
- With growing data size and different data sources getting added, it was becoming challenging to manage this platform.
- Client wanted to make platform faster for data processing and optimize cost.
- Schema changes while moving from Hadoop / HBase to Snowflake.
Solutions
- The Persistent team conducted series of technical sessions with client architects to brainstorm different options / approaches to migrate platform over GCP & Snowflake. PoCs were conducted to validate migration approach for Hadoop workloads to GCP or Snowflake.
- Developed Hadoop to Snowflake connectors in Spark to help migrate existing data to snowflake.
- The Persistent team also developed connectors to read / write data to Snowflake from GCP Dataproc.
Outcomes
50% reduction in time
to reach prospect customers
20% cost reduction
due to better resource utilization on GCP
2M records processed
per day through different data pipelines
