August, 2023
Worth $12 billion, with over 40 million products catering to 21 million global customers, the client is a leading e-commerce firm for home furnishings in the US. To serve its customers better, it built and deployed Machine Learning (ML)-backed intelligent recommender systems that analyze purchase histories to offer personalized recommendations.
However, these recommender systems failed to meet accuracy, latency, and performance benchmarks, impacting business growth by 10% in missed opportunities. The client’s in-house ML teams struggled to deploy them at speed and scale due to their inability to source, clean, and consolidate the required data from various systems, formats, or types. These teams also worked with different languages to encode processing logic, which created a language barrier. Moreover, since data requirements differ for real-time and batch-processing needs, the client’s data scientist teams had to step outside their core mandate to validate data for each.
The client’s marketing and advertising efforts – contingent on insights generated by recommender engines – also missed hitting customer acquisition and engagement targets.
The e-commerce giant commissioned Persistent to transform and optimize data pipelines for ready use by its ML teams.
Persistent Abstracts Data Requirements with Feature Engineering on Google Cloud Platform
Persistent developed a data platform for the client’s ML teams that abstracted below-the-line feature engineering, language homogenization, and data classification. This helped the client’s teams with ready-to-access optimized, cleaned, and consolidated data that can be queried to build ML models. Persistent helped operationalize this data platform to serve the client’s ML teams with:
- Feature Engineering: Features are critical data attributes that an ML model uses as inputs to train or infer from. We sourced data from various sources such as vendor listing, warehouse audits, and applications, and can exist as images, CSV files, or tables. We cleaned it per carefully defined features identified with a deep understanding of domain needs and customer behavior. For instance, features for the client’s recommender systems would include the furniture’s price, size or weight, color, etc., which must be separated from any other irrelevant data, such as listing date or seller address, commonly called noise. With our understanding of customer expectations from online transactions, we added another feature for delivery speed to further qualify recommendations to end users. All of this was packed together in a data platform – an accessible endpoint to store and query these features with required parameters.
- Language Portability: Real-time data processing is usually done in Java or Scala, whereas data scientists usually use Python, which is well-attuned to processing data batches. Since we were working to develop a centralized data repository to serve the varied needs of the client’s ML teams, we addressed this language friction by layering the platform with a dedicated service client that abstracted the need to port data from one language to another. The data scientists could run our service client and query the data platform for real-time or batch-processing needs.
- Marketing Optimization: The client deployed a marketing and advertising tool to customize the outreach of its recommendations. Persistent embedded features highly aligned with customer expectations, that enabled ML models to generate highly personalized recommendations. This helped the client streamline advertisements as per targeted audiences, leading to improved returns.
Persistent accelerates turnaround time for ML models backing recommender engines
With our data platform, the client could fast-track the training and deployment of ML models with high levels of accuracy, performance, and speed. We delivered:
- Accelerated model deployment time by 80%, reducing it from eight days to one and half days
- Model latency more than halved, from 70 milliseconds to 30 milliseconds
- Data platform handles 1000 features programmatically
- Customer engagement improved by 10% based on accurate personalization
- ML model’s speed of delivering insights improved by 1.1X
Technology Used
- Vertex AI
- ML Ops
- Analytics
- Data processing
- Data integration
