The client is a US-headquartered subsidiary of global airways and offers various non-air travel products like travel insurance, cruises, car rentals, and more. Their goal is to provide a one-stop-shop for travel products that complement their customers’ journeys. For that, they heavily rely on the flight reservation details to customize their offerings.
The client’s parent airline company stores the reservation details in a SaaS platform. Previously, the parent company sent messages in overnight batches to the client. As the messages weren’t in real-time, it meant missed opportunities to present the right offerings to the flyers and also to respond to any reservation changes like seat upgrades, flight changes, or any other request.
To deal with this, the client’s parent company architected a solution for real-time messages via an on-prem, IBM MQ-based messaging server. Since the client’s IT environment was different from its parent company, the client needed support to integrate their data analytics pipeline with the data on the SaaS platform via the on-prem IBM MQ messaging server. They partnered with Persistent, a premier digital engineering company, to assist them in consuming the events from IBM MQ and storing the data in Google Cloud Storage (GCS).
As there was no pick and use integration available on Google Cloud that could tap into IBM MQ, Persistent evaluated two solutions to build an integration. Based on the client’s needs and requirements, the team finalized Apache NiFi from the two available options. NiFi was deployed and maintained on Google Kubernetes Engine (GKE); GKE provides automatic management, monitoring, scale, updates, and more. This solution is first of its kind as there were no published use cases, bugs, or error handling for deploying and maintaining NiFi on GKE.
The engagement had a three-phased approach:
- Discovery and Design: Persistent analyzed IBM MQ’s sample payloads and designed scalable message consumption workflows
- Prototype and Build: After evaluating alternatives, the team decided to go with NiFi. They built a scalable solution on Google Cloud and carried out the integration of IBM MQ with NiFi
- Operationalize: Team carried out performance and functional testing, generated runbook, reviewed the solution, and planned and executed the handover
The data pipeline is near real-time and consumes incoming messages pushed by the client’s parent company asynchronously. Once received, the data is transformed and stored into GCS. The Persistent team also wrote a module to enable a test connection between the on-prem messaging server and Google Cloud. Additionally, Persistent also carried out CI/CD deployment and training sessions to acquaint the client teams with the solution.
More than 14 million messages per day
The Persistent team simulated a peak load of five million messages per day. Now, the solution is able to handle a peak load of more than 14 million messages per day.
