MediaAgility was acquired by Persistent in May 2022 and its capabilities form the foundation for Persistent’s Google Business Unit. As a Google Cloud Partner and a strategic IT Transformation services provider, the Google Business Unit is focused on delivering transformational services with cloud-native solutions that enable businesses to stay relevant and benefit from the public cloud platform capabilities.
National Stock Exchange of India Ltd. (NSE) is a leading stock exchange of India located in the city of Mumbai, Maharashtra. NSE has remained at the forefront of support to financial market research in India since early 2000 and has undertaken several research initiatives over the years, including research collaborations with top-notch global and domestic academic institutions, renowned think tanks, and global multilateral agencies. In an endeavor to extend their offerings in the research and development field and provide an enabling environment for researchers to access and analyze high-frequency orders and trades data, they worked with MediaAgility, which was acquired by Persistent Systems in May 2022, to set up a cloud-based research platform.
The need: A cloud-based research platform for academicians and researchers
NSE required a solution whereby researchers and users from within the organization as well as external researchers could access and analyze large data sets comprising of tick-by-tick orders and trades data (approximately a petabyte and rising by nearly a terabyte daily) in a cloud-based environment. The evolving scale of the data and its security was the prime concern of NSE.
The Google Cloud team introduced Persistent Systems to NSE as a System Integrator. Our existing association with NSE and Google’s confidence in our capabilities prompted NSE to finalize us as their solution implementation partner.
Our solution: A new research facility on Google Cloud
To provide a robust technological solution, we performed a phased lift and shift migration of data from the NSE’s Data Warehouse to Google Cloud Platform. This new research facility on the cloud was called the NSE Dataroom.
As a System Integrator, MediaAgility (now part of Persistent Systems) helped us build a cloud-based research facility called NSE Dataroom within tight timelines and stringent security requirements. They assisted us through the process of understanding our requirements via multiple brainstorming sessions and implemented an optimal, secure, and cost-effective solution that has truly transformed the research ecosystem provided by NSE
Dr. Tirthankar Patnaik, Chief Economist at National Stock Exchange
Some of the Google Cloud and other products that were used include:
- Virtual Machine: A virtualized instance of a computer providing heavy compute capabilities
- Cloud Storage: Used for storing and accessing data on Google Cloud Platform infrastructure
- BigQuery: Multi-cloud data warehouse used for analysis of big data
- Google SSO: Used for a secure log-in to cloud applications
- Cloud SQL: Used to store user-related data, such as user IDs, configurations, etc.
Given Google Cloud’s ability to scale up the offering without compromising on security aspects, the solution will be highly scalable and secure. Researchers can now easily log in to the virtual system to view, query, and analyze the data as per their project requirements. The solution also enables them to extract relevant codes and results from the platform provided they meet NSE’s policy guidelines.
MediaAgility (now part of Persistent Systems) along with Google shouldered the responsibility of understanding the solution architecture and providing the final solution. They have been instrumental in laying the foundation as well as fully complying with all security and operational requirements
Dr. Patnaik
NSE Dataroom on Google Cloud Platform helped NSE manage access controls
After implementing the solution, researchers and academicians could log in to a highly stable and secure platform to access historical data without NSE having to worry about data security.
Each researcher could get access to all data exploration and analytics tools in one place. With the help of Google Cloud IAM policies, authentication and authorization controls were set up to prevent unknown identities from accessing the data sets. NSE can now manage access control by defining who has how much access to the resources. Google Cloud’s high levels of security ensure that all sensitive data is stored solely in the cloud, away from local end-user devices.
We were thrilled to be associated with an esteemed institution like National Stock Exchange and developing a state-of-the-art, fully secure research platform for researchers that is scalable to handle large volumes of users and growing data at a rate of 1TB per day
Nirdesh Chahal, Google Business Unit, Persistent Systems
NSE Dataroom, as a cloud-based research platform, is immensely scalable, not just in terms of getting more and more researchers onboarded but also in terms of computational capabilities. It would enable NSE to replace the old practice of transferring high-frequency orders and trades data (often dozens of terabytes) to researchers by hosting it on the cloud. NSE benefits from the time and cost-effective solution that also assures data security.
The state-of-the-art solution provided by MediaAgility (now part of Persistent Systems) is highly secure, robust, cost-effective, and efficient. The research work, its results, and usage of data for EDA are fully controlled by NSE. Any researcher is allowed to carry out EDA only after a fully automated diligent approval process. The system spawns the computed resources on the fly in the form of a virtual data room as soon as a researcher is approved for EDA
Dr. Patnaik
As a result of the solution, NSE retained ownership of data by allowing researchers to work on this data using a virtual machine with all security protocols in place, ensuring restricted use and, therefore, no data misuse. The provision of high-performance hardware as part of the NSE Dataroom, on the other hand, helps researchers efficiently carry out analytics and research on this big data.
