Skip to content

HPDA-as-a-Service as Cloud service for Financial services

Partners involved

Easy Credit is one of the most dynamically developing non-bank financial institutions in Bulgaria, specializing in granting short-term loans with over 19 years of experience. Easy Credit is part of Management Financial Group AD (MFG), a holding company that unites leading companies specializing in the provision of non-bank financial services in Central and Eastern Europe. MFG manages a rich portfolio of successful business models in the field of home lending, personal loans, micro and small business financing, credit cards, digital business and alternative financial products and services. Every day, over 8,000 employees and associates in over 400 offices in Bulgaria, Romania, Poland, North Macedonia and Ukraine take care of serving over 330,000 customers.

Technical challenge

Designing a system that dynamically provisions a Hadoop cluster while seamlessly integrating it with a business intelligence and data analytics tool introduces a lot of technical and scientific challenges. These challenges span across several areas: system architecture design, robust data management, scalable performance capabilities and security and compliance requirements.

The system demands a sophisticated cloud management framework capable of customizing the number of name nodes and data nodes according to demand. This flexibility is pivotal for resource allocation depending on the workload requirements of the specific company. The architecture must also facilitate the streamlined deployment of data analytics tools, ensuring that the solution is not only effective but also user-friendly and easily deployable.

Solution

The solution consists of developing an environment for creating a Hadoop cluster on-demand, with a custom number of name nodes and data nodes, as well as a virtual machine with a Business Intelligence and Data Analytics tool, in this specific case – Metabase [1]. The solution is easily customizable both on the Hadoop cluster side, and on the data analytics tools side. This is achieved through the OpenStack [2] ecosystem, using services like Sahara (for Hadoop provisioning and management), Glance, Nova, Neutron (for Metabase provisioning and network configuration).

The Hadoop cluster also includes Hive, which is used as a communication intermediary for the data analytics tool.

By using Hadoop and Hive as a service, alongside the integration with an analytics tool, this solution allows EasyCredit to work on big data projects without the need for creating their own infrastructure and managing the integration with additional tools and external services.

A Hadoop image is a pre-configured virtual machine image that includes a Hadoop distribution along with all necessary dependencies. The image serves as a template for creating virtual machines that will act as nodes in our Hadoop cluster. The image must be compatible with the version of Hadoop being used. It must also include a compatible operating system (specific version of Linux) and a pre-installed Hadoop distribution.

Similarly, we have created an image for Metabase, with Ubuntu Linux OS, that has been deployed as well. The process of running the data analytics tool involves:

  • Uploading our custom Linux image to Glance
  • Using Nova to launch a VM instance from this image
  • Configuring networking with Neutron
  • Attaching any necessary storage with Cinder
  • Managing access and security with Keystone;

To showcase the capabilities of this approach, large financial datasets have been loaded into the Hadoop instance – stock market data for NASDAQ since 1970 (daily OHLC and volume); foreign exchange rates (different currencies to USD) for many of the world currencies; cryptocurrency rates for some of the popular coins (to USD). Being stored in Hadoop, this data can be analyzed and processed with the tools of the ecosystem in a variety of ways, not necessarily limited to what is available at this point.

With Metabase, we have created a dashboard with charts and indicators like: stock market analysis of a given company, its value by year and month, the highest closing price of the selected period and the average annual return on investment, analysis of the foreign exchange currency market for a selected currency pair and a correlation chart between the stock market and the foreign exchange market; cryptocurrency market analysis and performance comparison with the selected stock.

EasyCredit can easily load custom data into the cluster form different public and private sources and use the powerful tools in the Hadoop ecosystem, as well as Metabase as a data analytics tool.

Impact

  • The system’s ability to dynamically scale and manage distributed computing resources can provide practical insights into solving similar problems in other scientific and business domains
  • By simplifying the integration and management of big data tools, the system can lower the barrier to entry for small and medium-sized businesses
  • The system’s ability to handle diverse datasets and provide insights through user-friendly tools like Metabase encourages the adoption of the big data systems for a variety of businesses

Benefits

  • Flexibility to scale the number of nodes in the Hadoop cluster on-demand, accommodating varying workloads and data volumes without permanently high operation costs
  • Ability to tailor both the data processing (Hadoop cluster) and the analytics (Metabase) components to specific business needs, enhancing user experience and relevance
  • Can take advantage of the Hadoop ecosystem of tools and services for data ingestion from a variety of sources, processing, etc.
  • On-demand provisioning can reduce costs by only utilizing resources when necessary, rather than maintaining a large infrastructure at all times
  • With Hive bridging between Hadoop and Metabase, businesses can perform real-time data analysis, enabling timely decision-making based on the latest data insights
  • Automates complex data management tasks, such as data partitioning, indexing, and query optimization, thereby simplifying operations and maintenance
  • Integrated data analytics tools like Metabase provide powerful visual analytics, making it easier for decision-makers to understand trends, patterns and anomalies in data
  • Reduces the technical overhead for teams, allowing them to focus on extracting value from data rather than managing infrastructure.

Success story # Highlights

  • Analyzing big data in the financial domain has become accessible by providing the necessary infrastructure as a service;
  • The service is scalable and customizable, depending on the specific needs;
  • The solution can be easily deployed as separate instances in a single physical big data infrastructure.

Figure 1: Scheme of the HPDA Service HPDA-as-a-Service

Figure 2: Screen 1 of the tool: Metabase dashboard, showing stock market analysis of a given company, its value by year and month, the highest closing price of the selected period and the average annual return on investment. Below that we see an analysis of the foreign exchange currency market for a selected currency pair and a correlation chart between the stock market and the foreign exchange market.

Figure 3: Screen 2 of the tool: Metabase dashboard, showing cryptocurrency market analysis and performance comparison with the selected stock.

Contact:

[1] https://www.metabase.com/

[2] https://www.openstack.org/