The Challenge: How do you enable real-time search for over 1 billion customer records and return sub-second search response results
The Solution: Dell engaged the consulting services of Weblink Technologies a leader in ElasticSearch implementations and consulting to design a solution to provide ultra-fast search capability and real-time data indexing for more than 1 billion customer records
Case Study Highlights:
- Customized Searches for data sets of 1 billion records
- Posted up to 57 million documents per hours during loading
- Fully scalable architecture
- Focus on scalability and response time
Issues and business case:
Dell handles and supports billions of customers records in their databases and searching them is an important feature with previous customer orders.
Dell looked to the Elasticsearch to greatly reduce latency and query response times. This was the main reason to migrate to Elasticsearch as a solution. I order to search for customer records, Dell has several database systems to perform the search against to find an entire set of records belonging to a custom. This process of searching multiple systems can be complex and time-consuming, in many cases newly added information does not become available for search until several slower ETL processes have run.
The effect of this is responding to a customer request can occur much slower. Due to the size of the entire customer database, search response times can vary from anywhere between 3 to 30 seconds. With the combination of slow data updates and varying search times for customer records, Dell needed an alternative designed for fast search.
Solutions and Recommendations: Dells ElasticSearch cluster
Dell engaged the independent consulting services of Weblink Technologies (a third party vendor) a leader in Elasticsearch products, to provide a solution based solely on Elasticsearch. The setup and configuration of Elasticsearch were fast and very easy. During the initial phases of the project, we first initiated a POC to determine if Elasticsearch would be a fit for the overall project’s goals and proposed SLA’s.
The first accomplishment was drastically increasing the per document indexing rate. During this phase, Weblink was able to achieve a 16 million per hour document loading rate and was able to load a total of approx. 500 million customer records in less than a 24 hour period. This benchmark was achieved but suggesting and configuring Elasticsearch in a 3 tier cluster configuration with 1 client node, 2 master nodes, and 8 data nodes. During this POC we achieved search response times of less than 1 second with 2 million documents being returned. Results improved as Dell move the POC solution to production ready hardware.
Dell also realized another great advantage of Elasticsearch with its RESTFul API interface which made the development of new applications on top of Elastic’s search easy and fast. With Elasticsearch’s developer-friendly RESTFul API’s interface, integration into Dell’s existing service layer was fast and easy. Dell’s development team was able to integrate its current service hub directly to Elasticsearch via one of the many provided client API’s provided from Elasticsearch.
Solution Assessment
Recommended a 16 node-based search cluster to facilitate a sub-second real-time search for 1 billion customers. Hard figures show reduced costs with Elasticsearch, due to lack of license costs and reduced usage of computer resources, with a capacity largely exceeding Dell’s current peaks, we do not need to plan for additional solution costs in the near future.
Weblink Technologies, a leader in Elasticsearch products, provides a solution based solely on Elastic-search. As an Elastic partner and reseller, we have worked with many of customers across the globe to provide expert consulting and implementation service for Elasticsearch, Logstash, Kibana (ELK), and Beats and all of our consultants have hands on experience working primarily on Elastic search. Whether you are using Elasticsearch for a web-facing application, your corporate intranet, or a search-powered big data analytics platform, our Elasticsearch experts bring end-to-end services that support your search and analytics infrastructure, enabling you to maximize ROI.
Contact us at [email protected] to learn more about how we can help you leverage Elastic products for high-performing, easy-to-maintain, and scalable search and analytics solutions.