Multilevel Data Processing Using Parallel Algorithms for Analyzing Big Data in High-Performance Computing

Awais Ahmad, Anand Paul, Sadia Din, M. Mazhar Rathore, Gyu Sang Choi, Gwanggil Jeon

Research output: Contribution to journalArticlepeer-review

36 Scopus citations

Abstract

The growing gap between users and the Big Data analytics requires innovative tools that address the challenges faced by big data volume, variety, and velocity. Therefore, it becomes computationally inefficient to analyze such massive volume of data. Moreover, advancements in the field of Big Data application and data science poses additional challenges, where High-Performance Computing solution has become a key issue and has attracted attention in recent years. However, these systems are either memoryless or computational inefficient. Therefore, keeping in view the aforementioned needs, there is a requirement for a system that can efficiently analyze a stream of Big Data within their requirements. Hence, this paper presents a system architecture that enhances the working of traditional MapReduce by incorporating parallel processing algorithm. Moreover, complete four-tier architecture is also proposed that efficiently aggregate the data, eliminate unnecessary data, and analyze the data by the proposed parallel processing algorithm. The proposed system architecture both read and writes operations that enhance the efficiency of the Input/Output operation. To check the efficiency of the proposed algorithms exploited in the proposed system architecture, we have implemented our proposed system using Hadoop and MapReduce. MapReduce is supported by a parallel algorithm that efficiently processes a huge volume of data sets. The system is implemented using MapReduce tool at the top of the Hadoop parallel nodes to generate and process graphs with near real-time. Moreover, the system is evaluated in terms of efficiency by considering the system throughput and processing time. The results show that the proposed system is more scalable and efficient.

Original languageEnglish
Pages (from-to)508-527
Number of pages20
JournalInternational Journal of Parallel Programming
Volume46
Issue number3
DOIs
StatePublished - 1 Jun 2018

Keywords

  • Big Data
  • Four-tier system architecture
  • HPC
  • Parallel Processing algorithm

Fingerprint

Dive into the research topics of 'Multilevel Data Processing Using Parallel Algorithms for Analyzing Big Data in High-Performance Computing'. Together they form a unique fingerprint.

Cite this