Optimizing Apache Spark for Healthcare Big Data Management - Lian, Zihao; Lin, Xitong; Yin, Liuyuan (Transactions on Computer Science and Intelligent Systems Research, 2024)
The advent of big data in the healthcare sector necessitates real-time data analysis and processing capabilities to enhance medical decision-making. This study explores the optimization of Apache Spark, a powerful big data processing framework, for healthcare big data management. The research aims to assess Apache Spark’s performance in handling large volumes of healthcare data and its potential for integration with emerging technologies. Utilizing the Medical Information Mart for Intensive Care (MIMIC-III) dataset, the study conducts a comparative analysis and benchmarks Apache Spark against other analytics tools, focusing on its efficiency and effectiveness in the healthcare domain. The methodology includes an in-depth examination of Spark’s architecture, Spark Streaming for real-time data processing, and Machine Learning Library (MLlib) for machine learning tasks. Experimental results demonstrate that Apache Spark significantly improves the quality and efficiency of healthcare services through its high-performance and real-time computational capabilities. The study concludes with insights into future development, emphasizing the need for enhanced security and compatibility with evolving healthcare technologies. This research advances healthcare analytics, providing a roadmap for optimizing Spark’s performance while ensuring data privacy and security.