A Study on Performance Improvement Based on Meta-Heuristic Algorithms for Efficient Streaming Data Processing

초록

The need to efficiently process and analyze large amounts of streaming data has led to the emergence of various stream processing platforms. However, the challenge remains to optimize several aspects of performance, such as processing speed and usage efficiency. In this study, we introduce a novel approach to address this problem using the Whale Optimization Algorithm (WOA), a metaheuristic algorithm that simulates the hunting behavior of whales. Applied to a stream processing platform, WOA achieves performance improvements compared to existing algorithms and shows parallelism and overall performance gains by tuning the executor settings of distributed computing systems such as Apache Spark where parallelism is important. We also leverage grid search, a hyperparameter optimization technique, to fine-tune the hyperparameters of WOA to achieve additional performance gains. © 2024, Korean Institute of Communications and Information Sciences. All rights reserved.

키워드