Optimizing Message Transfers in Distributed Messaging Systems through Topic and Partition Management

초록

As stream data processing technology becomes more important, messaging systems such as Apache Kafka, RabbitMQ, and ActiveMQ are being used to transfer large amounts of data fast and without loss. Apache Kafka is a representative distributed messaging system, which can deliver data generated in real time. In Apache Kafka, a broker is composed of multiple topics with different numbers of partitions. As the number of partitions increases, its processing speed also increases, but problems with CPU and memory usages also occur. In this article, we show why the number of partitions should be configured to reduce resource usages without impact on target performance. Based on our extensive experimental results, we propose a mechanism that can change the number of partitions according to the amount of transferred message under different execution environments. © 2024, Korean Institute of Communications and Information Sciences. All rights reserved.

키워드