Comparison of Meta-Heuristic Algorithms for Task Scheduling in Distributed Stream Processing

With the emergence of IoT and cloud computing, the demand for big data processing continues to rise. To expedite such big data processing, distributed stream processing systems (DSPS) are commonly used. However, because the rate of incoming messages to DSPS can vary depending on a stream application and execution environments such as like network conditions, it can be challenging to provide the necessary quality of services (QoS). Modern DSPS typically use heuristic or meta-heuristic algorithms to find near-optimal solutions to meet QoS requirements; however, it is still difficult to accomplish multiple QoS goals at once. In this paper, multiple meta-heuristic algorithms are evaluated to determine if they can simultaneously achieve multiple objectives, including response time and system failure. We implemented schedulers using various meta-heuristic algorithms operating within DSPS simulation environments. Then, we executed three stream applications utilizing various scheduling algorithms and demonstrated that meta-heuristic algorithms outperform a conventional algorithm.