Publication
ZeroTune: Learned Zero-Shot Parallelism Tuning for Distributed Stream Processing
Pratyush Agnihotri; Paul Stiegele; Roman Heinrich; Boris Koldehofe; Carsten Binnig; Manisha Luthra
In: 40th IEEE International Conference on Data Engineering (ICDE 2024). IEEE International Conference on Data Engineering (ICDE-2024), Pages 1-14, IEEE, 2024.
Abstract
This paper introduces ZeroTune, a novel cost model for parallel and distributed stream processing that can be used to effectively set initial parallelism degrees of streaming queries. Unlike existing models, which rely ma- jorly on online learning statistics that are non-transferable, context-specific, and require extensive training, ZeroTune proposes data-efficient zero-shot learning techniques that en- able very accurate cost predictions without having observed any query deployment. To overcome these challenges, we propose ZeroTune, a graph neural network architecture that can learn from the structural complexity of parallel distributed stream processing systems, enabling them to adapt to unseen workloads and hardware configurations. In our experiments, we show when integrating ZeroTune in a distributed streaming system such as Apache Flink, we can accurately set the degree of parallelism, showing an average speed-up of around 5× in comparison to existing approaches.