Publication
Deco: Fast and Accurate Decentralized Aggregation of Count-Based Windows in Large-Scale IoT Applications
Wang Yue; Rafael Moczalla; Manisha Luthra; Tilmann Rabl
In: Proceedings 27th International Conference on Extending Database Technology ( EDBT 2024 ). International Conference on Extending Database Technology (EDBT-2024), 27th International Conference on Extending Database Technology, March 25-28, Italy, ISBN 978-3-89318-091-2, OpenProceeedings.org, 2024.
Abstract
In the realm of large-scale Internet-of-Things applications, aggregating data using count-based windows is a formidable challenge. Current methods, either centralized and slow or decentralized with potential inaccuracies, fail to strike a balance. This paper introduces Deco, a novel approach tailored for swift and precise aggregation in distributed stream processing systems. Accomplishing this balance is complex due to the dynamic nature of event distribution: events arrive at varying rates, unordered, and at diverse times, making accurate window computation a challenge. To overcome this, we propose a lightweight prediction method that derives local window sizes based on the previously observed event rates and performs corrections when necessary to ensure accurate and fast query results. These windows are processed in a decentralized manner on local nodes, verified for correctness, and then aggregated on a root node. Our evaluation showcases Deco’s superiority over centralized methods, outperforming others significantly. Deco reduces network traffic by up to 99% and exhibits linear scalability with node count.