Publication
Network-centric Complex Event Processing: Adaptive, Efficient, and Flexible Placement and Execution of Internet of Things Applications
Manisha Luthra
PhD-Thesis, Technische Universität Darmstadt, TU prints, 9/2021.
Abstract
Complex Event Processing (CEP) is a widely used paradigm to detect and react to events of interest for various applications. Numerous companies, including Twitter and Google, build on CEP in a broad spectrum of applications to perform real-time data analytics. Many of these applications require to efficiently adapt to the dynamic environmental conditions and changes in the quality requirements. An essential building block to perform adaptations in CEP is through an operator, which encapsulates the event detection logic in the form of a query often coupled with an execution state. Despite significant contributions to concepts for operator specification, placement, and execution, there are multiple research gaps concerning adaptivity, efficiency, and interoperability in CEP. Thereby, this thesis identifies and contributes appropriate methods and their analysis to overcome these fundamental research gaps in CEP: (i) The lack of adaptivity between CEP mechanisms hinders meeting the changing quality requirements of applications. (ii) Absence of suitable network-centric abstractions that hinder efficient event processing. (iii) Absence of suitable programming abstractions that hinder reuse of CEP mechanisms across multiple programming models.
To close the first gap, we contribute a novel programming model, named TCEP, that enables transitions between so-called operator placement mechanisms. The programming model provides methods for the research questions "when" and "how" to perform a transition while ensuring crucial properties of transition such as seamlessness. In particular, we propose transition strategies that minimize the costs for operator migrations and ensure seamlessness in performing adaptations. A learning-based selection algorithm guarantees a well-suited operator placement mechanism for given quality requirements. By integrating and evaluating six operator placement mechanisms, we showed that the programming model allows the use of distinct mechanisms for adaptations, and it provides a better understanding of their cost and performance characteristics. Our extensive evaluation study using a real-world workload and implementation shows that TCEP can adapt to the dynamic quality requirements of applications in a quick, seamless, and low-cost manner.
To close the second gap, we propose a novel unified communication model named INETCEP. The proposed concepts of INETCEP contribute to the research question of "how" to enable efficient continuous event stream processing and network-centric CEP query execution. We build INETCEP using the concepts of Information-centric Networking, which has been proven to facilitate in-network programmability. As part of the unified communication model, we propose an expressive meta query language and query execution algorithms for CEP that efficiently place operators over Information-centric Networks. Our detailed evaluation study of INETCEP shows that event forwarding can be achieved in a very short time of a few microseconds. Similarly, using our network-centric abstractions, CEP queries can be resolved at very high incoming event rates in a few milliseconds while incurring no event loss.
Finally, we propose a novel unified CEP middleware, named CEPLESS, based on the serverless computing principles to close the third gap. The middleware provides concepts for the research question of "how" to specify CEP queries independent of their programming and execution environment. Specifically, the middleware contributes a programming abstraction that hides away the complexity of heterogeneous CEP programming models from the application developers. Moreover, we propose mechanisms for an efficient exchange of events using so-called in-memory queues and allow event processing across different execution models. By extending the CEPLESS middleware programming abstraction with five different programming languages, we show extensibility as well as the platform and language independence of the concept. Our evaluation using a real-world workload and implementation shows that event processing using the CEPLESS middleware is equally performant as native CEP systems.
Overall, this thesis contributes (i) a novel programming model and methods for transitions in CEP systems to support changing quality requirements, (ii) a novel unified communication model and efficient algorithms that accelerate query execution using the concepts of Information-centric Networking, and (iii) a novel serverless middleware with programming abstractions to achieve efficient execution and reuse of multiple and heterogeneous CEP execution environments.