Publication
Toward a Holistic Evaluation of Edge-Cloud Systems: Insights from a Task Force
Nils Jahnke; Nina Großegesse; Brian-Frederik Jahnke; David Jilg; Justus Leskow; Rebecca Schmook; Alexander Schultheis; Jiaqi Wang
In: SSRN Electronic Journal (SSRN), Pages 1-16, SSRN, 8/2025.
Abstract
Edge-Cloud Systems (ECS) are essential for realizing the potential of Internet of Things (IoT) data by integrating software and hardware components along the Edge-Cloud Continuum and combining the advantages of both computing paradigms. Given their diverse configurations and viable alternatives such as cloud-only solutions, ECS must be systematically assessed to demonstrate their tangible benefits to stakeholders. However, existing evaluation and benchmarking approaches across industry and academia remain fragmented, and a holistic evaluation framework is lacking. This article synthesizes the outcomes of a 2024-2025 task force involving 27 researchers and practitioners from Germany and Austria, consolidating existing approaches, tools, and metrics toward a comprehensive framework for ECS evaluation. The framework defines 16 evaluation criteria across three dimensions: technology (e.g., reliability, availability, resilience), organization (e.g., maintenance and administrative effort, interoperability, cost), and environment (e.g., regulatory compliance, data localization, environmental impact). Each criterion is linked to context-specific metrics and measurement methods. We illustrate the framework's practical application through use cases in networking, edge-enabled Artificial Intelligence, and carbon accounting. Finally, drawing on task force insights, we propose a practice-oriented research agenda to advance holistic ECS evaluation.