Publikation
Evaluation of Edge-Cloud Systems: Approaches, tools and metrics
Benjamin Alt; Sebastian Bast; Matthias Babel; Julian Buitmann; Lars Creutz; Guido Dartmann; Marvin Ehaus; Thomas Engel; Fabian Gast; Nina Großegesse; Achim Guldner; Nils Jahnke; David Jilg; Marc-Fabian Körner; Justus Leskow; Johannes Mundorf; Stefan Naumann; Alexandra Pehlken; Marvin Schacht; Leo Schick; Tobias Schlagenhauf; Alexander Schultheis; Karsten Schörner; Sebastian Straub; Jens Strüker; Ralph Traphöner; Moritz-André Weiher
2025.
Zusammenfassung
Edge-cloud systems (ECS) serve as the foundation for new types of intelligent services that provide economic and social value across many industries. ECS address key challenges in modern software systems, such as meeting latency requirements, ensuring data security and privacy, and optimizing resource utilization by leveraging diverse software and hardware components along the continuum from the edge to the cloud. Depending on the use case, various ECS configurations are possible.
However, the tangible added value of a particular ECS configuration over its alternatives can only be demonstrated if its advantages are empirically validated. The primary tool for this purpose is the evaluation of an ECS. Yet, the unique characteristics of ECS make it difficult to plan, conduct, and assess such evaluations. The main obstacles include the distributed structure of ECS, the wide variety and heterogeneity of the hardware and software components involved, and the dynamic nature of ECS—for example, individual system components may be replaced or updated over their lifetime. As a result, users often find it challenging to identify context-relevant evaluation criteria and select appropriate metrics for analyzing these aspects in a specific ECS configuration.
Against this backdrop, early adopters of edge computing from projects within the technology program "Edge Data Economy" of the German Federal Ministry of Research, Technology and Space (BMFTR) and two projects from related BMFTR funding programs have joined forces to form a task force. The aim of this task force was to discuss and consolidate the individual findings and solutions of these early adopters concerning the evaluation of ECS based on their practical project experiences. Members of the following projects were involved in the task force: EASY, EDNA, ESCOM, DEER, and SECAI (Edge Data Economy), ECO:DIGIT (GreenTech Innovation Competition), and GEMIMEG-II. This guide makes the task force's results accessible to a broader audience.
The guidance is primarily aimed at practitioners responsible for system architecture and system integration of ECS, as well as those in development areas who wish to carry out a holistic evaluation of their ECS or obtain information on the assessment of individual characteristics of ECS. These readers will gain a comprehensive insight into the relevant concepts for evaluating ECS and receive practical guidance on how to carry out an evaluation of ECS in practice. In addition, executives, project management teams, and other stakeholders interested in ECS assessment will benefit from a general overview of the topic. Finally, the guide also addresses policymakers and researchers by offering recommendations for action to create the necessary framework conditions for sustainable ECS assessment capabilities in organizations.
Specifically, this guide first presents an overview of the core concepts of ECS assessment that has been harmonized across the task force participants. This includes, in particular, the presentation of a five-step procedure model from the field of software systems for the assessment of ECS and a consolidated overview of relevant assessment criteria and associated metrics from the areas of organization, technology, and the environment. To provide hands-on insights into the evaluation of specific criteria—such as resource utilization, network performance, or the environmental impact of ECS—the early adopters also present selected approaches and findings from their research projects that extend beyond the current state of the art.
Drawing on insights from research and development activities, the guide highlights shared economic, technical, and organizational challenges in ECS evaluation. Key challenges include the dependency of evaluations on specific application contexts, the high costs associated with evaluating certain criteria, and the limited practical expertise in ECS assessment.
Finally, the guide presents jointly developed recommendations for action in two key areas. The first area provides practical guidance for stakeholders directly involved in ECS evaluation (e.g., users, platform operators, system integrators, and component manufacturers). It outlines how a company-wide evaluation framework can be established and identifies critical criteria—such as resilience, data protection, and regulatory compliance—that should be considered. The second area addresses needed improvements to the general conditions required to enable organizations to evaluate ECS sustainably over the long term. This section is directed particularly toward research institutions and policymakers. The recommendations include capacity building, the establishment of transparency standards, semantic standardization, enhancing the explainability of (AI-based) systems, and creating low-threshold entry points for ECS adoption.
