Publikation
DIN SPEC 91527:2025-12, Goals, Methods and Metrics for Automated/Semi-Automated Runtime Monitoring of AI Systems for Non-Adversarial Performance Degradations
Daniel Weimer; André Meyer-Vitali; Anne Sielemann; Jens Ziehn; Jingxing Zhou; Yunus Bulut; Michael Graf; Sebastian Krauß
DIN Media GmbH, 12/2025.
Zusammenfassung
The monitoring of system components that are using artificial intelligence (AI), including but not limited to machine learning (ML), is a critical task during the operation. On a technical level, the operation of systems at advanced levels of autonomy requires monitoring of these systems either by another system, by the system’s capability of self-monitoring or by human experts. The health of the systems using AI can degrade over the operation time for various reasons. These reasons can include, but are not limited to:
— a degradation of the internal AI system, e.g. through flawed updates or erroneous learning;
— a degradation of the surrounding technical systems, e.g. through wear and tear in sensors or actors;
— a “domain shift” in the operational environment, for example changed lighting conditions, changed seasons, changed population characteristics, etc.;
— a “concept drift” due to changes in the meaning of concepts over time or location, as well as the emergence of new semantic concepts.
NOTE Cyber security-related monitoring is excluded from the scope of this document and left for future work.
To support the implementation of technical measures during the runtime of an AI component of a system, this document provides guidance on methods and metrics that can be used to detect a degradation in AI component health or performance at runtime, and that – to a certain degree – can operate without or with only limited human supervision. The implementation of such methods can then enable the selection of adequate, use case-dependent thresholds on which external measures, such as a re-evaluation or maintenance of the system are prescribed.
The list of methods is also provided as a digital attachment (xlsx-file) for convenient usage.