No-IDLE – Foundations for Cooperative and Human-Centered AI: When AI Works Together with People

With No-IDLE, the Interactive Machine Learning (IML) research department successfully completed its first strategic research project and established new scientific foundations for Interactive Deep Learning (IDL) – a research approach that actively involves humans in the learning, decision-making, and adaptation processes of AI systems.

Funded by the German Federal Ministry of Research, Technology and Space (BMFTR), the project pursued the vision of creating AI systems that are not only powerful but also understandable, adaptable, and collaborative.

Thinking Human and AI Together

At the heart of No-IDLE was the question of how humans and AI systems can collaborate effectively, particularly in situations where:

• only limited amounts of data are available,
• users do not possess machine learning expertise,
• tasks are highly individual or creative, or
• fully automated decision-making is neither desirable nor appropriate.

The project explored how people can interact with AI systems that:

• interpret gaze behavior,
• understand spoken language as feedback,
• explain medical decisions transparently, or
• support creative tasks such as the creation of personalized photo books.

Technologically, No-IDLE combined modern deep learning approaches with Explainable AI (XAI), foundation models, multimodal interaction, and user-centered interaction design. This included technologies such as eye tracking, virtual reality, multimodal human-computer interaction, and so-called mixed-initiative systems, in which humans and AI jointly take initiative.

A central research principle was the so-called “binocular view” from the field of Intelligent User Interfaces: algorithms and user interfaces are not developed separately but conceived together. This enables technical performance and usability to be optimized simultaneously.

Interactive AI Instead of a Black Box

Throughout the project, numerous interactive AI systems and demonstrators were developed, illustrating how AI can become more transparent, adaptive, and user-oriented.

With CUTIE, the project created an interactive human-in-the-loop system for personalized and context-sensitive image captions. Users can correct, enrich, or contextualize automatically generated descriptions—for example by identifying people or locations—allowing the system to gradually adapt to individual preferences.

The multimodal annotation tool eyeNotate supported the analysis of mobile eye-tracking data. AI-based suggestions helped users annotate gaze behavior more efficiently and interactively analyze multimodal data streams.

Train the Spire, a game-based system for AI-supported data collection, combined methods from gamification and interactive machine learning to generate high-quality training data through playful interaction.

In addition, No-IDLE developed VR-based environments for interactive photo book creation, enabling users to collaborate with AI systems through speech, gaze interaction, and controller-based input.

Particular attention was given to multimodal forms of human feedback, including speech, eye movements, gestures, and implicit interaction patterns. This resulted in AI systems capable of continuously adapting to user behavior and integrating human feedback directly into their learning processes.

From Explainable AI to Trustworthy AI

A central research question was how trust between humans and AI is established. Even highly capable systems lose their value if their decisions cannot be understood or if users feel they are losing control.

For this reason, No-IDLE investigated not only how AI systems can explain their decisions, but also how trust can be established, damaged, and restored.

To address these questions, the project developed new methods for:

• user-adaptive explanation generation,
• concept-based explanations of medical AI decisions using so-called Concept Bottleneck Models,
• multimodal presentation of explainable AI outputs, and
• investigating trust formation and trust recovery in human-AI interactions.

Particularly in sensitive application domains such as healthcare, the project demonstrated how explainable and interactive AI systems can support physicians without replacing human expertise. Among the outcomes were AI-based decision support systems that make diagnostic processes more transparent and assist medical professionals in interpreting complex imaging data.

Scientific Excellence and Sustainable Transfer

Over its three-year duration, No-IDLE achieved all defined project milestones and made fundamental contributions at the intersection of:

Key outcomes include:

• 50 peer-reviewed scientific publications,
• including 22 full papers,
• 17 student theses,
• 2 doctoral dissertations, and
• several functional demonstrators and prototypes.

The technologies and methods developed within the project are already being applied in a variety of domains, including medical AI, sustainable AI systems, multimodal human-machine interaction, and future human-AI collaboration projects.

At the same time, No-IDLE is closely connected to numerous cross-site DFKI research activities in multimodal, explainable, and human-centered AI. Related research questions are being addressed in areas such as Cognitive Assistants, Medical AI, Cooperative AI Systems, and Interactive Human-Machine Communication.

Furthermore, key research topics from No-IDLE are being continued within the recently launched AMENABLE project in collaboration with the MAP and CAS research departments. In doing so, DFKI is not only advancing research on interactive and trustworthy AI but also strengthening collaboration among its research departments in Lower Saxony.

Contributing to the Next Generation of AI Systems

No-IDLE demonstrates that the future of powerful AI does not lie solely in ever-larger models, but in systems that understand people, collaborate with them, and make their decisions comprehensible.

The scientific foundations for Interactive Deep Learning developed within the project therefore make an important contribution to a new generation of trustworthy AI—cooperative, transparent, and human-centered. At the same time, No-IDLE strengthens Germany’s position as a leading research location for interactive and trustworthy artificial intelligence.