The work was the result of a collaboration between researchers from TU Darmstadt, Ruhr University Bochum, hessian.AI, and the SAIDE team. Contributors included Alessandro Ferri, Uélison Jean Lopes dos Santos, Szilard Nistor, Pratyush Agnihotri, Manisha Luthra, and Carsten Binnig.
With the rapid growth of video, image, and text data, traditional stream-processing systems are increasingly reaching their limits. These systems were originally designed for structured data and have limited ability to process multimodal data streams. While multimodal large language models (MLLMs) understand such data very well, their direct use in streaming pipelines is computationally intensive and significantly slows down processing.
The award-winning Samsara system optimizes the processing of multimodal data streams through the intelligent use of AI models and dynamically adapts processing plans to data and queries. Acting as a kind of “super-optimizer,” Samsara combines three optimization approaches:
- Semantic optimization: Using knowledge about the data to avoid redundant work. For example, nearly identical camera images do not need to be analyzed multiple times.
- Logical optimization: Rearranging operators so that low-cost filters are executed before computationally expensive model calls.
- Physical optimization: Using smaller, task-specific models when they are sufficient for the task at hand.
At the conference, the team demonstrated the technology using a live traffic monitoring application. Visitors were able to try out various analysis tasks firsthand and compare the performance of a conventional processing plan with that of an optimized one. Samsara achieved processing rates of up to 54 frames per second, significantly outperforming naive approaches.
The scientific paper is available online: https://www.vldb.org/pvldb/vol19/p4666-santos.pdf.