Publication

Mesh-based Object Tracking for Dynamic Semantic 3D Scene Graphs via Ray Tracing

Lennart Niecksch; Alexander Mock; Felix Igelbrink; Thomas Wiemann; Joachim Hertzberg

In: Proceedings of the RSS'24 Workshop on Semantics for Robotics. RSS Workshop on Semantics for Robotics, From Environment Understanding and Reasoning to Safe Interaction, July 15, Delft, Netherlands, RSS Foundation, 2024.

Abstract

In this paper, we present a novel method for 3D geometric scene graph generation using range sensors and RGB cameras. We initially detect instance-wise keypoints with a YOLOv8s model to compute 6D pose estimates of known objects by solving PnP. We use a ray tracing approach to track a geometric scene graph consisting of mesh models of object instances. In contrast to classical point-to-point matching, this leads to more robust results, especially under occlusions between objects instances. We show that using this hybrid strategy leads to robust self-localization, pre-segmentation of the range sensor data and accurate pose tracking of objects using the same environmental representation. All detected objects are integrated into a semantic scene graph. This scene graph then serves as a front end to a semantic mapping framework to allow spational reasoning.