Publikation
Differentiable Inverse Graphics for Zero-Shot Scene Reconstruction and Robot Grasping
Luis Octavio Arriaga Camargo; Proneet Kumar Sharma; Jichen Guo; Marc Otto; Siddhant Kadwe; Rebecca Adam
In: IEEE Robotics and Automation Letters (RA-L), Vol. 11, No. 6, Pages 7684-7691, IEEE, 5/2026.
Zusammenfassung
Operating effectively in novel real-world environments requires robotic systems to estimate and interact with previously unseen objects. Current state-of-the-art models address this challenge by using large amounts of training data and test-time
samples to build black-box scene representations. In this work, we introduce a differentiable neuro-graphics model that combines neural foundation models with physics-based differentiable rendering to perform zero-shot scene reconstruction and robot grasping without relying on any additional 3D data or test-time samples. Our model solves a series of constrained optimization problems to estimate physically consistent scene parameters, such as meshes, lighting conditions, material properties, and 6D poses of previously unseen objects from a single RGBD image and bounding boxes. We evaluated our approach on standard model-free fewshot benchmarks and demonstrated that it outperforms existing
algorithms for model-free few-shot pose estimation. Furthermore, we validated the accuracy of our scene reconstructions by applying our algorithm to a zero-shot grasping task. By enabling zero-shot, physically-consistent scene reconstruction and grasping without reliance on extensive datasets or test-time sampling, our approach offers a pathway toward more data efficient, interpretable and generalizable robot autonomy in novel environments.
Projekte
- PhysWM - PhysWM - Lernen eines kausalen Weltmodells der Physik
- RIG - Robotics Institute Germany
- ActGPT - Adaptive robot ConTrol with Generative Pre-trained Transformers