Publikation
Automatic Physically-Based Sim2Real for Tactile Images through Differentiable Path-Tracing Rendering (^mbox*)
Guillaume Duret; Anna Samsonenko; Florence Zara; Jan Peters; Liming Chen
In: IEEE International Conference on Robotics and Automation, ICRA 2026, Vienna, Austria, June 1-5, 2026. IEEE International Conference on Robotics and Automation (ICRA), Pages 22516-22522, IEEE, 2026.
Zusammenfassung
(Already published work) High-fidelity simulation
of vision-based tactile sensors is essential for developing data-
driven robotic manipulation algorithms. However, a significant
sim-to-real gap persists due to the difficulty in modeling
complex optical effects, such as refraction through protective
glass layers, and in accurately estimating physical parameters
like sensor pose and lighting. To bridge this gap, we introduce a
novel, fully differentiable pipeline for visual tactile simulation.
Leveraging a differentiable path tracer, our method optimizes
critical parameters—including camera pose, lighting conditions,
and object texture—directly from just three real images. This
approach achieves highly realistic simulations with physically
accurate light transport and glass refraction. We validate our
method through a comprehensive benchmark against real-world
data, demonstrating state-of-the-art sim-to-real accuracy. We
also enable novel applications, such as mesh reconstruction
from a single tactile image via inverse rendering. To overcome
the computational cost of path tracing, we further use an
image-to-image translation model. This model uses high-fidelity
simulated data alongside Normalized Object Coordinate Space
(NOCS) maps as input, preserving crucial deformation infor-
mation while enabling rapid inference. The code is available.