gradSim: Differentiable simulation for system identification and visuomotor control

gradSim: A differentiable simulator (physics and rendering engine)

Abstract

In this paper, we tackle the problem of estimating object physical properties such as mass, friction, and elasticity directly from video sequences. Such a system identification problem is fundamentally ill-posed due to the loss of information during image formation. Current best solutions to the problem require precise 3D labels which are labor intensive to gather, and infeasible to create for many systems such as deformable solids or cloth. In this work we present gradSim, a framework that overcomes the dependence on 3D supervision by combining differentiable multiphysics simulation and differentiable rendering to jointly model the evolution of scene dynamics and image formation. This unique combination enables backpropagation from pixels in a video sequence through to the underlying physical attributes that generated them. Furthermore, our unified computation graph across dynamics and rendering engines enables the learning of challenging visuomotor control tasks, without relying on state-based (3D) supervision, while obtaining performance competitive to/better than techniques that require precise 3D labels.

Publication
In International Conference on Learning Representations
Click the Cite button above to demo the feature to enable visitors to import publication metadata into their reference management software.
Click the Slides button above to demo Academic’s Markdown slides feature.

Supplementary notes can be added here, including code and math.

Krishna Murthy Jatavallabhula
Krishna Murthy Jatavallabhula
PhD Candidate

My research blends robotics, computer vision, graphics, and physics with deep learning.

Florian Golemo
Postdoc

I lead the Montreal robotics and embodied AI lab. I am affiliated with Université de Montréal, Mila, and I hold a CIFAR AI chair.

Liam Paull
Assistant Professor

I lead the Montreal robotics and embodied AI lab. I am affiliated with Université de Montréal, Mila, and I hold a CIFAR AI chair.

Florian Shkurti
Assistant Professor
Derek Nowrouzezahrai
Associate Professor
Sanja Fidler
Associate Professor
comments powered by Disqus

Related