Dexterous manipulation requires careful reasoning over extrinsic contacts. The prevalence of deforming tools in human environments, the use of deformable sensors, and the increasing number of soft robots yields a need for approaches that enable dexterous manipulation through contact reasoning where not all contacts are well characterized by classical rigid body contact models. Here, we consider the case of a deforming tool dexterously manipulating a rigid object. We propose a hybrid learning and first-principles approach to the modeling of simultaneous motion and force transfer of tools and objects. The learned module is responsible for jointly estimating the rigid object's motion and the deformable tool's imparted contact forces. We then propose a Contact Quadratic Program to recover forces between the environment and object subject to quasi-static equilibrium and Coulomb friction. The results is a system capable of modeling both intrinsic and extrinsic motions, contacts, and forces during dexterous deformable manipulation. We train our method in simulation and show that our method outperforms baselines under varying block geometries and physical properties, during pushing and pivoting manipulations, and demonstrate transfer to real world interactions.
We show one-step predictions of object motion, deformable tool contacts, and extrinsic contacts on test simulation interactions for our proposed method. Ground truth is shown as semi-transparent, predictions as opaque. Friction cones are shown in grey.
We demonstrate qualitative transfer to real world interactions, performed using a Franka Emika Panda. Point clouds are segmented using SAM and blocks are tracked using the Apriltags. In the paper, we evaluate block tracking and use a force-sensor in the environment to evaluate force predictions.
We apply our model to a force tracking task, where we pivot an object while trying to track a desired extrinsic contact force. We use our model to plan the arc length of the pivot, allowing the model to press in or out as it pivots.
@article{vandermerwe2025deformrigidcontact,
author = {Van der Merwe, Mark and Oller, Miquel and Berenson, Dmitry and Fazeli, Nima},
title = {Estimating Deformable-Rigid Contact Interactions for a Deformable Tool via Learning and Model-Based Optimization},
journal = {IEEE RA-L 2025},
year = {2025},
}