Modeling and reasoning about "stuff"

Shaoxiong Yao, Jiaheng Han, Baoyu Liu, Yunzhu Li, Kaifeng Zhang, Noah Franceschini, Joao Correia Marques, Kris Hauser

Summary

Prevailing models in robotics reason about the world as a collection of rigid objects, but this is an inappropriate abstraction for manipulating cloth, ropes, piles of objects, plants, and natural terrain. We are investigating novel representations of "stuff" that are built de novo from visual and tactile perception data, whose properties are learned continuously through interaction. Volumetric Stiffness Fields, Graph Neural Networks, Neural Dynamics, and 3D metric-semantic maps are examples of models being investigated in this research. Such representations allow robots to learn about their environment without having preconceived notions of individual objects, their physical properties, or how they interact. For a variety of domains and materials, we are investigating the strengths and weaknesses of these techniques in modeling complex interactions, uncertainty, and multi-modal correlations between appearance and physical properties. Applications are being studied in agriculture, construction, and household object manipulation.

• K. Zhang, B. Li, K. Hauser, and Y. Li. Particle-Grid Neural Dynamics for Learning Deformable Object Models from Depth Images. Robotics: Science and Systems (RSS), June 2025 (to appear) Supplemental video
• J.M.C. Marques, N. Dengler, T. Zaenker, J. Mücke, S. Wang, M. Bennewitz, K. Hauser. Map Space Belief Prediction for Manipulation-Enhanced Mapping. Robotics: Science and Systems (RSS), June 2025 (to appear) Supplemental video
• J. Han, S. Yao, and K. Hauser. Estimating High-Resolution Neural Stiffness Fields using Visuotactile Sensors. IEEE International Conference on Robotics and Automation (ICRA), May 2025. Supplemental video
• N. Franceschini, P. Thangeda, M. Ornik, and K. Hauser. Autonomous Excavation of Challenging Terrain using Oscillatory Primitives and Adaptive Impedance Control. IEEE International Conference on Robotics and Automation (ICRA), May 2025. Supplemental video
• S. Yao, S. Pan, M. Bennewitz, and K. Hauser. Safe Leaf Manipulation for Accurate Shape and Pose Estimation of Occluded Fruits. IEEE International Conference on Robotics and Automation (ICRA), May 2025. Supplemental video
• N. Dengler, J.M.C. Marques, J. Mücke, T. Zaenker, S. Wang, K. Hauser, and M. Bennewitz. Manipulation-Enhanced Spatial Mapping via Belief Prediction. German Robotics Conference, March 2025.
• S. Yao, Y. Zhu, and K. Hauser. Structured Bayesian Meta-Learning for Data-Efficient Visual-Tactile Model Estimation. Conference on Robot Learning (CoRL), November 2024.
  Also in ICRA 2024 4th Workshop on Representing and Manipulating Deformable Objects, May 17, 2024.
• K. Zhang, B. Li, K. Hauser, Y. Li. AdaptiGraph: Material-Adaptive Graph-Based Neural Dynamics for Robotic Manipulation. Robotics: Science and Systems (RSS), July 2024.
  Also in ICRA 2024 4th Workshop on Representing and Manipulating Deformable Objects, May 17, 2024. Best Workshop Paper Award
• S. Yao and K. Hauser. Estimating Tactile Models of Heterogeneous Deformable Objects in Real Time. IEEE International Conference on Robotics and Automation (ICRA), May 2023.

A Volumetric Stiffness Field of a tree is predicted in zero-shot fashion using a meta-learned prior, and then updated in real time to match touch data.

NeuralVSF is a NeRF-like representation of Volumetric Stiffness Fields that is amenable to learning through high-resolution visuotactile sensors. Applications include blind object localization.

Patrticle-grid neural dynamics learns to represent action-conditioned deformations of arbitrary 3D objects from RGB-D videos.