ContactWorld

Overview

ContactWorld contains 12 contact-rich manipulation tasks spanning insertion, screwing, disassembly, and exploration. Each task provides six synchronized visual-tactile observation modalities.

Wrist View and Front View provide RGB visual observations. Point Cloud captures structured 3D geometry. TacDepth represents tactile depth deformation, TacRGB records high-resolution tactile appearance, and TacFF encodes contact force distributions and directions.

World Model Framework

ContactWorld trains latent world models using multimodal visual-tactile observations and performs goal-conditioned planning through autoregressive latent rollout and model-predictive control.

(A) Training: multimodal latent world model learning. (B) Planning: goal-conditioned latent rollout with MPC.

What Matters in Vision-Tactile World Models?

ContactWorld reveals three key factors that determine world-model planning performance in contact-rich manipulation.

Representation structure matters. Representations with stronger spatial structure and temporal continuity consistently achieve better planning performance. Point clouds outperform image-based observations, while PointCloud+TacFF achieves the strongest overall results.

Cross-modal compatibility matters. Tactile sensing is not universally beneficial. The effectiveness of tactile feedback depends strongly on how well it aligns with the visual representation. PointCloud+TacFF forms the most compatible visual-tactile pair.

Long-horizon robustness matters. The benefit of tactile sensing grows as planning horizons increase. Lower rollout prediction error translates into larger planning gains for distant goals.

World Model Planning Gallery

Representative rollouts across all ContactWorld tasks. Each video compares four world models: Wrist View, Front View, PointCloud, and PointCloud+TacFF. More modality combinations can be reproduced using the released code.

Real-World Deployment

Autonomous vision-tactile world-model planning on physical robots.

BibTeX

@article{zhang2026contactworld,
  title={ContactWorld: What Matters in Vision-Tactile World Models for Contact-Rich Manipulation},
  author={Zhang, Zhiyuan and Zhou, Pokuang and Zhang, Kaidi and Desai, Adeesh and Amosa, Temitope and Soleymanzadeh, Davood and Lei, Jiuzhou and Zheng, Minghui and She, Yu},
  journal={arXiv preprint arXiv:2606.13877},
  year={2026}
}