ViHaTeleop: A Low-cost, Lightweight Visual-Haptic Teleoperation System for Dexterous Manipulation Learning

IROS 2026

Fucai Zhu Yanhou Lai Paul Maestre Koichi Hashimoto
Tohoku University
arXiv Paper Video Hardware Code BibTeX

Video

ViHaTeleop system overview
Hardware overview and control architecture of ViHaTeleop.

Abstract

Learning from demonstration is a promising approach for dexterous manipulation, but collecting high-quality contact-critical demonstrations remains difficult with low-cost teleoperation hardware. We present ViHaTeleop, a lightweight (0.7 kg), low-cost ($550) visual-haptic teleoperation system with SLAM-based wrist tracking, camera-based hand tracking, and finger-wise vibrotactile feedback through Linear Resonant Actuators (LRA). The system includes several design choices (LED illumination, fisheye hand camera, and tactile-aware retargeting constraints) and is deployed on Franka + LEAP Hand + 9DTact in both real and simulated environments. Under matched with/without-haptic conditions with nine participants across six contact-critical tasks, haptics improved success rates across all tasks (+2.2 to +15.6 percentage points), while completion-time effects were task-dependent. Subjective ratings showed significant gains in contact clarity and grasp confidence in both simulation and real-world settings (Wilcoxon signed-rank, p < 0.05). We also integrate a lightweight depth-camera-based tactile proxy in Isaac Sim, enabling a full pipeline from multi-modal demonstration collection to visual-tactile policy training. Preliminary downstream validation by training visual-tactile policies from collected demonstrations shows tactile cues benefit contact-critical subtasks (peg-in-hole: +17 percentage points over vision-only).

Highlights

Low-cost Wearable Setup

$550 total hardware cost, 0.7 kg wearable burden, wrist-mounted fisheye camera, tracker-based wrist pose, and finger-wise vibrotactile feedback.

Contact-Critical Teleoperation

Designed for delicate tasks where contact onset, stable grasping, insertion, and slip awareness materially affect demonstration quality.

Simulation-to-Policy Pipeline

Includes teleoperation, tactile-aware retargeting, Isaac Sim tactile proxy, multimodal data collection, and downstream visual-tactile policy learning.

Hardware components of ViHaTeleop
Hardware components and wearable configuration.
LED illumination comparison
LED illumination improves hand-tracking robustness in difficult lighting.
Tasks used in the ViHaTeleop evaluation
Contact-critical task suite used in the evaluation.
User study summary with and without haptics
User study summary under matched with/without-haptic conditions.
Tactile-aware retargeting constraints
Tactile-aware retargeting encourages more useful fingertip contact geometry for flat tactile sensors.

Policy Learning Pipeline

Visual-tactile representation for policy learning
Visual-tactile representation used for downstream policy learning.
Dual-purpose tactile proxy pipeline in simulation
Dual-purpose tactile proxy pipeline in simulation. The same depth-camera-based tactile proxy supports both real-time haptic feedback for teleoperation and visual-tactile point cloud generation for downstream policy learning.

The visual-tactile representation and the depth-camera-based tactile proxy together support the full pipeline from demonstration collection to downstream policy learning.

Code

Main Repository

This is the primary project repository for dataset generation, data processing, and policy-side rollout integration.

github.com/idombanker/ViHaTeleop

Citation

@inproceedings{vihateleop2026,
  title={ViHaTeleop: A Low-Cost, Lightweight Visual-Haptic Teleoperation System for Dexterous Manipulation Learning},
  author={Zhu, Fucai and Lai, Yanhou and Maestre, Paul and Hashimoto, Koichi},
  booktitle={IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
  year={2026}
}