Universal Manipulation Interface

Universal Manipulation Interface (UMI) is an open-source system for collecting robot manipulation training data with a handheld, camera-equipped gripper instead of an actual robot. An operator holds a 3D-printed gripper fitted with an action camera, performs a task by hand (pouring from a cup, folding a shirt, washing a dish), and the resulting video becomes a training demonstration for a robot control policy that can later run, largely unmodified, on a real robotic arm. UMI was developed by researchers at Stanford University, Columbia University, and Toyota Research Institute and presented at Robotics: Science and Systems (RSS) 2024, where it was a finalist for the conference's Outstanding Systems Paper Award.^[1][6] It sits within robotics, specifically the subfields of robot learning and imitation learning: rather than hand-coding a robot's behavior or training it through trial and error, UMI produces the kind of labeled example a policy can directly copy.

In brief

Strap an action camera to a 3D-printed gripper, add a fisheye lens and two angled mirrors, and carry it into a kitchen or a cafe. As a person uses the gripper like a pair of tongs to do an ordinary task, the camera records video and motion data precise enough to reconstruct exactly where the gripper moved and how wide its fingers opened at every instant. Feed enough of those recordings into a learning algorithm, and the result is software that a real robot arm, fitted with a matching gripper and camera, can run to perform the same task on its own.

Origins

UMI was described in the paper "Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots," by Cheng Chi and Zhenjia Xu (equal contributors), Chuer Pan, Eric Cousineau, Benjamin Burchfiel, Siyuan Feng, Russ Tedrake, and Shuran Song.^[1] Chi, Xu, Pan, and Song were affiliated with both Stanford University and Columbia University at the time; Cousineau, Burchfiel, Feng, and Tedrake were at Toyota Research Institute.^[1] Song has since moved her lab to Stanford full time, where she is an assistant professor of electrical engineering leading the Robotics and Embodied AI Lab, after previously holding a Columbia faculty position.^[8] Every UMI author except Chuer Pan also co-authored Diffusion Policy, an RSS 2023 paper from largely the same team supplying the policy-learning algorithm UMI trains with.^[9]

The paper first appeared on arXiv in February 2024 and was formally published at RSS 2024 in Delft, Netherlands, that July.^[1][4][5] Its hardware designs (3D-printable parts, a bill of materials, camera calibration steps) and its training and deployment code were released as open source alongside the paper and remain maintained on GitHub.^[2][3]

Hardware design

UMI's data-collection hardware is deliberately minimal: a trigger-activated, 3D-printed parallel-jaw gripper with soft, compliant fingers, carrying a single off-the-shelf action camera as its only sensor.^[1] The goal was an interface an operator could take to any location and start recording within minutes, with no calibration rig, no external cameras, and no separate computer to carry.^[1]

Scroll sideways for more →

The fisheye lens is used deliberately without correcting its distortion: a rectilinear correction would stretch the edges of a 155-degree view so much that it wastes most of its resolution on the periphery, while the raw fisheye image keeps resolution concentrated where the fingers and the manipulated object actually are.^[1] The side mirrors add stereo-like depth perception without the cost or weight of a second camera, but only helped once the images inside them were digitally flipped to correct their left-right reversal; in the paper's own ablation, that flip raised cup-arrangement accuracy from 85 to 100 percent.^[1] Grasp force is not measured by a dedicated force-torque sensor; instead, the soft fingers deform slightly under load, and that deformation, combined with continuous fiducial-marker tracking of finger width, gives a trained policy an implicit read on grip strength.^[1]

Tracking motion without a robot in the loop

A handheld gripper has no joint encoders and no fixed base, so UMI has to recover the camera's full six-degree-of-freedom pose (position and orientation over time) from video and inertial data alone, using an inertial-monocular SLAM (simultaneous localization and mapping) system built on ORB-SLAM3 that fuses the GoPro's visual feed with its onboard IMU readings.^[1] Combining vision and inertial data lets the system recover a trajectory at true, real-world scale, keep tracking briefly through motion blur, and avoid losing its position when the camera looks down at a bare tabletop for a moment.^[1]

For each new location, an operator first records a short mapping pass to build a 3D map of the space; every demonstration recorded afterward is localized against that same map, so all of them share one coordinate system.^[1] That matters most for two-handed tasks: because both grippers in a bimanual demonstration are localized to the same map, the system can compute one gripper's pose relative to the other at every instant, which a trained policy uses to keep two robot arms coordinated.^[1] Checked against a motion-capture rig across 7 single-gripper and 7 bimanual tasks, the pipeline reached a mean absolute trajectory error of 6.1 millimeters in position and 3.5 degrees in rotation, with a relative error between two grippers of 10.1 millimeters and 0.8 degrees.^[1]

From demonstration to a deployable policy

UMI trains a visuomotor policy: software mapping camera images and proprioceptive readings (pose and gripper-width history) to a sequence of future actions, using Diffusion Policy, which treats action prediction as a denoising process able to represent several valid ways of completing the same motion rather than regressing to one averaged answer.^[1][9] Camera images pass through a ResNet-34 or a CLIP-pretrained Vision Transformer encoder; the longer-horizon dish-washing task only became solvable with the CLIP-pretrained encoder, since a ResNet-34 policy scored zero.^[1]

Two interface choices make a policy trained on handheld video deployable on an actual robot. The first is a relative-trajectory action representation: rather than predicting a gripper's next pose in a fixed world coordinate frame (which requires calibrating demonstrations and robot to the same frame) or as a step from the immediately preceding pose (which accumulates drift), UMI predicts a short window of future poses, each expressed relative to the pose at the start of that window.^[1] That representation tolerates the small errors any tracking system introduces and survives the camera shifting slightly between demonstrations, and the same relative encoding serves as the policy's proprioceptive input, so little changes when a human hand is replaced by a robot arm carrying a matching gripper and camera.^[1]

The second is latency matching. A physical robot arm has real execution lag (about 100 milliseconds for the arm, 120 for the gripper, by the paper's measurements) between a commanded pose and the hardware reaching it, and its sensors report position, gripper width, and images on different delays; a demonstration video has none of this, since its frames and inertial readings are effectively synchronized.^[1] A policy trained on clean, synchronized data deployed naively onto laggy hardware drifts outside the range of inputs it learned on and performs worse, so UMI's policy interface measures each stream's latency in advance, aligns observations to whichever stream is slowest (usually the camera), and sends action commands early enough that the robot reaches the intended pose at the intended time once lag is accounted for.^[1] Removing latency matching in an ablation on dynamic tossing dropped success from 87.5 to 57.5 percent and made the arm visibly jittery.^[1]

Because the pipeline treats the target robot as a detail handled at the policy interface rather than something baked into the raw data, a UMI-trained policy can move to new hardware with no retraining. The RSS paper demonstrated this directly: a policy trained on demonstrations captured for a UR5e, a collaborative robot arm from Universal Robots paired with a Schunk WSG-50 end effector, was deployed unmodified on a Franka Emika FR2 arm and reached 90 percent success (18 of 20) versus 100 percent on the original hardware, with both failures traced to joint-limit violations a different robot placement would have avoided.^[1] For two-armed tasks the policy also takes the relative pose between the two grippers as an explicit input; without it, an ablation on bimanual cloth folding found the arms lost coordination and success fell from 70 to 30 percent, usually because one arm grasped the cloth before the other was ready.^[1]

Why decoupling data collection from the robot matters

Before UMI, most manipulation policies were trained one of two ways. Teleoperation has an operator drive an actual robot arm (by joystick, puppeted leader arm, or VR controller) while the robot's own sensors record the trajectory.^[1] It produces clean, directly usable data, but a robot arm and workspace are expensive, fixed in place, and slow to reconfigure, which caps how many environments and objects a dataset can cover. The alternative, learning from passive human video already online, is nearly unlimited in scale and variety, but it carries no explicit action labels and shows a human body with different proportions and reach than a robot, so recovering a usable robot action from it remains an open, noisy problem.^[1] An earlier handheld device, Dobb-E, mounted an iPhone on a "reacher-grabber" tool, portable and robot-free like UMI, but validated mainly on quasi-static, single-arm tasks and requiring environment-specific fine-tuning before deployment.^[1]

UMI's bet is that a handheld gripper with a wrist-mounted camera is close enough to a robot's own end-effector view that a policy trained on one transfers to the other with only a hardware change at deployment, while the interface stays as portable and cheap as a camera. That closes both gaps at once: the training signal is precise, since SLAM recovers full 6-degree-of-freedom poses and continuous gripper width rather than only pixels, and the interface is light enough to carry into a kitchen, a cafe, or a park.^[1] Scaling cup-arrangement data collection to 1,400 demonstrations across 30 real-world locations in 12 person-hours, something no fixed robot lab could match, is what let the resulting policy generalize to environments it had never seen (see Results, below).^[1]

The framing also echoes a version of Moravec's paradox: identifying a coffee cup in an image is now a well-solved computer vision problem, while reliably picking one up by its handle at the right angle remains hard, in part because there has never been a data source for physical manipulation as large and varied as the internet's supply of text and images. By making that kind of data cheap to collect, UMI targets the data bottleneck directly rather than proposing a new learning algorithm.

Results

The RSS 2024 paper evaluated UMI on four real-world tasks, each chosen to stress a different capability.^[1]

Scroll sideways for more →

Cup arrangement tested prehensile actions (picking up a cup) and non-prehensile ones (pushing it to reorient the handle), including a case where two opposite rotations are equally valid. Dynamic tossing tested fast, precise hand-eye coordination against bins beyond the arm's static reach, forcing it to release objects in flight. Bimanual cloth folding tested two-arm synchronization on a deformable object, and dish washing chased a long horizon of dependent steps (faucet on, pick up plate, grab sponge, wash, rinse, replace plate, faucet off), including recovering if the researchers added more ketchup mid-task.^[1]

To test whether the interface's portability translated into real generalization, the authors scaled up cup-arrangement collection: three demonstrators spent 12 person-hours collecting 1,400 demonstrations across 30 locations (homes, offices, restaurants, outdoor settings), using cups of different colors, shapes, and materials.^[1] They then tested the policy in two environments never used for data collection: a metal cafe table surrounded by pedestrians, and a black water fountain with running water across its top. The policy reached a combined 43 of 60 trials (71.7 percent), including 75 percent on cups it had never seen, while a policy trained only on the earlier narrow-domain lab data scored 0 percent in both unseen settings.^[1]

UMI is also faster to collect data with than the SpaceMouse teleoperation setup it is meant to replace, though slower than an unencumbered human hand. On cup arrangement, an operator using UMI worked at 48 percent of bare-hand speed but more than three times faster than teleoperation; on dynamic tossing, at 64 percent of bare-hand speed, while the teleoperation setup failed to complete a single successful demonstration in fifteen minutes.^[1]

Limitations

The authors identify three specific limits.^[1] First, because a demonstration is recorded before any target robot is chosen, the system cannot know that robot's joint limits or kinematics while collecting data; it can only filter out infeasible trajectories afterward rather than learning around a specific robot's constraints, and the paper suggests an embodiment-aware policy architecture as future work. Second, the SLAM pipeline inherits ordinary visual SLAM's dependence on textured surroundings, so it struggles in feature-poor spaces such as a room with bare white walls; the authors propose adding static third-person cameras or extra fiducial markers as a fix. Third, and by the authors' own account the most persistent limitation, collecting data with a UMI gripper is still slower and less natural than using a bare hand, a gap attributed to the gripper's added weight and bulk and to a parallel-jaw gripper's far more limited degrees of freedom compared with a human hand. That last limitation is what several of UMI's direct successors, described next, were built specifically to close.

Follow-on research and the wider ecosystem

UMI's open-source release turned it into infrastructure other labs could build on rather than a one-off demonstration.

Scroll sideways for more →

FastUMI keeps the handheld form factor but trades the SLAM pipeline for an Intel RealSense T265 tracking module with a built-in IMU, at some hardware cost but with simpler setup and no SLAM calibration, and adds interchangeable fingertip attachments so the same handle works across several commercial parallel-jaw grippers.^[10] A follow-up release, FastUMI-100K, packages more than 100,000 demonstration trajectories across 54 household tasks collected the same way.^[17]

DexUMI, led by Mengda Xu with Han Zhang, Yifan Hou, Zhenjia Xu (a UMI co-author, now at NVIDIA), Linxi Fan, Manuela Veloso, and Shuran Song, tackles the gap the original paper called its own biggest limitation: a parallel-jaw gripper cannot capture the finger-by-finger detail of human dexterity.^[11] The operator wears an exoskeleton instead of a gripper, recording full finger motion with haptic feedback; software built on the ProPainter video-inpainting model then erases the human hand and exoskeleton from the footage and renders a synthetic robot hand in their place, closing the motion gap and the visual gap together.^[11] Tested on the XHand and the Inspire Hand, DexUMI reached an average 86 percent success rate and was a Best Paper finalist at CoRL 2025.^[11][12]

UMI on Legs, by Huy Ha, Yihuai Gao, Zipeng Fu, Jie Tan, and Shuran Song, asks what happens if the robot is not fixed to a tabletop. It leaves an existing UMI-trained policy unchanged and wraps it in a separately trained whole-body controller, trained in NVIDIA's IsaacGym simulator on a quadruped carrying a custom arm, that converts the policy's gripper trajectory into coordinated leg and arm motion.^[13] Success rates across prehensile, non-prehensile, and dynamic tasks stayed above 70 percent, and a checkpoint originally trained for a stationary arm deployed with no retraining onto the walking robot.^[13][14]

An independently maintained Actuated UMI Gripper project replaces the original's hand-squeezed jaws with a Dynamixel servo motor and a GelSight Mini tactile sensor, closing under measured force instead of a human grip and narrowing the mechanical gap to an actual robot gripper.^[15] A separate line of work, including HumanoidUMI, pairs UMI-style handheld grippers with VR headsets to capture full upper-body humanoid robot demonstrations with no teleoperation rig at all, validated on the Unitree G1.^[16]

Industry adoption

Toyota Research Institute, whose researchers co-authored the original paper, has folded UMI-collected demonstrations into its own effort to pretrain general-purpose robot policies. TRI's Large Behavior Models were trained on a mixture that included 32 hours of UMI data alongside 468 hours of the institute's own teleoperated robot data, 45 hours of simulation data, and roughly 1,150 hours of internet video from the Open X-Embodiment dataset.^[18][19] TRI has pursued this approach jointly with Boston Dynamics under a research partnership announced in October 2024, aimed at applying large pretrained behavior models to humanoid robots.^[20]

UMI is also cited as related work in the technical report behind Physical Intelligence's pi-0 model, one of the more widely discussed generalist vision language action models released after UMI's publication.^[21] Newer papers, including RDT2 and VISTA, go further and train vision-language-action models directly on UMI-format data, treating its synchronized video-plus-pose recording as a shared standard for portable manipulation capture rather than building a competing one.^[22][23] Combined with datasets like FastUMI-100K, that pattern suggests the framework's lasting influence may rest less on its gripper design than on that recording format becoming common infrastructure across unrelated robot-learning projects.^[10][17]

Relevance to humanoid robotics and dexterous manipulation

The original UMI paper used simple two-finger parallel grippers, but the data bottleneck it targets is most acute for humanoid robots and multi-fingered dexterous hands, where the range of possible grasps and in-hand adjustments is far larger than for a parallel-jaw end effector. DexUMI's exoskeleton-and-inpainting approach carries UMI's core idea, precise demonstrations with no robot present during collection, into that harder setting, and HumanoidUMI pushes the same idea toward capturing full humanoid upper-body motion.^[11][16] The most capable multi-fingered robotic hands remain expensive and are rarely demonstrated on complex manipulation tasks; a cheap, portable way to collect the demonstration data their control software needs to learn from is one of the more direct routes toward closing that gap.

See also

• Imitation learning
• Robot manipulation
• Robot learning
• Dexterous hand
• Humanoid robot hands
• Sim-to-real transfer

References

1. Chi, C., Xu, Z., Pan, C., Cousineau, E., Burchfiel, B., Feng, S., Tedrake, R., and Song, S. "Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots." Robotics: Science and Systems (RSS) 2024, Delft, Netherlands, July 2024. https://www.roboticsproceedings.org/rss20/p045.pdf ↩
2. "Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots" (project website). https://umi-gripper.github.io/ ↩
3. real-stanford/universal_manipulation_interface (GitHub repository: hardware guide, calibration, training and deployment code). https://github.com/real-stanford/universal_manipulation_interface ↩
4. Chi, C. et al. "Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots." arXiv:2402.10329, submitted February 15, 2024, revised March 6, 2024. https://arxiv.org/abs/2402.10329 ↩
5. "Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots," Robotics: Science and Systems 2024 program page. https://roboticsconference.org/2024/program/papers/45/ ↩
6. "Awards," Robotics: Science and Systems 2024. https://roboticsconference.org/2024/program/awards/ ↩
7. Chi, C. et al. "Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots," full text. arXiv:2402.10329v3. https://arxiv.org/html/2402.10329v3 ↩
8. "Shuran Song," Stanford University faculty profile. https://profiles.stanford.edu/shuran-song ↩
9. Chi, C., Feng, S., Du, Y., Xu, Z., Cousineau, E., Burchfiel, B., Tedrake, R., and Song, S. "Diffusion Policy: Visuomotor Policy Learning via Action Diffusion." Robotics: Science and Systems (RSS) 2023. https://arxiv.org/abs/2303.04137 ↩
10. Zhaxizhuoma et al. "FastUMI: A Scalable and Hardware-Independent Universal Manipulation Interface with Dataset." arXiv:2409.19499. https://arxiv.org/abs/2409.19499 ↩
11. Xu, M., Zhang, H., Hou, Y., Xu, Z., Fan, L., Veloso, M., and Song, S. "DexUMI: Using Human Hand as the Universal Manipulation Interface for Dexterous Manipulation." Conference on Robot Learning (CoRL) 2025. https://arxiv.org/abs/2505.21864 ↩
12. "Awards," Conference on Robot Learning 2025. https://2025.corl.org/program/awards ↩
13. Ha, H., Gao, Y., Fu, Z., Tan, J., and Song, S. "UMI on Legs: Making Manipulation Policies Mobile with Manipulation-Centric Whole-body Controllers." Conference on Robot Learning (CoRL) 2024. https://arxiv.org/abs/2407.10353 ↩
14. "UMI on Legs: Making Manipulation Policies Mobile with Manipulation-Centric Whole-body Controllers" (project website). https://umi-on-legs.github.io/ ↩
15. actuated-umi/actuated-umi-gripper (GitHub repository). https://github.com/actuated-umi/actuated-umi-gripper ↩
16. "HumanoidUMI: Bridging Robot-Free Demonstrations and Humanoid Whole-Body Manipulation." arXiv:2606.27239. https://arxiv.org/abs/2606.27239 ↩
17. "FastUMI-100K: Advancing Data-driven Robotic Manipulation with a Large-scale UMI-style Dataset." arXiv:2510.08022. https://arxiv.org/abs/2510.08022 ↩
18. Toyota Research Institute. "Large Behavior Models." https://www.tri.global/our-work/large-behavior-models ↩
19. "TRI: pretrained large behavior models accelerate robot learning," The Robot Report. https://www.therobotreport.com/tri-pretrained-large-behavior-models-accelerate-robot-learning/ ↩
20. "AI-Powered Robot by Boston Dynamics and Toyota Research Institute Takes a Key Step Towards General-Purpose Humanoids," Toyota USA Newsroom. https://pressroom.toyota.com/ai-powered-robot-by-boston-dynamics-and-toyota-research-institute-takes-a-key-step-towards-general-purpose-humanoids/ ↩
21. Physical Intelligence. "Pi 0: A Vision-Language-Action Flow Model for General Robot Control." https://www.pi.website/download/pi0.pdf ↩
22. "RDT2: Exploring the Scaling Limit of UMI Data Towards Zero-Shot Cross-Embodiment Generalization." arXiv:2602.03310. https://arxiv.org/abs/2602.03310 ↩
23. "VISTA: Vision-Grounded and Physics-Validated Adaptation of UMI data for VLA Training." arXiv:2606.04708. https://arxiv.org/abs/2606.04708 ↩