Head camera or handheld gripper: what data actually teaches a robot
·Julian Enciso·6 min de lectura
Watching versus doing
Robot training data comes in very different forms, and the main difference is how directly each one tells a robot what to do with its hands. A video of someone wiring a socket shows what happened. It does not say exactly where the fingers were, how hard they pressed or how the tool moved in three dimensions.
At Klavax we record electricians at work, so we think about this trade off every day. This post walks through the main data sources robotics teams use today, what first person video and handheld grippers each capture, and why we start with a stereo head camera before adding grippers.
The main data sources today
There is still no internet scale dataset that pairs what a robot senses with the action it should take, so teams combine several imperfect sources. The Prosus Ventures robotics report from August 2026 sorts them into four approaches. We split video in two, because first person footage recorded for robotics behaves very differently from what is already online:
| Source | What it is | Strength | Weakness |
|---|---|---|---|
| Teleoperation | A person steers a real robot remotely | Exact robot actions, high quality | Slow and expensive to scale, carries the operator's awkwardness |
| Simulation | Robots practice in a physics engine | Nearly unlimited repetitions | Real materials and contact still differ from the model |
| Internet video | Footage already online | Huge and cheap | Precise actions are hard to read from it |
| First person video | Cameras worn by people doing real work | Natural behavior, scales across many workers | Hand actions must be estimated, human hands differ from robot grippers |
| Handheld grippers and gloves | Tools people hold or wear that log motion and grip | Good balance of scale and action quality | More hardware, changes how people work |
Prosus expects the industry to converge on the last category, possibly with workers wearing capture devices as part of their normal jobs. The two human centred sources are the ones that matter most for the trades, so the rest of this post looks at them closely.
First person video: strengths and limits
First person video is the fastest way to capture a lot of real work. A camera on the head sees roughly what the worker sees, including the hands, the tool and the part, and the worker barely changes how they move.
The research case got much stronger this year. NVIDIA's EgoScale project pretrained a robot model on 20,854 hours of first person human video and found a clean log linear relationship between hours of human data and how well the model predicts hand actions, a measure that tracked real robot performance on dexterous tasks. Put simply, more hours of good human video reliably made the robot better.
The limits are real too. Video does not contain the action itself, so hand and wrist poses have to be estimated from the footage, which gets harder when hands disappear inside a distribution board or behind a cable bundle. Human hands also differ from robot grippers, which is why EgoScale follows the big pretraining run with a smaller round of aligned human and robot data. People who run these pipelines describe human first person data as a multiplier for robot data, not a replacement for it.
Depth helps with several of these problems. A single RGB camera has to guess how far away things are, while a stereo camera measures it. An inertial sensor adds how the head moves, which makes it easier to place the hands in three dimensions.
Handheld grippers and action space data
A handheld gripper records the action directly, in a form a robot can use. The best known design is the Universal Manipulation Interface (UMI), presented at RSS 2024 by researchers from Stanford, Columbia and the Toyota Research Institute. A person holds a gripper shaped like a robot's, with a wide angle camera mounted where a robot's wrist camera would sit. The camera's motion sensor and visual tracking recover the gripper's path in 3D, and the opening of the fingers is tracked continuously.
The result is data in action space: where the gripper went, how it turned and when it closed, seen from the robot's own point of view. The UMI team showed that policies trained this way could run on different robot arms and generalize to new places and objects.
The trade off is speed and naturalness. A person holding a gripper works more slowly than with bare hands, can only do what a two finger gripper can do, and needs a scene with enough visual texture for tracking to work. For an electrician that rules out much of a normal day, but it is excellent for focused sub tasks like inserting a conductor into a terminal or routing cable through clips.
Why we start with a head camera, then add grippers
We start with stereo head capture because it lets electricians work exactly as they normally do, with their own tools, on real sites, for whole tasks from start to finish. Nothing about the job changes except a camera on a head strap.
Our first rig is simple on purpose: a compact stereo camera with a built in inertial sensor, worn on a head strap and cabled to a small computer and battery carried in a bag. Every recording gets depth, head motion and synchronized timestamps along with the video, which goes beyond the plain RGB footage that most public first person datasets offer.
This gives buyers what video does best: long, natural sequences of real trade work across many sites and many people, the kind of data EgoScale showed scales well.
Grippers come next. Once a partner network is recording reliably, we add UMI style rigs for the sub tasks where precise action data matters most and where a gripper can do the job, such as terminating conductors or routing cable. Head video gives breadth and gripper rigs give precise actions, and we expect most robotics teams to want both.
What makes a recording useful to a buyer
Hours alone do not make a dataset valuable. HumanSignal argues that robot foundation models are short on four properties rather than raw volume: varied conditions, grounded language, coverage of failures, and sensing beyond RGB. Three of those are fixed at capture time and cannot be added once the rig comes down. That shapes how we record:
- Every stream is synchronized and calibrated, so video, depth and motion line up frame by frame.
- We record whole tasks with a clear start and end, not short clips cut from the middle.
- Each task is labeled in plain language, such as "install a double socket in a drywall box".
- We record across many sites, buildings, lighting conditions and electricians, because variety teaches more than repetition.
- Mistakes and corrections stay in the data, since recovering from a slip is a skill too.
- Every recording carries documented consent and clear usage rights, and personal details from the site are handled before delivery.
If you are building robots for real buildings and want to see sample footage from our first recordings, get in touch.