In summary:We tested four open-source 3D hand tracking models in Benchmark and self-collected head-mounted camera work videos: WiLoR, HaWoR, EgoForce, and Dyn-HaMR. They performed well on Benchmark, but they repeatedly showed the same kinds of errors in real-life work scenes: hand confusion in low light, motion blurring, missing or blank-filled hands, partially obscured hands leaking, feet and forearms being identified as hands, and deep estimates of inconsistencies with physical reality. These errors began with hand detection, which is why we are building our own detectors.
If you want robots to learn from humans, hands are data. A strategy based on first-person video training involves knowing where each hand's individual joints are, and which are left-handed and which are right-handed.
Four recent open-source models are natural starting points: WiLoR, HaWoR, EgoForce, and Dyn-HaMR. We first examined the performance of these four models on the well-known Benchmark and made no modifications to use them in our work videos.
Problem: Hand tracking that seems to have been solved fails in real-life work scenes
In the Benchmark video, hand tracking appears to have been resolved
Meta's HOT3D was recorded in an action-capture laboratory. Lightwheel's EgoStandard showed people assembling and operating objects at the table. Both were well lit, the field of view was wide, and the hands were mostly inside the screen.
These models performed well in such scenarios. For the detected hands, HaWoR correctly marked the left hand at 99.9% to 100%, with a median shape error of only 6 to 11 millimeters per hand. Dyn-HaMR's error on the EgoStandard was 4 to 8 millimeters, and the left hand was never confused in the five videos we ran.
In controlled video, the data performed very well. Hand-shaped error refers to predicting the distance between the joints and the actual joints after alignment, rotation and measurement. HaWoR and Dyn-HaMR use different EgoStandard video clips to evaluate, so their data cannot be directly compared.
Table 1. Data performed very well in controlled video. Hand-shaped error refers to the distance between the joints and the actual joints being predicted after alignment, rotation and measurement. HaWoR and Dyn-HaMR use different EgoStandard video clips for evaluation, so their data cannot be directly compared.
Figure 1. HaWoR's performance on HOT3D: Both hands were detected and correctly labelled in a bright, clean motion capture lab. Source: HOT3D(Meta), sequence P0001_a68492d5, CC BY-SA 4.0; fish eye images are twisted and rotated in the direction of the past, with model output superimposed on the image.
In real work videos, hand tracking fails
Our videos come from people who wear head-mounted binoculars and do their daily work: washing dishes in the kitchen, giving supplies to warehouses, removing fillers from construction sites, and cleaning the leaves with blowing blowers. No one moves slowly for the camera.
We use the same four models for these videos without re-training. Several of the same types of errors occur repeatedly, and each model has at least two of them.
Table 2. Problems with models in our videos. The percentage of models that detect hand-held filters as statistical objects. EgoForce and Dyn-HaMR do not run on warehouse video footage and WiLoR does not run on dishwasher video footage.
The following shows what each type of error is in our videos. We will explain why they happen in the following sections.
Weak Light: Left to Right
The most obvious error occurred in the warehouse: an employee reached out his hand into a darkened cardboard box. In the video footage, more than a quarter of the footage was darker than any of the two Benchmark video footage. In the darkest footage, HaWoR confused the left hand in 84% of the footage and WiLoR in 78%. WiLoR was still completely undetected in nearly 40% of the footage. In the bright footage in the same video footage, the two models almost never went wrong.
Sequence 1. HaWoR (left) and WiLoR (right) appear in the warehouse scene, the image is raised and slowly placed. The arms are never crossed, but the tags are exchanged.
Rapid movement and blurredness: missing limbs, or being filled in empty
In household kitchens, the wearer's right hand quickly scrubs the dishes under the towel, rapidly to blur. In 32 dishes during the scrubbing process, HaWoR detects the number of dishes in the hand at 4, EgoForce at 8, Dyn-HaMR at 6.
What happens in the rest of the pumps is different depending on the model. EgoForce won't fill in the missing pieces, and WiLoR won't, so they mostly miss out on the hand. HaWoR and Dyn-HaMR estimate one hand to fill in the missing pieces based on the front and back movements. It looks like a hand, and it moves like a hand, but here, it hovers over a microwave.
Sequence 2. At the same time as the dishwashing process, the image is slowly placed: the original image (top left), HaWoR, EgoForce and Dyn-HaMR.
The image is blurred when the wearer turns quickly. In a video clip of the dishwasher, HaWoR detects an average of about one hand per cuff during a quick turn; while the head is almost stationary, it detects an average of just over 1.5 hands per cuff. In another kitchen video, the left hand that HaWoR fills falls on the wearer's feet during the turn.
Sequence 3. HaWoR appears during a rapid turnaround, the image is slowly placed. The image becomes blurred, the left hand is lost, the left hand is replaced, and the left hand (L) falls on the wearer's feet.
Shadows and edges: showing only half a hand is equivalent to not having one
In real-life scenarios, the hand is often in a partially obscured state: reaching into the box, with a towel or plates stuck in it, or half out of the picture. When the hand performing the operation is obscured by a towel at the top right of the picture, the EgoForce does not output the hand, while the HaWoR can still detect it. In a video of the staff using the blowing machine, the empty left hand appears at the bottom edge of the screen, but the Dyn-HaMR does not output the hand in all 1,795 frames. When both hands are deep in the box, the WiLoR confuses the left hand.
Figure 2. Hands partially shaded. Above left: EgoForce leaks the right hand behind the towel. Above right: Dyn-HaMR leaks the left hand of the staff using the blowing machine at the bottom edge of the image. Below left: WiLoR confuses the two hands in the box and the image is illuminated.
Losing one hand is only part of the problem. When the washed hand moves to the right edge of the image, Dyn-HaMR loses the hand 128 times in a row while jumping the left hand tag to the right hand.
Sequence 4. When the washed hand moves to the right edge of the image, the Dyn-HaMR performance.
Scene of chaos: feet, forearms and ground turned into hands
Head-mounted cameras shoot the wearer's own body downwards. Feet, legs and forearms appear simultaneously with the hands, and from the top they may be very similar to the hands. Each model has at one point painted the hands on something that is not the hands: EgoForce and Dyn-HaMR painted the hands on the wearer's feet, WiLoR painted the hands on the forearms and bowls, and HaWoR painted the hands in a dark shape at the bottom edge of the image.
It is not possible to correctly estimate the depth
When the wearer looks down at the bowl rack, the EgoForce places their hands behind the camera for about 1.5 seconds. Throughout the entire bowl washing video footage, the output of the EgoForce is 12% of the depth that a real hand cannot reach. Even in video footage that correctly estimates the camera parameters, the Dyn-HaMR places one hand at a distance of 3.6 meters from the wearer's head, while the other hand is only half a meter from the head.
Sequence 5. When the wearer looks down at the bowl rack, the performance of EgoForce is slowly postponed. The joints of both hands are behind the camera, so the rack lines are scattered on the floor.
Why this is important: the wrong hand data teaches the wrong knowledge to the robot.
Based on a robotic strategy known as first-person video training, the robot learns progressively from each hand's position of the joints and from the position of the left hand. Each of the above errors translates into an incorrect learning signal:
The exchange of labels teaches the wrong arm.If the left hand is confused, the robot will learn to use the wrong arm to reach out. In dim warehouses, this is most often the case in 84% of models that detect the hands.
The addition of a hand is a seemingly certain guess.HaWoR and Dyn-HaMR use the look and movement of a real hand to fill in the missing pieces, so that even if the handles are placed above the microwave or on the wearer's feet, they can be considered as real data.
Drawing a hand on the wrong object means a grabbing that never happened.It is used to record movements that have never been made by a person.
The unlikely depth means the unlikely movement.A hand at a distance of 3.6 meters from the wearer's head, or a hand with a joint behind the camera, describes movements that no arm can complete.
These are not marginal situations. Real work can create a number of difficult conditions at the same time: warehouse video clips have dim, fast movement and scene clutter problems at the same time.
Worse, common quality checks miss most of these problems. The coordinate check compares each 3D hand to the 2D joints calculated by the hand itself, so the default passes; while the coverage statistics will add up the hand as the detected hand. Almost all of the errors in this article would be preserved if the data sets were only filtered according to these two indicators.
Reasons: The difficult conditions for missing Benchmark
The model itself is not bad, the difficulty lies in the actual conditions.
In our videos, all four models correctly track when their hands are still, light is good and there is no overshadowing. Therefore, the error comes from the actual conditions, not the camera failure or the person wearing the camera.
Figure 4. All four models can track correctly when both hands are still, full of light and in the field of view; the head-mounted cameras used are the same as those used to produce all of the above error videos.
These are exactly the conditions that Benchmark omits. In order to get accurate labeling, a set of data needs good lighting, stable cameras and clearly visible hands, and these are the best conditions for models to perform. Each type of error in our videos has its specific mechanisms.
Weakness of light: the traces of left-handedness disappear gradually
See: Sequence 1 and Table 2
These models judge the right side by the appearance of the single-handed hand: which side of the thumb is seen by the fingers or the palms of the hands. Under dim light, these clues disappear first. The image noise increases, the edges become blurry, and the camera's automatic exposure prolongs the opening time of the door, so any moving object produces a drag shadow. The detector can usually still detect the hand, but the judgment of the left hand becomes unreliable. HaWoR and Dyn-HaMR both use WiLoR's hand detector, so it inherits its judgement results.
Fast movement and ambiguity: Finger missing
In our videos, the model typically moves 1.5 to 4 times faster than normal before losing the hand. Models such as WiLoR and EgoForce that do not fill in missing segments will directly miss the hand. HaWoR and Dyn-HaMR These video models will also lose the hand, but will then fill in the missing segments: HaWoR uses the learned motion model to fill in up to 4 seconds at a time; Dyn-HaMR will smoothly connect the hand positions on both sides of the missing segments. The result will be smooth and determined, but they can only be guessed.
Shadows and edges: incomplete hands thrown away
See Figure 2 and Sequence 4.
Our camera has a narrower vertical angle of view than the Benchmark camera: our camera is EgoStandard for HOT3D. Therefore, when operating near the body's position, the hands are usually at the bottom or side of the screen and part of the screen is cut off. In addition to a towel, plate or box side shade, only a few fingers may remain visible. The detector learns mainly from the complete, visible part of the hand, so it will take down and discard these incomplete branch views. The EgoForce code goes further: if one key in the image is less than half located, it will be discarded.
Scene of confusion: No mechanism to check whether the hand belongs to the wearer
See: Figure 3
Benchmark scenes are clean, the hands are in the center of the field of view. Our scenes are full of skin-colored shapes, and none of these models check whether a handkerchief is attached to the arm, or whether it is within reach of the arm of the camera wearer.
Depth: No single lens can measure distance
See: Sequence 5
These models use only one lens in our two-eyed camera, and a single lens cannot measure depth. They infer depth based on the size of the hand seen and the knowledge of the hand learned from the data. When input conditions are more special, such as when the camera is directly below, half of the hand is out of the picture, or the hand is blurred into a piece, this inference may fail. Rapid motion makes the situation worse: when the hand is slowly moving, the depth of the hand is estimated to vary by about 0.4 cm between adjacent hands; when the hand is fast moving, the width changes by about three times.
Common roots
Most of these errors start with the same link: detecting the hand and judging whether it is left-handed or right-handed. HaWoR and Dyn-HaMR both use WiLoR hand detectors, so the first step of the error judgment is passed on to all models built on top of the detector.
Solution: Humyn-HART 21, a handheld tracker built for real work scenes
Humyn-HART 21 is known as Hand Analysis Recognition and Tracking.
If a hand has never been detected, or if it carries the wrong label, then a good 3D model can't fix this. This is why we built our own hand tracking system based on the conditions in this article:
It's been a long time coming.The distinction between left and right hand visual cues gradually disappear, and the labels are exchanged.
Movement blur:When the hands move quickly and the head rotates quickly, the existing detectors lose their limbs.
Hands on the edge or partly obscured:Existing detectors give low scores and throw them away.
The scene of the riots:The feet, forearms and skin shapes are mistaken for hands, and the hands of another person may be mistaken for the hands of the wearer.
How effective the solution is: what we know now and how to measure the future
Humyn-HART 21 is designed to continuously lock the wearer's own hands, even when other people's hands are in the field of view. We first tested it at a multi-person shared sorting site, with a colleague always within reach. Within six minutes, a baseline tracker switches the tracking identity to a colleague's hand 13 times.
The mistake of identity is not noise, but the mistake of teaching robots knowledge.
Two of the findings in this study show why this first step of modification can be beneficial:
When the hands are clearly visible, the model can work properly.In the flatter sections of our videos (Figure 4), all four models correctly track the hands; the head-mounted cameras used are the same as those used to produce all of the above-mentioned errors.
Multiple models are built on top of the same detector.HaWoR and Dyn-HaMR both inherited WiLoR's detectors and their judgement results, so improved detection can address these errors from the source without having to modify the model individually.
For now, it's just a conditional test. Next, we're going to measure the number of ID switches and the accuracy of key points that occur in more crowded workplaces. We're also going to use the same methodology to evaluate the four models in this article, including dim lighting, motion blurring, hand positioning on the edge of the screen and scenes of confusion, evaluating HART 21, and sharing its performance under the conditions covered in this article.
Testing methods: We ran the released versions of WiLoR, HaWoR, EgoForce and Dyn-HaMR without re-training. Benchmark scores used reference labels for each dataset; our video clips were taken for about a minute using a head-mounted ZED binocular camera.
