References
Today, we're releasing Perceptron Egocentric 1.0. It's a self-help API that converts first-person view video from humans and robots into structured data needed for robot training. It provides a front-end submission timeline, continuous activity tags for each hand, and state-of-the-art 3D wrist reconstruction that charges $29 per hour for video, less than half the cost of a submission timeline for the GPT-6 Astra, which generates about $76.
Homework
Homework: Washing dishes
Homework: Gloves to wash dishes
Homework: Washing dishes
Mechanical arms
Mechanical arms: Robot selection
Mechanical arms: robots organized
Manufacturing
Manufacturing: assembly of circuit boards
Manufacturing: containers
Manufacturing: Connector assembly
Washing dishes, hand-to-hand operation (photographed, unmarked): washing the bottom of the dish while the other hand operates the hose with a sponge.
Submission
Wash the blue sponge under the tap water.
Left hand
Hold the handle of the bottom of the tank so that it stays stable in the tank.
| Fields | Signed |
|---|---|
| Action | Hold your hand |
| Objects | Flat-sided handles |
| Contacts | Yes, it is. |
| Visibility | See you. |
Right hand
Wash the blue sponge under the tap water.
| Fields | Signed |
|---|---|
| Action | Cleaning |
| Objects | The Blue Sponge |
| Contacts | Yes, it is. |
| Visibility | See you. |
1 times the speed.
The hand:
2D Key Points
3D Grids
Switch between 3D wristbands and 2D key points.
Explore Perceptron Egocentric's output on home, robotic and manufacturing video: select a clip and play it, and click any step to jump to the right moment, or switch between 2D key points and 3D handles. The robot clip comes from the Galaxea Open-World dataset; the manufacturing clip comes from Build AI's Egocentric-10K.
Despite recent progress in understanding first-person video, large-scale, high-quality markup is still difficult. Perceptron Egocentric addresses three shortcomings:
Providing submission labels at low cost.Front-end models like the GPT-6 Astra can generate high-quality submission labels, but the cost of running on large datasets is too high. The standard benchmark cannot evaluate segments of less than a second. At EgoHome, we have a score gap of less than 0.01 with the GPT-6 Astra, costing only one-eighth of that, at $9 and $76 per hour of video respectively.
Structured labels are provided by hand.Few pipelines are able to simultaneously provide continuous activity labels for each hand, a consistent movement classification system, precise 2D key points and 3D hand reconstruction. We provide all of the above features, including a new 3D hand reconstruction model with hand shaping accuracy at the most advanced level in HOT3D and ARCTIC.
It's a simple, self-help tool.Open source tools will leave the computing, expansion, and deployment to your engineers, while hosting service providers typically require customized collaboration. Through our API, you can submit videos, track tasks, and download structured labels, with price transparency, and open without manual assistance.
These capabilities together constitute a major upgrade from our earlier version of the experience in July. Behind the API, our customized first point of view markup framework integrates specialized models for subtasking, single-handed activity markup, and 2D/3D handheld tracking to deliver high-quality markup on a large scale in an economically efficient manner.
Low cost labeling submission
In first-person video, it may take less than a second to grab, release, or transpose an animal. Full-second time-resolution labels mix these movements together, making it more difficult for robots to learn the timing of each step. Perceptron Egocentric divides each video into time-lapse subtasks, including these short-term movements.
At the cost of one-eighth, we provide timelines at the forefront level.
Under the default 4fps setting, the Perceptron Egocentric scored 0.787 on the EgoHome. EgoHome is our newly launched sub-second homework step benchmark, which will be explained below. The score is less than 0.010 compared to the GPT-6 Astra, which uses the same 4fps setting.
Figure one:EgoHome marks the ratio of quality to the cost of video per hour, with the cost taking into account the number of frames. The null line represents the Pareto front, meaning that there are no other systems that can outperform systems at the same time in scoring and cost; the orange region above the left represents lower costs and higher scores, and neither of our measurement systems has entered the region. Both Perceptron Egocentric settings are on the front.
On the public WGO-Bench benchmark, the Perceptron Egocentric achieved the highest segment F1 ((0.583) and the second highest semantic F1 ((0.474).
Table one:WGO-Bench, 100 video, the higher the numerical value. The semantic F1 will be correct when both the time positioning and the labels are matched only in the steps; the F1 segment only checks the time positioning. Both use aggregate computation: first statistics the matches in all 100 videos, then calculates F1, rather than averaging the F1 for each video, in order to be consistent with the way the other teams report WGO-Bench results.
†Reported by the relevant team itself, we have not returned.
‡ FLORA uses its own evaluation protocol to report the results of the WGO-Bench, so it cannot be directly compared: we omitted its semantics F1, showing only the fragment F1 of its report.
Capture steps in less than a second
The WGO-Bench does not have a reference clip of less than a second, so it is impossible to judge whether it captures short-term action. To measure this, we constructed EgoHome, a new benchmark that contains 137 pieces of artificially verified footage covering about two hours of first-person home video. Of these, 24 contain at least one step of less than half a second.
Figure two:A sub-task timeline of the overlay video in the EgoHome is played at twice the speed: the reference tag ((real value) ), the default Perceptron Egocentric, and the 4 ton GPT-6 Astra per second. Both the real value and the Perceptron Egocentric divide the segment into 19 steps; the GPT-6 Astra returns eight steps, marking the 30-second overlay towel process as one step.
We also proposed a new indicator, called the Jaccard indicator, which allows sub-sections to be matched ((permissible subsegment Jaccard)), to be used to consider the difference in the grain size of the labeling. The same 45-second process can be labeled as a dishwasher, or can be broken down into a brushwasher, washer, washer, and scrub dryer; both labels are effective. The indicator will not punish either of them, but will use an LLM evaluator to determine whether the roughness and fineness descriptions cover the same activity and then measure the consistency of the forecast timeline with our reference timeline. EgoHome, along with the details and code for the indicator, will be released in the coming days.
Table two:The higher the subtasks score, the better the video time ratio of the predicted steps to the reference steps, measured using the permissible subsegment Jaccard; the overall results are shown in the table, as well as the results divided by the reference steps. The lower the cost per hour of video.
All systems use the same scorer, scoring on all 137 clips. Perceptron's results take three runs at 4 fps and two runs at 8 fps; all other systems run. GPT-6 Astra receives a time-bar image box; Gemini and Glimpse receives native video. Argus's tags describe the state of objects, such as the Mark Cup being placed in the upper bowl shelf, rather than moving, and the evaluator of EgoHome judges these labels as not matching.
The cost for each system is the cost of an hour of video-generation timeline: Perceptron uses customer-oriented pricing, GPT-6 Astra, Gemini and Glimpse uses API fees, FLORA uses GPUs. The cost of Argus also includes wrist tags. The cost of ≈ is the estimated value. GPT-6 Astra, Gemini and Glimpse combined all the tasks, so we counted only the corresponding portion of the timeline: the cost of video as a single input and processing plus the cost of timeline output.
For steps less than half a second, our GPT-6 Astra (0.595) and FLORA (0.579) scores at 1 fps are slightly higher than ours (0.559); for steps from half a second to a second, our 8 fps are set to a maximum score of 0.672.
3D tracking and activity marking of each hand
Perceptron Egocentric processes each hand individually: 3D reconstruction of the hands and timing of each hand's activity.
A 3D reconstruction of the opponent's arm can be used to convert human samples into monitoring signals for robotic operations. Recent work, such as EgoMimic, has shown that using a first-person video co-training operation strategy with 3D arm tracking, can improve the performance of long-range single-arms and arm missions. VideoDex extracts data from Internet videos and redirects estimated 3D arm movements to robotic agility as a precursor to strategy learning.
The most advanced hand shapes are reconstructed without labeling
Perceptron Egocentric rebuilds the 3D arm directly from the first-person view video. Our model is based on the Mk1.5 and estimates the 3D shape, finger joint structure, and position of each hand relative to the camera. In all four settings, its arm-shaped rebuilding accuracy is the strongest available method in Table 3, with ACE-Ego-Hand rising to 12% and 25% respectively, and no camera markings required: whether or not it provides labeling information, the upper arm shape error in HOT3D is 5.21 mm.
Figure three:Two segments were filmed using a mobile phone; the first view of the bowl washing video was not marked by the camera: a) hands; b) wearing gloves. Each hand has a separate 3D grille and marks the left hand and the left hand, detecting confidence and its distance from the camera. Each line is a 2D key point on the left side; the right side is a 3D grille on the video screen, and the same two hands as observed from the other side.
Table three:The hand shape and gesture accuracy are measured on HOT3D and ARCTIC. Both of these standard benchmarks contain first-person videos showing the hand-operated object in the hand. The lower the error, the better in millimeters. The hand shape indicator PA-MPJPE is aligned with the plane, rotation, and scale to measure the finger joint structural error separately. The hand gesture indicator, or wrist alignment MPJPE, is only against the position of the wrist, and therefore punishes the overall orientation and scale error.
† Data from Table 1 of the ACE-Ego-Hand paper is not reproduced. The paper only reports HaMeR, WiLoR and ViDiHand's hand shape results in the labelled settings. Its list of fragments has not been made public, so the fragments we used are different.
Each hand has a separate event timeline.
Each hand also receives its own activity tag: each hand has a separate timeline that briefly describes the activity of the hand at each step. The robotic claw also uses the same method, with each claw having a timeline. In Figure 4, the left hand holds the bottom of the claw and leans it down to the water, the right hand opens the plunger and shoots the sponge, and the two hands receive their own tag. For arm robots, the left claw moves a cup and a claw bottle into a basket, and the right claw moves a peach into another basket.
Figure four:Perceptron Egocentric generated single-handed activity tags: ((a) Hand washing bowl clips in Figure 3 to play at half-speed; ((b) Two-armed Galaxea robots sort objects into clips in the basket to play at normal speed. Each hand or claw has its own timeline; orange steps indicate the steps that are currently taking place, which are labeled above the timeline.
Starting to mark right away
The Perceptron Egocentric openbox provides subtasking, single-handed activity tags, and 2D/3D hand tracking. It operates as an asynchronous batch processing API. You can send up to 1,000 video links in one request, select the output you want, then query each task, and download the results after completion. Each task contains subtasking segments; 2D or 3D hand tracking and single-handed activity tags are added to the batch selection.
A completed batch of Perceptron controllers: 240 tasks, 172.8 billing hours of video, detailed charges divided by output type, and a list of tasks.
Figure five:A batch in the Perceptron console that displays labeling fees broken down by output type, including submission timelines, 3D wrist tracking and activity labels, and status and fees for each operation.
The API provides seven endpoints:
Submit the Batch：
POST /v1/egocentric/batchesA batch sends a maximum of 1,000 video links in a single request, and includes buttons such as a batch; the entire batch will be accepted or rejected as a whole.List of Batches：
GET /v1/egocentric/batchesClick on the page to browse the batch you are organizing.I've been waiting for you to come back.：
GET /v1/egocentric/batches/{batch_id}I'm trying to get the status and timing of the batch.List of Batch Jobs: List of Batch Jobs：
GET /v1/egocentric/batches/{batch_id}/jobsIn order of entry, the batch of work is listed.List of jobs: List of jobs：
GET /v1/egocentric/jobsYou can create a list of all the activities that you are organizing in the order in which they are created.Retrieve a job and get a job.：
GET /v1/egocentric/jobs/{job_id}It is used to obtain state of operation, input metadata, error and execution time information.Retrieve the results of the job.：
GET /v1/egocentric/jobs/{job_id}/resultFollowing the success of the JSON operation, the results were downloaded to the JSON format.
For information on identity verification, restrictions and error handling, please refer to the first point of view function guide.
UsedPOST /v1/egocentric/batchesSubmit the video:
curl \
-H "Authorization: Bearer $PERCEPTRON_API_KEY" \
-H "Content-Type: application/json" \
-H "Idempotency-Key: $(uuidgen)" \
-d '{
"inputs": [
{
"type": "video_url",
"video_url": { "url": " }
}
],
"hand_tracking": { "output": "3d" },
"hand_activity": true
}'
UsedGET /v1/egocentric/jobs/{job_id}Check the status of work:
curl \
-H "Authorization: Bearer $PERCEPTRON_API_KEY"
UsedGET /v1/egocentric/jobs/{job_id}/resultDownload the results:
curl \
-H "Authorization: Bearer $PERCEPTRON_API_KEY" \
-o result.json
Output:Always include submission timelines. 2D or 3D wrist tracking and single-handed activity labels as optional add-ons, batch-selected.
According to the type of output, it is billed per hour of video:
Egocentric standard hourly video prices for various types of output
Assessment notes
The gesture of the hand:We followed the ACE-Ego-Hand evaluation protocol, using 244 segments on HOT3D participants' P0001 data and 302 segments on ARCTIC subjects' s05 data. All models and evaluation settings were determined on independent verification data, followed by only one test evaluation.
EgoHome：We used the Gemini 3.5 Flash as a judge. All systems in Table 2 were scored in the same round of evaluation and cached the results of each pair of labels to ensure consistency. Perceptron's settings ran two or three times; the competing system ran once. The models evaluated on EgoHome did not include the EgoHome snippet in their training data.
WGO-Bench：We follow the guideline's open evaluation rules. After aligning the external boundary with the reference boundary, the predicted steps are considered matched if the timing of the predicted steps and the reference steps is met. The semantic F1 is also required by the Gemini 3.5 Flash judge to verify whether the predicted tags match the mission instructions.
The results of the self-report:Scores marked as self-reported come from outside sources and are not independently reproduced.
References
@article{perceptron2026egocentric,
author = {PerceptronAI},
title = {Introducing Perceptron Egocentric 1.0},
journal = {PerceptronAI Research Blog},
year = {2026},
note = {
}
References
T. Jiang, T. Yuan, Y. Liu and others. The Galaxy Open-World Dataset and G0 Dual-System VLA Model is expected in 2025.
Build AI―Egocentric-10K―Hugging Face data set―huggingface.co/datasets/builddotai/Egocentric-10K―
Macrodata Labs ‒ Segmenting Robot Video into Actionable Subtasks (WGO-Bench) ‒ June 2026 ‒ macrodata.co/blog/annotating-robot-video-subtasks
P. Banerjee, S. Shkodrani, P. Moulon and others. HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videos.
Z. Fan, O. Taheri, D. Tzionas and others. ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation.
Midcentury.xyz/blog/optimizing-argus.
FLORA: An Open Model Suite for Scaling Robotics Data Engines, published by S. Leng, Y. Man, Z. Li and others.
Pantheon Industries Argus: an open source harness for annotating robot learning data GitHub warehouse Github.com/Pantheon-Industries-Inc/argus
S. Kareer, D. Patel, R. Punamiya and others. EgoMimic: Scaling Imitation Learning through Egocentric Video.
K. Shaw, S. Bahl, D. Pathak。VideoDex: Learning Dexterity from Internet Videos。CoRL 2022。arXiv:2212.04498。
Y. Liu, X. Wang, H. Li and others. ACE-Ego-Hand: Repurposing Video Diffusion Models for Occlusion-Robust Egocentric 3D Hand Motion Recovery.
G. Pavlakos, D. Shan, I. Radosavovic and others. Reconstructing Hands in 3D with Transformers (HaMeR) CVPR 2024 arXiv:2312.05251
R. A. Potamias, J. Zhang, J. Deng and others. WiLoR: End-to-end 3D Hand Localization and Reconstruction in the wild.
Y. Wang, C. Jin, Y. Liu and others. The Surprising Effectiveness of Video Diffusion Models for Hand Motion Reconstruction (ViDiHand) in 2026.
