Runway is a New York-based generation AI company best known for its Gen series of video generation models. The company was founded in 2018 by Cristóbal Valenzuela, Anastasis Germanidis and Alejandro Matamala Ortiz.
In the past few years, Runway has continued to attract capital. In April 2025, the company completed more than $300 million in D-Round financing; in February 2026, it announced the completion of another $315 million in E-Round financing, with investments from General Atlantic, NVIDIA, Adobe Ventures, AMD Ventures, and others.
While Runway initially received widespread attention for its video generation tools, the company's research goal has long been beyond video generation. Runway uses General World Models as a long-term research direction, hoping to train in understanding and simulating models of the visual world and its dynamic changes. For robots, this means that video models can not only generate images, but can also be used to predict the consequences of actions, evaluate strategies, and even directly generate robot actions.
Prior to Praxis-1, Runway had begun to explore a combination of world models and robotic learning. In late 2025, Runway launched GWM-1 and GWM-Robotics for robotic applications; in February 2026, the team further demonstrated research to evaluate robotic strategies in world models and collaborated with NVIDIA, Berkshire Grey to explore related applications.
Praxis-1 has taken this route one step further:From using world models to predict and evaluate robot behaviour, to building robot control strategies directly using video pre-training.
Praxis-1 is based on Runway's 27 billion-parameter video base model, first through mid-range training with body video, and then with the introduction of Action Expert, which predicts joint learning movements with video. The model predicts a small portion of future video and corresponding movements at the same time, and can be adapted to different entities such as single-arms, two-arms and mobile operating robots.
In the NVIDIA RoboLab-120 simulation review, Praxis-1 achieved a 46.2% success rate in 1,200 tests, ranking first in the list of world action models and open source robot base models at the time of publication. In 10 missions in the real Franka mechanical arm, Praxis-1 completed 88 out of 100 tests. Runway has released open model weights under the Apache 2.0 license and provides code and fine tuning tools.
1. Quotations
Robot deployments repeatedly face the same problem: real-world robot data is scarce, and collecting enough data to train general strategies is costly. Some laboratories choose to add remote data, while others choose to add first-person video. We believe that the most extensible data source is general video, which is video that people have already shot in their daily lives. Models trained in video prediction can learn how objects move, how hands move, and what tasks look like in the middle. Robots can learn further based on this knowledge.
With Praxis-1, we transformed a 27 billion-parameter video base model into an open-weight world model to predict the future in short clips. A small Action Expert reads the internal state of the video model as it imagines what will happen next. Each time the robot performs a move, Praxis-1 simultaneously predicts how the scene will appear and the movement required to make the scene reach that state.
Modeled after DROID, the Praxis-1 is the highest ranked world action model on the RoboLab-120 ranking and the highest ranked open-source robot base model. On the real Franka mechanical arm, it achieved a competitive success rate compared to the most advanced world action models before it.
Praxis-1 can accommodate a wide variety of robot bodies, from single-armed, two-armed robots to mobile operating robots. We show a variety of everyday tasks such as turning the Mark Cup on a tray, putting items in boxes and covering them, putting bottles in boxes or closing them. The model can also accommodate robot bodies that have never been seen in training data. Google's RT-1 and SO-101 details this below.
2. Model architecture
Praxis-1 consists of three parts. The video mainstream network predicts what the next scene will look like, and Action Expert reads the characteristics of the predicted video to infer the movement needed to produce the scene. Each robot is equipped with a lightweight projection layer, which converts its positions and actions into a shared format, so that a model can control different robot entities.
At each call, the model receives an instruction, the latest camera image and the current position of the robot. The model returns a future video clip, and 1.6 seconds of action: 24 actions with 15 Hz output. The robot performs the action clip, returns the image observed by the camera, and then calls the model again.
Figure 1: Backbone and Action Expert
Both video and action are predicted in block-by-block sequences: each action block contains 24 actions, corresponding to 1.6 seconds below 15 Hz. Each robot has its own lightweight input and output projection layers; the main network and Action Expert are shared between the different robots.
2.1 Video backbone
Praxis-1 builds a video base model based on Runway: it is a distributed Transformer, broken down into conditional encoders and speed decoders, i.e. the DDT structure proposed by Wang et al. in 2025. The model runs in frozen Wan2.2 VAE sub-space and reads instructions through frozen Qwen3.5-2B text encoders. 17 of its 20 front-end layers use a hybrid expert model (MoE); only the first two blocks of the encoders and the first block of the decoders remain crystalline.
Each MoE layer contains an expert who shares all of the tokens, as well as 128 smaller routing experts, from which each token selects eight. A token is passed by nine experts whose width is equivalent to the width of a dense layer. Thus, although the layer contains the parameters of 129 experts, the calculation of each token remains roughly the same as dense blocks.
Experts in load balancing use the DeepSeek-V3 method: using routing deviations that do not depend on auxiliary loss, while adding a smaller sequence-level auxiliary loss.
The foundation model was originally trained to generate a single complete video clip in which each block could see all the other bits, including the future bits. The robots could not see the future, so during mid-term training, we changed the attention mechanism to a block-level causal form: the model progressed along time by 1.6 seconds each, containing six potential bits, with each block only seeing the previous block.
Each block starts to make noise from its own noise level, while focusing on its own clean copy of the previous block. When training, the front block comes from the real video, not the prediction of the model itself. This setting is called teacher forcing, which allows us to train four blocks simultaneously through parallel forward computation.
When reasoning, the model is processed block by block according to the same rules. By default, it reads a real camera image of the previous block from the robot, so you can see the context on the same type of clean as when training.
2.2 Action Expert
Action Expert is a compact 16-layer Transformer with a concealed layer width of 1024, containing about 314M parameters. Its design concept is closest to DeVA: DeVA also uses independent Action Expert to read video models at different depths.
Each layer of Action Expert switches between performing cross-attention on the mainstream network and performing self-attention. Each cross-attention layer reads one block of the mainstream network encoder, collectively reading eight blocks distributed from the shallow layer to the deep layer.
For each block, Action Expert includes:
One absolute position token;
8 learn-through future tokens;
Six action tokens, each action token corresponds to four actions.
In the first block of queries, you can see the clean mainstream network noise before the first block, as well as the noise itself in the first block of the mainstream network. Therefore, Action Expert reads the future of the mainstream network.
The mainstream network does not read the information from the Action Expert, but the loss of action still trains the mainstream network through the cross-attention reading process.
We have also tried to put action tokens inside the mainstream network using a mixture-of-transformers structure. In our offline agent evaluation, we performed a short-term arm minimization, with no apparent difference to the independent Action Expert based on the action error rating. The error is only one centimeter, but the calculation costs are higher.
Figure 2: What Action Expert is interested in
The chart shows the relationship between the mainstream network encoder, Action Expert's cross-attention and Action Expert's self-attention.
Universal tokens, including registers, tokens and commands.
It's a lot of noise.
A clean copy of each piece.
Action Expert token for each block.
The horizontal axis is the key to be read and the vertical axis is the query. The high brightness element represents the content of the corresponding query.
2.3 Input and sampling
| Projects | Set up |
|---|---|
| Cameras | The images of the outer camera and the wrist camera, respectively, are scaled up into linearly folded canvases, with a torque of 15 fps. |
| Time window | The training uses 93 tonnes at 15 fps: including one tonne, one shorter head and three complete pieces. |
| Moving on | DROID Franka) is a 10-dimensional: 6-dimensional rotation and claw; two-arms YAM are 20-dimensional; the human hands in the first view video use a 21-joint three-dimensional position. Position and rotation are represented relative to the position at the start of the block; position and claw use the 1st to 99th percentile number boundary to integrate and cut. |
| Moving blocks | Line 24, corresponding to 1.6 seconds below 15 Hz; the first call from the above context window returns to line 21. |
| The state of the robot | The absolute position at the start of the movement block is standardized by z-score; 10% of the sample is discarded during training. |
| Exclusive components for robots | Action input, pitch input and action output projection layers. Shared between mainstream networks and Action Expert robots |
| Training Goals | Both the video stream and the motion stream use Rectified Flow. Each block corresponds to a video noise level, with sampling biased towards higher noise; each block corresponds to a motion noise level, with independent homogeneous sampling. |
| Samplers | A two-step Euler sampling of video and action, using a single block of context, guides only the text of the video application. |
| Model size | A total of 27.2B parameters; 54.4 GB when using bf16; run on an 80 GB GPU |
3. Training Stage
Praxis-1 first learns how the world works and then learns how to act. The training is divided into four stages, each starting with the weight of the previous stage, with video prediction always being part of the training goal. After the introduction of the movement, the model learns the movement while learning the video prediction, rather than replacing the video prediction with the motion prediction.
3.1 Video pre-training
The backbone is based on Runway's video-based model. It is a unified model that is pre-trained in 12 generating tasks: generating images, videos, audio, or audio-containing videos based on text; and generating the same type of content, such as generating videos based on images and text, with the additional requirement of up to three reference images, video clips, or sound in addition to text.
The model was trained on general video data, using full sequence attention, and trained video clips of up to 20 seconds.
3.2 Physical video mid-training
The mainstream network switches to block-level causal attention and is trained on 15 fps body video, which records the process of human hands and robots performing various activities.
The data is primarily human first-person video, covering activities such as cooking, cleaning, assembly, packing, and includes robot video, general video, and a handful of game scenes.
The training sequence lasts up to 20 seconds and uses teacher forcing: the model predicts the current block based on a clean copy of each block in front.
Figure 3: Medium term training data
For the initial checkpoints used in this release, the chart shows the proportion of medium-term training tokens divided by video type.
3.3 Video and action combined mid-term training
Training components: mainstream network + Action Expert
Access an Action Expert that starts with a new weight and adds action targets to the robot's data window. The data includes single-arms, two-arms and low-cost two-arms, each of which is accessed through its own projection layer.
In order to protect the video model, in the initial 1,500 parameter updates, Action Expert's learning rate multiplier and gradient scaling coefficients from the loss of action to the mainstream network were gradually increased from 0 to 1.
Figure 4: An Action Expert, a switchable robot exclusive output head
Each robot body uses its own input layers (movement, position) and output layers and switches according to the body.
The mainstream network and Action Expert are shared between different robots; each robot only needs to add its own lightweight input and output layers. Figure shows some examples of mid-term training mixed data.
The loss function is the weight equivalent of video speed loss and motion loss. Video and motion noise levels are independently sampled, so motion noise levels are not restricted by the degree of video cleanliness.
Muon is responsible for optimizing most weight matrices; AdamW is responsible for optimizing embedding, output heads, input and output projection layers and routers.
At this stage, we still retain about 30% of the pure video tasks that continue from the second stage, so that the video models can continue to train on video data that has no action targets.
3.4 Specific body subtlety
Training components: Backbone + Action Expert + In-body exclusive output head
The final stage is to fine-tune the entire mid-term training model for target robots. Other fine-tuning experiments in this article, namely the two-arm YAM and RT-1, follow the same training configuration.
The backbone and Action Expert originally loaded the weight at the end of the mid-term training, with only the input and output projection layers exclusive to the robot beginning training from scratch. Except for the frozen encoders, the rest participated in the training while retaining video loss.
The table lists the specific settings of the examples used in this article: In DROID, the single-arm Franka is minimized. DROID is a public real-world Franka robot demonstration dataset.
Scaling of mid-term training
We expect increased mid-term training computing to provide a stronger base model for micromanaging. We have verified this with two robots: Franka appears in the DROID mid-term training data mix; RT-1 is a completely new robot entity that has never appeared in any pre-training or mid-term training stages.
Each data point is a checkpoint in the same mid-term training run, and the horizontal coordinates represent the total amount of computation required to reach the checkpoint, including video pre-training. Since later checkpoints use the same mid-term training data repeatedly, it is measured by increasing the effectiveness of the training calculations, rather than increasing the effectiveness of the data.
In experiments with two robot cores, more mid-range training did yield better mitigation results. We selected five checkpoints from video and action combined mid-range training, using the same training program to mitigate on DROID. On RoboLab-120, all data points used the same sampling setup, and the success rate of the first four checkpoints increased to 25.5%, 35.2%, 39.2% and 41.2% respectively; the fifth checkpoint was stable at 41.2%.[3]
RT-1 is not in the mid-term training data, so it allows us to test the same problem on robots that haven't been seen before. We use the same training configuration to fine-tune five mid-term training checkpoints on RT-1.
While there has been a trend of improvement in the performance of closed-loop bodies, both of which we have seen in training and which we have not seen in training, the exact relationship between the two remains to be studied further.
Figure 5: Relation of the mitigation effect to the total training calculation
RoboLab-120 Success Rate. DROID is included in the mid-term training data. Each data point corresponds to a mid-term training checkpoint, a model that is modeled on the DROID. Dual-pair coordinates are used in the diagram.
Figure 6: Relation of the fine-tuning effect to the total training calculation (RT-1 ((unseen kernel))
The SimplerEnv success rate, using Google's robotic visual matching setup; contains four tasks, each data point evaluates 864 episodes, each model minimizes 30,000 steps. The error bar represents 95% of the range.
5. Reasoning
Our model predicts the movements block by block: the robot performs the complete block of movements, returns the image observed, and then calls the model again using the same WebSocket.
Each tactical call returns 24 actions at 15 Hz, or 1.6 seconds of motion. The action, along with the imagined video, is de-noise through a two-step combined Euler sampling. The first call in the above context window returns only 21 lines, as the first of the six potential calls only covers the current call, not the 4 calls.
On single-chip H100 servers, with the default block execution mode, a call takes about 0.55 to 0.6 seconds back and forth, about one-third of the return 1.6 seconds. By default, we run the video in a sequential manner without classifier guidance (classifier-free guidance, CFG).
Figure 7: How Praxis-1 predicts the next block of action
Each call reads a clean upstream block (blue) while predicting the next 1.6 seconds of video and action (pink). The robot performs the action block, and the image observed by the camera during the execution becomes the background of the next call.
We also explored real-time chunking (RTC): robots do not have to wait for the current action block to be executed, but instead request the next action block to be executed halfway through, and continue the movement.
The difficulty lies in the welding of the two motion blocks. The direct mixing of the new and old plans would cause the mechanical arm to oscillate, so the new plans were changed to use the model's own motion prediction to continue to execute from where the mechanical arm was actually going to reach.
At our two-arm workstation, this reduces the proportion of time spent waiting at a standstill by about five times: from 32% to 5% in an experiment that took 0.8 tonnes of 0.9 seconds per call; from 16% to 3% with faster servers; about 0.3 seconds per call.
In the task of grabbing five objects and putting them in a bag, the RTC achieved a success rate close to that of the block execution: RTC was 24/30, block execution was 28/30; at the same time, it was faster, and the average time taken for successful tasks was 9.1 seconds and 15.2 seconds respectively.
In our released code warehouses, RTC needs to be activated; the default setting is block execution.
6. Results of experiments
All of the results in this section come from the same Praxis-1 checkpoint, which is modeled on DROID in accordance with the configuration described in the pump training phase. We evaluated it in the RoboLab-120 simulation environment while evaluating it on a real Franka FR3 mechanical arm operated by Positronic.
6.1 RoboLab-120
At the time of this post's release, NVIDIA's RoboLab-120 rankings include:
The first model of action in the world;
The first foundation model of open-source robots.
The RoboLab-120 is a simulation benchmark launched by NVIDIA, containing 120 DROID-style machine-armed desktop tasks. The official evaluation protocol for each task consists of 10 trials, with Praxis-1 finishing 555 out of 1,200 episodes on DROID, with a 46.2% success rate, ranking first among all world action models and open-source robot base models on the list. Followed by FLUX 3 Action, with a 42.9% success rate. Praxis-1 also outperformed all open-source VLAs, including π0.5 and GR00T N1.6.
Figure 8: RoboLab-120 success rate compares world action model with open weight VLA
The world action model is shown in dark and the open-weight VLA is shown in gray; 10 trials are performed on each mission.
The following shows the successful episodes, with each mission showing one.
There are three bananas in the box.The mission scored 10/10. External view and wrist view, 1 times the speed.
Put yellow blocks on red blocksThe task was completed with a score of 10/10. External angle and wrist angle, 1x faster.
Adjusting the direction of the White Mark CupNote: Make sure all the white-marked cups are in position and the mouth of the cup is facing upwards. Note: The mission score is 10/10.
Put the devil behind the bowl.The task was completed with a score of 10/10. External view and wrist view, 1x faster.
Putting bricks in boxesThe result of the task was 10/10. External view and wrist view, 1x faster.
Let go of the devil, then the champagne.The task was completed with a score of 9/10. External angle and wrist angle, 1x faster.
Putting smaller objects in boxesThe task was completed with a score of 10/10. External view and wrist view, 1x faster.
Putting green frogs in platesThe result of the mission was 10/10. External view and wrist view, 1x faster.
Put three fruits in a bowlThe result of the task was 10/10. External view and wrist view, 1x faster.
Put the pen in the Mark CupThe task was completed with a score of 7/10. External view and wrist view, 1x faster.
Unravel the stack of witchesThe result of the task was 10/10. External view and wrist view, 1x faster.
Wrapping up a white Mark CupThe mission score was 9/10. External view and wrist view, 1x faster.
Extraction of uraniumThe task was completed with a score of 10/10. External view and wrist view, 1x faster.
Putting spices in boxesThe task was completed with a score of 8/10. External view and wrist view, 1x faster.
Put the pink one in the pot.The result of the task was 7/10. External view and wrist view, 1x faster.
Take the keyboard out of the boxNote: Take the keyboard out of the box and put it on the table. The task scored 7/10. External view and wrist view, 1x faster.
More demos can be found in the original
In RoboLab-120, there were four competing models that scored higher than Praxis-1. The first two were Agent, which directly controls the robot using the front-end model, rather than a trained strategy: AMP1 had a success rate of 86.0%, classified as a zero-shot front-end model Agent; SenseNova-RoboRSI had a success rate of 57.1%, using the front-end VLM as a strategy.
SenseNova-RoboRSI is much slower than Praxis-1. Its authors report that on a similar desktop benchmark, RoboDojo, it takes about 55 minutes to compute each episode for an 8×76-second instructional movement, and finds this execution time difficult for actual deployment.
The other two models, Lakeside (51.0%) and BiMind v0.5 (49.2%) are closed-source models that partially use simulation data training, with BiMind v0.5 also modeled on RoboLab data.
6.2 Real robots
On the Franka FR3 workstation operated by Positronic, we evaluated Praxis-1 using 10 single-item operations missions released by FLUX 3 Action. Each mission ran for 10 episodes, divided into two rounds of five rounds each, running on the same device on November 2.
Figure 9: Direct contrast with real robots from FLUX 3 Action
On the same 10 missions, each model has 100 episodes of success.
Figure 10: Results by mission, 10 episodes per mission
Praxis-1 and FLUX 3 Action success rates on each mission.
The following shows the successful episodes, with each mission showing one.
True robots - Successful
Put the bananas on the left side plate
Mom put the bananas on the left side plate.
External point of view (left) and wrist point of view, 1 times faster.
True robots - Successful
Put the pen in the bowl.
She put her pencil in the bowl.
External point of view (left) and wrist point of view, 1 times faster.
True robots - Successful
Put the cup in the bowl.
She put the cup in the bowl.
External point of view (left) and wrist point of view, 1 times faster.
See more demos in the original
7. Real-time execution of daily tasks
It's one thing to be number one on the list, and it's another to run on real robots. Here, Praxis-1 controls two real robots performing daily tasks: two-armed YAM robots with two mechanical arms to complete tasks that can't be done independently with one arm, and the other is single-armed Franka. Each of the videos below shows the process of strategic autonomous control, with no remote operation, playing at 1x the speed.
YAM microchoice uses the same training configuration as DROID: Starting at the mid-motion training checkpoint, two-armed YAM data collected from three cameras is used for approximately 1,400 hours, 2,600 hours.
7.1 Robot with two arms (YAM)
The following data was collected from the two-armed YAM workstation in the Runway office to be played at real speed; each video is marked with corresponding instructions.
Mom picked up the purple tie on the right and put the purple tie in the bag.
The mother takes a red pencil, a black pencil and a yellow sponge and puts them in an empty plate.
She turned the white Mark Cup over on a yellow tray.
Mom picked up a teddy bear and put it on a towel.
Mom took the lid and covered the box.
Mom picked up the red onion and put it in Mark's cup.
She picked up the board and put it on a yellow tray.
She picked up plastic bottles and put them in silver trash cans.
All of the above videos are played at 1x the speed.
7.2 One-armed robot (Franka, DROID)
The following data was collected by Positronic at the Franka FR3 workstation and played at real speed; each video is marked with corresponding instructions.
The monkey puts the monkey in Mark's cup.
I don't know what I'm doing.
She moves the cup to the right.
She put the pen in the cup.
Mom put her pencil on the table.
The mother put the fork in the plate.
In the blue cup, the bricks are made of yellow bricks.
All of the above videos are played at 1x the speed.
7.3 Comparison with the Frontline LLM Agent
Front-end multimodal models can now control the mechanical arm in a zero-shot manner through tools or code interfaces. When first evaluated, the GPT-6 Astra surpassed all open access models on RoboDojo with a 22.5% success rate and reached a 76.0% success rate on RoboTwin 2.0.
These agents are versatile: they can solve a variety of tasks in a zero-shot manner. Currently, they lead the RoboLab-120 rankings, with the top two being LLM Agents, with a success rate of 86.0% and 57.1% respectively.
Their disadvantages are delays and costs. On the AA-Robotics benchmark, Astra takes an average of 30 minutes to complete each desktop task, costing $16.36, compared to only 20 seconds for humans. Zhou and others reported that their reasoning for delays hampered actual deployment.
To measure this gap ourselves, we let Praxis-1 and GPT-6 Astra perform the same three tasks on the same YAM remote control workstation via the same controller. On average, the time taken by Astra to complete a task is 15.6 times that of Praxis-1, and the cost of each successful task is 51 times that of Praxis-1, as shown in Figure 11 and Figure 12.
The same robot, playing at 8 times the speed of real time
Praxis-1:13 seconds completed.
GPT-6 Astra: 7 minutes and 54 seconds.
Figure 11: Remove the pencil caps from the Praxis-1 and GPT-6 Astra
The time on both sides is robot execution time. Praxis-1 is on the left side and takes 13 seconds; GPT-6 Astra is on the right side and takes 7.54 seconds.
Figure 12: Time and cost of completing the tasks of Praxis-1 and GPT-6 Astra
The average time taken for successful episodes is calculated from the beginning of strategic takeover control to the end of the episode. The cost of each successful task is included in all episodes, including the failed episode: Praxis-1 calculates an H100 at $3.95 per hour, Astra calculates an API bill. Data was collected on November 9, 2026.
The Frontline LLM Agent is progressing rapidly, with two complementary approaches: Agent can plan a new task, and a fast strategy like Praxis-1 can execute each of its steps in real time.
8. Adapting to new robots
8.1 Google RT-1
To further test how Praxis-1 adapts to robots that have never been seen before, we minimized data collection on Google's mobile operating robot RT-1, which has never appeared in our mid-career training mixed data set.
We started from the latest intermediate training checkpoint, using a complete RT-1 training set for 30,000 step tweaks, which took about 28 hours on 32 H100s. The training set contained about 78,000 episodes, 95 hours of data, using head cameras alone. We evaluated the model on the official Visual Matching Grid at SimplerEnv, which includes 4 missions, 864 episodes, and is the benchmark for most RT-1 strategy reporting results.
Praxis-1, which was tweaked over RT-1, had an average success rate of 80.0% on four missions, which was 52.4% higher than RT-1 itself. In comparison, we used the same RT-1 data tweaked FLUX 3 Action to achieve a success rate of 39.7%, based on the DROID tweak configuration published by Black Forest Labs.
Figure 13: SimplerEnv Google robot by task results (visual matching)
Success rates of missions evaluated on the official visual matching grid.
The following simulation video shows the perspective of the head camera:
8.2 SO-101
The SO-101 is a low-cost open-source single-arm robot aimed at enthusiasts and researchers, and its community has already uploaded thousands of small LeRobot datasets. We collected 3,956 public SO-100/SO-101 datasets from 1,783 contributors, with a total duration of about 1,260 hours, mostly collected at 30 fps.
Each dataset contains 1 to 3 scene cameras; 2,438 of these data sets also contain wrist cameras, corresponding to approximately 860 hours of data.
The name of the camera in the community data set is unreliable: a camera called a wristphone jack or a wristtop jack is usually installed on the wrist. Therefore, we only count it as a wrist camera based on the type of labeling for each camera based on the video, when the two checks are consistent:
Exercise checksIn a sequence of 64 frames, most of the pixels of the fixed-scene camera remain stationary, while the wrist camera does not.
External inspectionUse the zero-shot CLIP model to judge.
We left 102 complete datasets out of the minimized data, for a total of 64 hours, for validation; using the rest of the training episode in the minimized Praxis-1, the training data came from 3,814 datasets with a total duration of about 1,070 hours.
Each sample uses a scene camera; if there is a wrist camera, the wrist image is also included. For data sets containing wrist cameras, only a scene camera is used when sampling, so that a model can run regardless of whether there is a wrist camera.
The model predicts the movements of the five joints and claws of the mechanical arm at 15 Hz, 1.6 seconds each. In this, the movement of the joints represents the deviation relative to the current state. The instructions are to use a video clip to describe the task, or to use substeps that fall within each training window, to automatically generate subtitles.
Each of the videos below shows the strategy of controlling a single-line instruction, playing at 1x the speed, with perspective as a scene perspective.
9. Looking for the best guidance settings
Praxis-1 takes a two-step noise sampling of each action block and guides only instructions to apply without a classifier: the guided forward computation retains the video history and status, but replaces the instructions with blank text.
The video guidance coefficient determines the intensity of the predicted video following the instruction; the coefficient is 1 h, which is equivalent to shutting down the guidance.
The second adjustable parameter is flow shift, which changes the time grid for two steps to noise. A larger shift will leave more noise to the second step.
In a set of fixed simulation environments, we set these two parameters in a step-by-step search, with the starting point for shutting down video routing, shifting to 8. Adding video routing helps to improve performance, while improving the video routing coefficient and shifting can further improve performance, ultimately rising 4.0 percent above the starting point; continuing to increase one parameter will lead to a drop in scores.
The models we released used video CFG for 4, shift for 12.5, and go-noise for 2. These guides were set up for specific robot entities: they were adjusted for DROID Franka, and other robots needed their own adjustments.
Figure 14: The degree to which the CFG guides the action in different video settings
The circle represents the setup that has been tested, and the number marked represents the change in the success rate of the task to 8 tons, per unit point, relative to the initial setup. The arrow represents the search process.
10. Offline evaluation
In addition to the closed-loop benchmark, we also evaluate the model in two ways: the accuracy of the prediction of movement and whether the model imagines the future following the instructions.
10.1 Is the error from the prediction video or the prediction action?
On 32 DROID windows, we define motion error as the distance between the predicted absolute position of the end-executor and the actual position. We use two conditional input patterns to distinguish between two types of error sources: imagining the wrong future and reading the wrong movement from the right future.
Deployment patternsModels: like running on a robot, while sampling their own videos and actions.
Oracle modelReferring to the Oracle test in mimic-video, we provide the model with a sample-only motion after the actual future video is added to the sampler's corresponding noise level.
We observed that the remaining errors came primarily from the future imagined by the video model, rather than Action Expert. Therefore, improving future video predictions should bring the greatest benefits.
It should be noted that deployment mode errors are not a completely reliable indicator: they contain both future video and future action prediction errors, and multiple different futures may perform the same task.
Figure 15: Oracle has released a checkpoint error with deployment mode action.
Average end-user error calculated on the DROID window left; error line as a standard error. The lower the numerical value, the better.
10.2 Can the model imagine the process of a task before it is executed?
In order to understand how the model plans and envisions the future, we provide it with a single header of a three-dimensional constructing scene, an instruction, and a real starting position of the robot, rather than a real episode. In this scene, only the background comes from a real image and other objects are added in different colors, sizes and locations.
Subsequently, the model itself rollout for 12.6 seconds, simultaneously predicting video and action; no real observations are available except for the first shot.
A visual language judging model looks at each segment of the prediction video and only determines that the video follows instructions when the model operates the correct object, moves it to the correct position and performs the correct moves. These scenarios contain objects that are similar in appearance and are used to test the model's robustness following the prompt.
The ability to follow instructions in the imaging process is an intermediate indicator for measuring the model's understanding of the task; the relationship between it and the success rate of real-world tasks still needs to be further studied.
The following example is from the DROID prompt set: Starting with the first shot of the display, a 12.6-second turnaround from the rollout using the DROID-modulated Praxis-1 gives the judgement of the judging model.
The same imagined rollout method is also used to evaluate the mid-term training base model, which is not minimized for any specific robot entity. The following long-term example comes from the results of the base model on a two-armed YAM workstation, which aims to show video quality rather than mission success rate.
Each example starts with a single headshot of a three-dimensional constructing scene rather than a real episode. The model imagines up to 20 seconds of video from three camera angles without real robots and action inputs. We also change instructions during each rollout to see if the model responds accordingly in the video prediction. The instructions for each example are listed below.
See more demos in the original
10.3 Can the model predict the success of the mission?
We want to know if the model can predict its own future and whether the mission will be successful.
In an internal study, we used an earlier, unreleased version of the model and minimized it on Franka's mechanical arm. We froze the model weight and trained a small output head on top of its next prediction block characteristic to estimate the probability of the episode's ultimate success starting at the present time. Its only monitoring signal was the end result of the entire episode, with no artificial scores.
We evaluated 272 real Franka episodes from unseen experimental sessions and strategies from the model. After performing 4 action blocks, about 6 seconds, the score distinguished between successful and unsuccessful AUROCs by 0.77,95% and confidence intervals by 0.70 to 0.83; AUROCs were 0.64 when used only as an indicator.
After averaging each strategy, the score can also be ranked based on the actual success rate of the 9 group checkpoints and sampling combinations, with a Spearman coefficient of 0.79, but a wider range of 0.19 times 1.00.
This is still early evidence. The output head does not currently read instructions, so whether the robot is performing the correct task or not, it gets the same score as long as it makes a clean landing and crawling placement.
Nevertheless, this result points in the direction that we are most concerned about: allowing models to use their imagination of the future as a rating for action, and then improving the action accordingly, so that they learn from their own video predictions.
Figure 16: Real robot mission success rates predicted for the future based on the model itself
Of the 272 actual Franka episodes left behind, AUROC was performed after 4 blocks of action; 0.5 corresponded to a random level. The error line was 95% of the range obtained from the re-sampling of experimental sites and task groups. The results came from earlier, unpublished model versions, which were the results of the study.
11. Quantitative
The performance of Praxis-1 remained stable after MoE experts calibrated its routing to 4-bit NVFP4.
We used quantitative perception training, further subtracting 5,000 steps from the released checkpoints, and quantified the routing expert's matrix multiplication to NVFP4: the numerals in the E2M1 format, each of the 16 numerals in a fraction, the fractional scaling coefficients in the E4M3 format, and an FP32 global scaling coefficient.
We took 600 episodes from RoboLab-120 and compared them directly to two checkpoints:
The NVFP4 model was successful 279/600, with a success rate of 46.5%;
The same fine-tuned checkpoint was successful at 287/600 with a 47.8% success rate when it was run on bf16.
Both are close.
We also repeated the test on a real two-armed YAM workstation: starting from a two-armed YAM checkpoint and performing 5,000-step quantified sensory tuning.
The NVFP4 model was successful 26 times out of 30 rollouts, with a success rate of 87%.
The bf16 checkpoint, which serves as a fine-tuning starting point, has been successful 28 times out of 30 rollouts, with a 93% success rate.
The crawling mass of the two is 2.73 and 2.40, respectively; the average time taken for successful missions is 20.7 seconds and 20.9 seconds respectively. These differences are not statistically significant under the conditions of 30 rollouts per setup.
We will release the NVFP4 checkpoint in the Hugging Face warehouse. After routing experts are stored in the NVFP4 format, the strategic weight is reduced from 54.4 GB to 17.5 GB.
Figure 17:bf16 compares to NVFP4 with the real robot Praxis-1 (minimized on DROID) and Praxis-1 (minimized on two-arms YAM)
Left: Using the same RoboLab-120 tasks and episodes. Right: In a real two-armed YAM workstation, 5 objects are picked up and put into a bag.
12. Limitations of Praxis-1
It's an object that looks similar.The most common failure of a real robot is to reach for an object that is similar to the target, and this incorrect selection has often appeared in videos imagined by the model. This is consistent with the results of section 10.1: most of the movement errors come from the future imagined by the model.
It's a vague instruction.When instructions clearly indicate the properties of objects, such as color or size, the strategy performs better; whereas vague instructions are significantly more difficult to handle. When the instructions alone do not determine the target, the model must make guesses between several possible objects in the scene; if there are similar objects in the scene, it may choose to make mistakes.
It's a long mission.When each step in a multi-step task has its own instructions, the model is able to complete the task; however, sequencing multiple steps with a single prompt is still difficult. The model must track which steps have been completed and continue to execute the next step without additional instructions. Combining the model with a language model or VLM that can break down the task into multiple steps is one of the research directions presented in Section 13.
13. The direction of the future
We will release three checkpoints on Hugging Face under the Apache 2.0 license: Praxis-1 on DROID, Praxis-1 on YAM, and a mid-term training base model.
We will also provide model code, strategy servers, MuJoCo simulation environments under the same settings, as well as full-parameters and LoRA code in GitHub warehouses. See our post for more details.
We will continue to invest in WAM research, with a particular focus on long-term reasoning, agent planning and self-improvement.
