I'm going to tell you something.What's the matter?
In the early hours of this morning, Invidia released a report from the robotics forecaster on November 6.VFlash is a general-purpose decoding framework for fast, precise visual planning.The core idea of VFlash is to allow the model to generate not coordinates individually, but the geometry of the visual information itself, such as points, boundary frames, and trajectories, so that a lightweight model can simultaneously predict the complete spatial information. That is, it can now be predicted as a whole. The decoding framework can allow the model to more quickly generate spatial planning information, such as robot positioning, motion trajectories, and more.Object positioning, three-dimensional motion forecasting and self-driving planning.In the test, an initial single-generation test showed that the VFlash's generation speed was approximately increased in a three-dimensional point motion prediction task.25 times.In four sets of MolmoSpaces mission suites, the average mission success rate was reached with a VFlash visual programming-conditioned world action model.70.2%，And the baseline model is62.6%。Following the release of VFlash, Jim Fan tweeted on X and commented:The time has come for the System 1 visual model to roll out!The next reasoning revolution is physical reasoning at the rate of reflection. At the same time, Jim Fan adds that VFlash is possible.Up to 12 times the speed of decoding, 5.1 timesIn the meantime, we're going to have to start thinking about the future of robotics.And VFlash is even more accurate than VLM.3D trajectory errors decreased by up to 61%It also avoids cascading errors that return to decoding.
Jim Fan comments
01.
Breaking the restrictions on generating tokensLet the model output the entire trajectory at once.VFlash is the first thing that changed.Models generate visual informationThe model generates a coordinate and then predicts the next one based on the contents of the previous one until the entire trajectory is completed. This method is suitable for text generation, but not necessarily for robots.A clear geometrical structure.Not only does the robot need to know where the cup is, from which position it should be crawled, along which route it should be moved, and where the next step should be taken in the crawling process, but it also needs to translate this information into specific spatial coordinates and movement trajectories. For example, the model needs to generate a three-dimensional trajectory containing 10 flight points. Each flight point contains spatial coordinates.VFlash uses a systematic set of parallel decoding designs to support the multiple output formats required by physical AI.The team proposed Flash Primitive Decoding.
VFlash technology architecture
Specifically, it takes a complete geometric object as a decoded block, which is generated by the system using tools to call the declaration output length and then determine the ordering of the coordinates according to the type of visual base element. Subsequently, under the context conditions of frozen VLM, the lightweight draft model predicts the complete visual base element through a parallel computation. The predictions are then processed by the target VLM, updating the context itself, and extracting the characteristics used for downloadable action models. The research team found that there were three key points in the process:Border alignment, position alignment and the need to learn in geometric structures.1 - The boundaries are alignedWhen training draft models and actual decoding, the block boundaries must align with the boundaries of a complete visual base. Thus, the draft model learns the complete geometry rather than the segments cut by a fixed token window. VFlash introduces a structured tool called to declare the length of the base, the way in which the type of the visual base determines the coordinates.It can predict the entire nucleus at once in parallel.Two, aligned position.Before starting a parallel forecast, each output location needs to have a clear meaning.Different coordinate values may occupy different numbers of text tokens, resulting in subsequent location deviations. To solve this problem, the research team used dedicated coordinate tokens or fixed-width numeric fields to ensurePoint of departure, coordinate axis and separatorAlways in a fixed position. This unified arrangement allows models to make parallel predictions and apply corresponding constraints to each position.3 Learn from geometric structuresIn model training, the target VLM is always frozen when training lightweight draft models. The draft model contains five layers of non-causal Transformer and uses a marked visual base to train the Token cross-section. It is able to predict the complete visual base in parallel.No need to align with the target model Token by TokenFor submission text, the model still uses speculative decoding and verification by the target VLM.02.
Robot trajectories, object positioning and 3D motion predicts synchronous accelerationVFlash is not specifically designed for predicting robot trajectories.Three-dimensional robot trajectories, intensive three-dimensional motion forecasting, object positioning and submission planning,VFlash can be used to predict the robot's trajectory in order to verify that the decoding framework is suitable for different types of visual tasks.ShareRobot-Bench and VABench-VThe results showed a decrease in VFlash on all 6 trajectory error indicators compared to auto-retro decoding. With three robot datasets and three target models configured, its average three-dimensional bit shift error (ADE) decreased compared to auto-retro targets.42.6%。In the intensive 3D motion prediction task, the team tested using a frozen MolmoMotion-AR model. Compared to the original model, VFlash reduced the average bit displacement error on the DAVIS dataset.27%At the same time, relatively close overall mass is maintained in dense 3D motion prediction missions.
3D robot trajectory predicted
In the H100 GPU, the batch size is 1.The single-generation rate increased by about 25 timesIn object positioning tasks, VFlash demonstrates higher boundary frame generation efficiency. In the COCO and Dense 200 datasets, it is able to predict the number of boundary frames per second to reach the return target model.2.6 to 2.8 timesThe average F1 score is close to the target model. Specifically, flash-speed native language decoding can generate all the coordinates of a boundary frame in a parallel forecast. Each decoding block is responsible for a boundary frame, and subsequent decoding blocks continue to process other objects. For subtasking in text form, VFlash also retains a speculative decoding mechanism.The same draft model also supports verified decoding of submission text.VFlash divides the text intoThe length is 16.the location block, which containsOne confirmed token and 15 candidate tokens.The target model validates the entire decoding block in a single calculation and retains the longest matching prefix, reproducing it into its next Token. In the example shown in the blog, VFlash and the target model end up generating the same subtasking description.03.
The success rate of robot operations is 70.2%.5 times the speed of final reasoningProducing trajectories faster, more accurate, does not necessarily mean that robots can complete tasks. For robots, visual planning also needs to be further transformed into actual action. Models of downstream world action and the architecture of distributed action heads differ, but they use the same design:Through a cross-attention mechanism, VLM characteristics are read to encode the generated visual programming.These features come from the visual basis of VFlash predictions. The research team then inputs the target VLM and extracts the corresponding characteristics from it to test whether it can improve the robot's operational performance. In this, the action model is trained for robot operation or autonomous driving tasks respectively, while the VLM is permanently frozen. In one of these experiments, the research team has the robot strategy adopted the world action model (WAM). The model is trained by controlling the Cosmos-Predict 2.5 mission.Receiving freeze VLM issues, subtasks and trajectory characteristicsWAM predicts a movement block containing 20 steps each time and then replans it after completing three movements.
Results of the test
The results of the test showed that after using VFlash Visual Planning,The average mission success rate for WAM is 70.2%, and the baseline model is 62.6%.The team also compared different programming information. Two-dimensional and three-dimensional programming, which is generated by the regression model, uses the same WAM architecture.The success rate was 61.0%, compared to 70.2% for VFlash.If the generated subtasks are retained, but the trajectory characteristics are replaced with image characteristics, the success rate decreases to63.7%。This suggests thatVisual geometry can provide additional value for robot operations beyond language informationThe research team is still working.LIBERO series of missionsIn these experiments, the team used the PV3 movement head to read the characteristics of the frozen VLM through cross-attention. The baseline model used Qwen3-VL-4B encoded images and instructions. The VFlash provided the model with state information through predicted subtasks or complete planning.
Results of the LIBERO series of missions
It's the first time I've ever been to Libero-Pro.The success rate of the baseline model is40.2%；After joining the generated subtasks, the success rate reached45.3%；Following the use of comprehensive visual planning, the success rate further improved to47.0%。On LIBERO-Plus, the success rate of the baseline model is60.7%；Success rates are high, whether using generated subtasks or using complete visual planning.62.6%。On LIBERO, the baseline model and the model that uses submission information only have the same success rates.93.4%；After using full visual planning, the success rate reached94.2%。In addition to robot operation, VFlash is also used for autonomous driving planning. In the frozen planning VLM and motion heads built on ReCogDrive, VFlash achieved competitive autonomous driving performance on NAVSIM and Waymo datasets while reducing code latency. The same set of variable-length decoders also supports the trajectory output used by Alpamayo. In hardware, the team used the TensorRT-Edge-LLM operating environment and the TensorRT engine to conduct inferential testing on Jetson Thor, DGX Spark and H100.75% to 87%.VFlash has accelerated this phase.12 to 42 timesIt was eventually made available on Jetson Thor and DGX Spark.Robotic reasoning has improved five-fold and NAVSIM autonomous reasoning has improved more than three-fold.It should be noted that these data are measured in batch sizes of 1 per request processed. The model phase delay is measured through CUDA events, which include the characteristic preparation time of the target model. All acceleration ratios are on the same task and hardware platform, as opposed to self-return decoding and VFlash.04.
Physical AI is more than just reading.We need to create spatial planning faster.As visual language models gradually enter the field of robotics and autonomous driving, models not only need to understand the environment, but also need to be able to understand the results at a faster rate to translate them into specific spatial planning. VFlash's practice is to start by decoding the link, changing the way the model generates visual information, using the geometry of the visual information itself, and transforming the coordinates originally generated into the complete visual basis for parallel predictions.
