Waymo Tests a New Framework for Physically Consistent Driving-Video Generation

Motion Forcing team posted
QbitAI is public.

The car next to the runway suddenly clogged, and the car rushed to slow down and avoid the lane.

This image of the car was not taken by a vehicle recorder, but it was created by AI.

For self-driving cars, the hardest things to fix are the unusual moments that can cause immediate danger once they happen.

Now, all you need to do is generate videos like this.A photo, along with several vehicle tracksIt's enough. How to get in the front car, how to slow down the avoidance, the whole process is in accordance with the laws of physics, the car and the car will not wear each other's moulds.

△Two cars are ahead of each other, slowing down and avoiding traffic.

This is the new work of a team from Guangzhou, Hong Kong.Motion Forcing。

It's a simple idea to figure out how things move and then to paint what they look like.

In Waymo's real driving data, its action consistency and physical rationality are both benchmarks.The firstIt's over.Seedance 2.0andWan 2.6These are the two main video models.

In addition to driving scenes, it can also generate videos of dominoes collapsing, mechanical arms pushing animal bodies, as well as complying with physical laws.

△Domino's skeletal chain collapsed△Mechanical arms pushing the animal body

It's like, you have to be right, you have to be right, you have to be right.

Today's video models are increasingly sophisticated, allowing complex scenes to be easily exposed: a car drives through another car, the car is blocked, and the reaction after the collision is not the same.

What is the problem?

The team believes it's because the model has to deal with two things at once: how things move: where the car goes, how fast it goes, who makes it go, what it looks like: body color, light shadow, road texture.

For the model,It's a lot easier to draw the details of the picture than it is to make the physical laws right.Two things come together, and it naturally prioritizes the image, putting the physics aside. There is also a layer that is easily confused, the camera is installed on the car, and in the image a car is moving, perhaps it is turning itself, or it may be his car and the car are moving.

Some methods have already attempted to give a rough direction of motion first, and then to make the model look like the painting. However, there is too much distance between a few sparse trajectories and an entire high-definition video, the scene is complex, and the model either doesn't listen to the commands or is physically wrong.

Motion Forcing is a two-dimensional method of solving problems: first, dismantle the steering wheel and "how to dismantle the steering wheel", then separate the car from its control and divide the generation into three layers:Dots, shapes, shapes。

It's like painting: first you mark a few points in a good position, then you sketch out the outline and the back-to-back relationship, and then you color it.

In Motion Forcing, the point-and-shoot is the trajectory of the user's image, the point-and-shoot is a very close, colourless, deep video, and the ultimate colour video.

This is the main line of production. In addition to the main line, the team added two key designs: one used in training to force the model to really learn physics; one used on input to tell the model how to drive a car.

△Motion Forcing overall framework: points, tracks, shapes, depths of video, appearance, colour video

Point: Every car is a circle

Each vehicle is simplified into a circle: the center of the circle represents where it is, and the size of the circle represents how far and near it is from the lens.

This is a very flexible way of controlling: hand-drawing arrows, writing an instruction, or even directly adjusting the speed of a particular moment to turn the trajectory of a point.

Form: Prepare who's ahead and who's behind

Next, the model does not draw a color image, but instead generates a deep video (a gray image, where each pixel indicates how far away from the lens is).

There's no color or texture in the deep video, only spatial structure: who's in front, who's behind, who's stuck, and how far between the two cars.It's the first step to deal with collisions, obscuring these physical relationships.

In the end, the colour changes.

After the spatial structure is established, the model uses it as a blueprint, adding colors, shades and textures to produce the final video.

It is worth noting that the two steps are the same model. In training, the model switches randomly between the two tasks of pushing the structural thread and color thread on the thread.One model learned at the same time how to pump the physics engine and the filters.。

With a layer of deep video in the middle, there is an additional benefit: users can check once before the color changes, move the vehicle to the wrong location, and then decide to render.

It's about hiding the answer and forcing it to learn physics.

In reality, users often draw only the first few steps, or only a few of the cars.

So the team deliberately concealed the answer during training: sometimes the second half of the trajectory was obscured, sometimes the entire trajectory of a vehicle was removed, but the model was still required to produce the full, correct depth video.

If you want to answer yes, the model should.It's about learning about inertia, obscuring these physical laws.For example, the trajectory is drawn only in the first half, and the model must start at the speed before the car, depending on where it will start from behind; the stuck car cannot disappear in space until it reappears.

You can move the lens and paint it directly to the model.

Look at the design on the input. It's all about how the surrounding car moves. The car is also moving, the camera is installed on the car, and the whole picture follows the turn.

The common practice is to give the model a series of digits, such as how many degrees of turn, how many meters of turn, but it is difficult for the model to accurately match the series of digits to the picture.

Motion Forcing changed its approach: First, take a deep map of the first shot, scroll through the direction of the car, and then give the model a preview of the motion of the camera.

What the model sees is no longer abstract numbers, butWhere should each pixel move?In this way, the surrounding car and the car can be adjusted in different ways. The video generation by the model can be done, and at the same time, the car can slow down in different ways; the same part of the car is driving, the front car can also be cut at different times or angles.

It's not the most sophisticated, but it's the most physically demanding.

The team tested Waymo's 100 real-life driving videos using three dimensions: images that don't look like real-life videos, inconsistent action, and inconsistent with physics.

The results are interesting.

Seedance 2.0 and Wan 2.6 are based on mass data pre-training, which is much better.

But when it comes to motion and physics, the situation is reversed: Motion Forcing.Action Consistency Score of 205.2The lower the better than the 316.2 of Wan 2.6 and the 345.6 of Seedance 2.0.Physical rationality score of 33.2It is also higher than both 31.2 and 30.5.

In other words,The larger model is more beautiful and the Motion Forcing opens more tightly to the wall.In the case of autonomous driving simulations, the latter tends to be more stringent.

△Comparison results on the Waymo dataset: Motion Forcing ranks first in both motion coherence and physical rationality

However, the team also admitted that there is a shortcut to Motion Forcing: in scenarios where pedestrians and bicycles are particularly dense, several circles are difficult to describe the movements of such small targets; when multiple vehicles are heavily overlapping, the depth may also be unclear.

Presentation by the team

The first author of this paper is Xu Xuan, who is currently pursuing a PhD at Hong Kong University of Science and Technology, Guangzhou.

The author of the communication is Prof. Ying-Cong Chen, from the Hong Kong University of Science and Technology, Guangzhou/Hong Kong University of Science and Technology, assistant professor in the field of artificial intelligence at Guangzhou University of Hong Kong, and associate professor in the Department of Computer Science and Engineering, having previously been a postdoctoral researcher at MIT.

Currently, the team has opened source code, models and interactive demos: upload a picture, click on the car in the picture, draw trajectories and generate video.

Link to the paper:https://arxiv.org/abs/2603.10408
The main page of the project:https://tianshuo-xu.github.io/Motion-Forcing/
The code is:https://github.com/Tianshuo-Xu/Motion-Forcing
The model:https://huggingface.co/TSXu/MotionForcing_driving

Well, we will (as soon as possible) get back to you:)