Tsinghua Team Open-Sources a 3D World Model for More Consistent Video Generation

Currently, the most powerful video generating model, such as seedance 2.5, has some obvious drawbacks when generating videos based on user intent.

Using seedance 2.5 to create two specified characters for chess as an example, it can be found that the sota video generation model still lacks much consistency in detail.

Tip: Generate a live video of two people playing chess face-to-face, show the full Spanish start, and strictly execute e4 e5, Nf3 Nc6, BB5 a6, until the next king moves with the opponent's response.

seedance 2.5 accurately generates the corresponding characters and the correct chess moves, but as the video continues to be generated, the position of the chessboard becomes increasingly confused.

The first chess piece is still in place, and the next chess piece is replaced or even added to the chess piece.

The characters, landscapes, and shadows are all rendered correctly, while the chessboard details are incorrect.

This type of problem is also a familiar "draw card" experience for video creators: the creator writes out where the person should go from where, how the lens should be adjusted, whether the result is through the wall or running away.

When you're lucky, you generate the video you need once, and when you're unlucky, you repeatedly generate the entire video, which not only consumes a lot of time and energy for the creators, but also a lot of computational resources.

For content that involves precise action and spatial relationships, user requirements have been gradually upgraded from whether the image is beautiful to whether the content is happening as planned and whether the details of the video are consistent back and forth.

The problem isn't that the model won't draw the board, but that it must remember in a continuous lens where each chess piece is at the moment, where this step should be moved, and what state the other pieces should be in after moving.

At the same time, the video-generating model must also simultaneously generate characters' chess movements, footage, and shots.

This requires models with precise detail control capabilities as well as high-quality video rendering capabilities, and almost all current video generation models do not have the ability to do both at the same time.

A team from Tsinghua University and the University of Pennsylvania came up with OpenMise, a set of space-scene editing tools for Agents, where ecolocation is a bridge to Agent Linking Space Scenes.

OpenMise achieves consistency in detail by breaking down "difficult video generation tasks" into "two simple stages".

Address of the project:https://github.com/Rath-Team/OpenMise

First, OpenMise uses Agent to call Blender to place scenes, characters, actions, cameras, and lights in the same three-dimensional world to generate easy-to-edit sketches.

The creator then gives the draft video to an existing video generation model, completing the rendering of the actual picture or a specific style.

The simplest way is to "directly generate video".」Improved to "Builder + Render"」Two stages are generated.

OpenMise uses coding agent to design precise mobile drafting and rendering details.

The Chinese team believes that the core of the problem is:At the same time, the creators put too many demands on video generation models.The team's solution is to divide a complex task into two simple tasks.

1. For video consistency and compliance with the physical laws, the OpenMise team believes that creating specific actions directly through a Sota-level coding agent (e.g. Deepseek V4.1 Flash) can ensure that the whiteboard complies with the physical laws.

2. For a video generation model such as Seedance 2.5, rendering a white model to a real video is a simpler task, as the model no longer requires precise maintenance of back-to-back consistent chessboard states, but only rendering the precise white model state that the Coding Agent has sorted into a clear, real video.

Through this task breakdown, OpenMise succeeds in allowing the two-stage tasks respectively to be accessed by the most appropriate AI tools.

Continuing with the chess example below, in the face of the same chessboard requests, OpenMise will first create the chessboard, chessboard, characters and mobile sequences in the Blender, so that each step of the chessboard has a clear start, finish, and time. The white model video generation by the three-dimensional scene allows the creator to first check how the chessboard progresses. The team then gives the white model video with the characters, reference charts, and hints to Seedance 2.5 to generate a written version.

The following is a chessboard white paper by OpenMise:

OpenMise is a video game.The first step is generatingChess Whiteboard Draft (Guaranteed to be a perfectly accurate chess track)

And here's a look at the real chess board that was created as a reference in the draft:

Video references to OpenMiseDraftedChess matches produced

As can be seen, the consistency of OpenMise videos generated in two phases has been significantly improved.

If the same video is generated using a seedance 2.5 iteration and then selected from it in accordance with the laws of physics, it may take dozens of iterations to achieve the same effect as OpenMise, and there is never a way to achieve the same effect as OpenMise.

This example illustrates how well OpenMise solves an industry problem for video creation using AI.

Chess is an easy-to-understand example, and in the current field of video creation, character walking and indoor mirrors are the same problem.

The creators may propose requirements such as "from A to B", "through this door", "always shooting behind the scenes with the protagonist", when the model needs to maintain a stable spatial state, including where the door is, how wide the character is, can the camera go backwards, who moves the next second... These are problems that cannot be done directly with the video generation model, while OpenMise can be very prominent.

The core idea of OpenMise is to break down a complex problem into two simple problems.。

Creating a consistent video in detail relies on two stages: Builder and Render.

Builder is solely responsible for designing a draft of specific physical laws and action details, while Render is solely responsible for rendering the draft into a true finished piece.

OpenMise in two phasesSystems for producing videosArchitecture

Video: OpenMise controls Agent using Blender to automate asset creation and asset management.

Builder enables Agent to break down the natural language script into an executable 3D production plan. Skill specifies when to calibrate the scene, how to arrange it with or around it, and what to check after rendering.

The tool is then responsible for importing assets, calculating paths, creating skeleton movements with camera keys, configuring lights and starting rendering.

Agent calls these tools through Blender MCP, puts the characters, footage, and lights in the same set of coordinates and time axis, and then produces a draft image.

The Builder gives editable Blender engineering and structural parameters, so the problem can be positioned at specific nodes. If the path needs to be changed, the path is adjusted, if the person is going right but the lens is going through the wall, the camera is modified, and if the movement is convenient, the light is wrong, only the light is modified.

Compared to the whole reproduction, this intermediate process leaves the creator with a place to check and modify locally.

In terms of professional technical details, Builder has the following innovations:

To enable Agents to truly understand the 3D assets of creators in video generation scenarios

Builder supports Blender native scenery and can access 3DGS scanning, as well as 3D assets such as GLB, GLTF, FBX, OBJ, Blend, PMX, etc. But in an experiment with video-generation creation by the OpenMise team, scenery can be shown and the system already understands that there is a large gap in the scene. For example, 3DGS mainly preserves Gaussian distribution and appearance information in space, and does not naturally provide the location of room name doors, real dimensions, walkable ground or collision boundaries.

Use the natural language command OpenMise to import three-dimensional assets, and comply with the language command to complete the import in a specific location.

Therefore, the scene is accessed by coordinate calibration, scale calibration and space marking. The agent needs to align the ground direction with the Blender plane, establish a corresponding relationship between the scene unit and the person's height, and record the starting point, endpoint, gate, bedside, accessible area, obstacle area and range of camera activity. After completing these steps, a "visible room" becomes a space where routes and locations can be planned.

Separate treatment of total displacement and skeletal movements in personality movements

In video-generation scenarios of human objects, characters can be generated either by simplified geometry and skeletal procedural systems, or by external character assets. The former is suitable for fast white model drafts, with the system able to clearly master character sizes and skeletal structures, while the latter requires additional coordinate direction, proportions, bone, mould and animation compatibility checks.

Not only does the agent need to be able to successfully import the content, but it also needs to be able to process the content more thoroughly.

Builder ensures that the character's position and details of action are monitored in the video. Path control determines the position and direction of the character in the scene, and motion control is responsible for walking, lifting, standing, and body posture. Path control not only checks the center line, but also takes into account the character's occupation of space, avoiding situations where "the path passes through the door and the shoulders hit the door frame".

Bone movements and leg restraints are conventional, accompanied by overall displacement, and examine the relationship between the legs and the ground.

Let the camera and the lights share the same timeline

In video production, scene creators often encounter language such as "behind-the-scenes", "slow forwarding", "going into the bedroom after surrounding". In video production models, these languages are converted into the camera's position, direction, focus, observation objectives, and key points.

In a narrow space, the camera can follow a person's previous path, adjusting the distance to the direction of observation, reducing the risk of crossing the wall when turning.

Human movements, camera movements, and light changes share the same timeline, so "people entering the bedroom cameras start gradually moving around the turning point" can correspond to the precise position of the cameras.

The lights are also controlled in a parametric manner. Types, location, direction, color, power and luminous size can be set separately and can change over time.

For 3DGS scenes, Builder also accessed KIRI's approximate re-lighting setting, allowing the scanning of walls and floors to present new lighted changes, providing space for atmospheric adjustments in the drafts.

Clean intermediate documents to be checked by creators before execution

When the scene, characters, action, footage, and lights are ready, Builder builds the scene in a separate blender process, generates animation and rendering, and then encodes the image sequence into video.

Before execution, the tool checks the range of parameters, object references, time axis and key axes, spatial paths, floor support and actual animation data. After execution, engineering files, structured plans, running logs and inspection reports are retained to facilitate positioning problems and targeted modifications.

Render completes visual presentation with the help of an existing video model

The Builder itself can render the draft in Blender, output viewable white video and editable scene files, and is not dependent on a cloud video model. When you need to write in-depth, you can enter the Render: refer to the white video along with the character image, and give script hints to the existing video generation model.

In the chess case, the team enters the material into Seedance 2.5; the draft is responsible for providing a continuous chess movement and shooting structure, and the video model is responsible for completing the character's appearance, material, and image style.

At the end of the first phase, this two-stage production method has already yielded valuable results. For the verification of paths, locations and compositions, the white model itself is a useful result. For the finished footage aimed at the audience, it can provide a reference to the movements of the characters, the rhythm of the footage and the structure of the scene for subsequent production.

The OpenMise team's showcase of indoor cases begins with a scan of a 3DGS room. The original input retains the visual appearance of the corridors, doors, and bedrooms, and the Builder then completes the ground direction and scale calibration and enters the characters' starting points, bedrooms' endpoints, walkable areas, and range of camera activity into the space.

Thus, "From the corridor to the bedroom" became a three-dimensional route that could be calculated, verified and modified continuously.

The 3DGS scenes used in indoor cases are as follows:

OpenMise imported indoor 3DGS scenes

The local snapshot of the room is as follows:

3DGS room locations and walking routes

From the corridor to the bedroom Builder, two things need to be addressed simultaneously:

1. The path controls the direction of the person as a whole and determines the direction and direction of the person as a whole, and it is based on the route of regional inspection that can be passed through. When examining, look at the path center line and calculate the actual width of the person to enter, otherwise the "centre through the door" will also become a shoulder impact door frame.

2. Motion control is used to arrange movements, landings, and standing with bone movements and leg restraints to keep body movements consistent with overall posture and to check the foot's relationship to the ground. External roles must also be checked for bone, pelvis and animation compatibility.

Let the characters, lenses and lights share timelines

The camera is also an object in the three-dimensional world. The "behind-the-scenes" needs to be dismantled into positions, observing targets, focusing, tracking distance and time, and then transformed into the camera's key holes; if the camera is only attached to the person, the person can still cut into the wall when the narrow corridor is turned.

The Builder allows the camera to follow the path of the person before, adjusting the direction and distance. To the bedroom, the person looks around the room, the camera also switches the observation objective on the same time axis.

Lighting is also controlled by structured parameters: type, location, direction, color, power, and dimensional lighting can be set separately, and changes can be arranged on the time axis.

The size mainly affects the degree of softness of the shadow, and the power determines the intensity of the lighting; "larger lights" and "brighter lights" are therefore not the same operation.

In front of the 3DGS scanning room, OpenMise also accessed KIRI's approximate re-lighting setting, giving new light changes to the walls and floors; it retains the visual basis of the scanning scene while leaving room for fluorescent adjustment.

After the fixed route and lens, the lights can also be adjusted separately:

OpenMise lighting modification demonstration

White Paper Drafted and Finished

In the team's indoor case, users entered a 3DGS room and asked the person to walk from the corridor to the bedroom, with the camera continuing to follow the background; the person entered the bedroom and looked around, changing the lens as well.

Builder arranges scenarios, character routes, movements, and camera trajectories accordingly, and produces a draft video.

The most important use of this version of the image is to check whether the person is in the right proportion to the room, whether the door can pass, whether the camera switches from the corridor to the bedroom, whether the observation movements after entering the bedroom have been taken.

The biggest change from a white model to a finished film is the visual appearance; where the person walks from, how the camera follows, has already been made clearer.

OpenMise 3DGS Rooms Walking in White

Today's video models are increasingly good at producing realistic characters, materials, and images.New standards for complex video creation have been upgraded to pair spaces、Continuous control of movements and shooting processesThe person needs to move along a designated route, the object needs to remain in a stable state, the lens needs to accurately complete the scheduling in complex scenarios...

OpenMise's technological exploration was to add such a calculable middle layer to the generation of models. The creative intent in natural language was first transformed into three-dimensional coordinates, object relations, motion paths, camera keys, and a unified time axis.This idea completely solves the industry's problem of video generation not conforming to the laws of physics.A validated white paper is used as a basis for the production of video models.

As a result, spatial consistency no longer relies entirely on "guessing" models from pixels, but on a set of predefined assets that can be executed, checked, and modified.

OpenMise is changing the way creators interact with generating systems. Past problems often have to wait until the finished video is discovered, and modification often means re-generating entire videos.

After the introduction of OpenMise, the problem can be localized to specific objects and parameters. Routes are incorrect for modifying paths, character-to-ground relationships are abnormal for adjusting movement constraints, cameras do not meet the expected critical thresholds for resetting cameras, and visual styles are handed over to the Render stage to continue.

OpenMise breaks down the idea of a complex task into two simple tasks that are simple and effective for today's day-to-day video production business, such as the production of high-quality shorts, which not only greatly improves quality, but also releases creators from the drawing board, greatly reducing production costs.

If past video generations mainly addressed "turning a sentence into a picture", the next step would be to make everything in the picture "go according to plan".

From pixel-level visual generation to co-modeling of the relationship between space, time and objects, OpenMise attempts to move AIGC video from one-off results to truly modifiable and reproducible content production: where people want to go, what the camera wants to see, can already be discussed clearly in 3D space.