Today, people in AI circles are playing games.
Lifeifei's startup, World Labs, has launched its first "application" Frameboy, which uses AI to generate the first real-time human image, turning generated AI into an experience that can be explored directly.
Frameboy is an optimized demo version of an Atlas World model that uses cloud GPUs for streamlined transmission, with a maximum torque of up to 15 fps. The detected model's response time is about 70ms, which is close to cloud gaming.
I'm going to try it again.
It currently offers a selection of scenes for users to explore, including scenes from the real world, cyberpunk spaces, and Stanford University campuses. Some of these scenes are from a single image, allowing the Atlas brain to complement other perspectives, gradually forming an explorable environment, or by entering multiple photos from the same location, and then generating new perspectives between and outside the photos by the model.
World Labs also integrates Frameboy directly into the beta web application Viewfinder as a streaming coverage rendering model, allowing users to build scenarios with their own inputs and then explore and expand the virtual world in real time.
Just like using a camera, you can navigate from different angles and generate consistent 3D worldviews. Each shot is produced by Atlas, containing color and depth information. Once ready, users can trigger the Frameboy layer to play the generated shot in real time as they explore.
Atlas: A three-dimensional world from a photograph
To understand the significance of this real-time demonstration of Frameboy, we first need to look at what the Atlas is behind.
On September 1, World Labs, founded by Li Feifei, officially released Atlas, a general-purpose world model that can process text, images, video, and 3D spatial information from scratch. Unlike video models that primarily generate continuous images, Atlas attempts to solve a more complex question: What would the world look like if the camera moved to another location?
World Labs presented a very intuitive example at the time of its release.
The researchers first provided Atlas with a picture of the garden, asking it to generate a picture of the area from the air. Since the original photographs contained only part of the garden, the model had to make its own estimates of the surrounding houses and environments, although it reproduced the existing landscapes.
The researchers then added a second photo showing a cottage next to the garden. This time, the Atlas produced a picture that included both gardens and cottages, but the buildings further away were still completely replenished by the model. It was not until a third photo was added showing the house nearby that the model reconstructed the entire scene closer to the real environment.
This process demonstrates an important feature of Atlas: it can both reconstruct the space already observed based on limited images and use the world knowledge learned in training to speculate on what might exist outside the lens.
World Labs refers to its core mechanism as "Spatial Context".
Simply put, Atlas not only identifies which objects are in the image, but also associates each image with the camera's location in 3D space. When a user specifies a new observation location, the model generates the image that should be viewed from that location based on existing spatial information.
This is an important difference between Atlas and ordinary video generation models, which typically rely on words such as "moving to the left" and "rotating around a character" to describe the control lens, while Atlas can directly receive accurate camera positions and angles, allowing users to clearly specify the world view.
In its technical architecture, Atlas uses the multi-modal back-to-back distributed Transformer. It combines language models with the idea of distributed generation models: organizing text, images, camera positions, and depth of information into a unified context in space, and then gradually generating new images and spatial data.
In the initial demonstration, users were able to pre-design the path of the camera, allowing Atlas to produce the longest video of a minute, with a resolution of up to 1440p, and to maintain the consistency of the scene between different shots as much as possible.
But Atlas's capabilities include more than video production.
It can also generate images with depth information at the same time and output 3D representations such as point clouds, 3D Gaussian Splatting, etc. In other words, the model can not only present an image from a certain angle, but also generate data that can be used for workflows such as 3D rendering, spatial reconstruction, etc.
For example, researchers used three to five ordinary cell phones and motion cameras to capture the same scene from different angles. Based on these images, Atlas reconstructed space, allowing users to switch viewing angles at the same time as the action takes place, achieving a "bullet time" effect similar to that of a film-making hack empire without having to deploy a traditional large-scale multi-camera shooting system.
Another application is robotic simulation. Researchers can record real environments using mobile video, rebuild space using Atlas, and generate images and depth data that can be observed by their on-board cameras as robots move along different routes. This provides a new way to build a robotic simulation environment using a small amount of real records.
Now, with the advent of Frameboy, the way people create based on models of the world has changed, and we can observe side-by-side, side-by-side decide the next step. This experience has potential in film exploration, spatial conceptual design, and game design.
References are given below:
