If you take four photographs of an object, you can usually identify the front, side and back of it. But if you want to turn them into a free-spinning three-dimensional model, the problem is complicated.
These problems are actually closely related. To judge the direction of shooting, it is necessary to know the shape of the object; to restore the shape of the object, it is necessary to know the direction of shooting each photo. The less input, the more shading, the more difficult the two are to solve separately.
It was developed by the Hong Kong University of Science and Technology.GenSfMIn the same three-dimensional generation process, the object and camera are gradually shaped. Four unlabeled object images are entered without prior input from the camera, and the model can be combined to restore the entire object and the corresponding camera parameters of each image.NeurIPS 2026Received.
About 76 seconds of presentation: starting from four input images, observing the shape in conjunction with the camera's noise, re-projecting the input perspective, and reconstructing the synthetic and actual shot objects.
A three-dimensional result requires two answers at the same time.
3D-generated displays often begin with a turn-based video: objects turn a circle, look complete, and the texture is clear. But there is an equally critical question for multi-point reconstruction.
For example, a folding trailer in a video. In addition to observing whether the shelves, wheels and fabrics are complete, it can also be used along the predicted direction of the camera to render the three-dimensional results into two-dimensional images and contrast them with the original photographs. The position of the wheels, the edges of the shelves and the surface patterns can be used to check whether objects match the camera.
What GenSfM hopes to get is a set of mutually corresponding outputs: a complete three-dimensional object, and a camera capable of interpreting the input images.Therefore, camera recovery is also involved in object generation and provides the basis for subsequent detail recovery.
Objects and cameras in the same three-dimensional space. Blue represents a predictive camera and red represents a reference camera.
Let the shape and the camera participate in inference
Traditional motion restoration structures (Structure from Motion, SFM) typically look for characteristic matches from multiple images, then use geometric relationships to estimate cameras and sparse 3D points, and combine optimization.
The input is reduced to a few points of view, and the correspondence available and the surface coverage are reduced. A narrowed outline, which may be due to the shape of the object itself or to changes in the angle of observation, is difficult to rule out, with local image information alone.
The generated model provides another kind of clue. It can study the distribution of the shape of objects from a large number of three-dimensional instances, inferring parts that have already been seen to be obscured or not captured. The team previously CUPID [2] had explored the synthesis of objects with cameras under single-graph conditions, and supported multi-angle expansion. GenSfM further incorporated multiple images into a unified synthesis process.
In GenSfM, each image is bound together by a shared object, with each image retaining its own camera representation. Together, they enter the native three-dimensional scattering model and are updated together during the de-noise process. Thus, the missing structure in one image can be supplemented by other images, and the gradually formed shape of the object can also provide clues to the camera's judgment.
The original three-dimensional geometry here means that the model is generated directly on the representation of the three-dimensional object.
From spatial relationships to surface details, two steps away
GenSfM follows the rough and subtle framework of TRELLIS.[1] In the first phase, it combines rough structures with cameras. In the second phase, it uses these cameras to construct a projection relationship between images and three-dimensional positions to further restore geometry and appearance.
The first stage combines the rough structure with the camera and solves the camera parameters through DLT. The second stage samples the image characteristics along the projection relationship of the predicted camera, refining the three-dimensional results.
The entire reasoning process does not require a sequence of independent external camera estimators and reconstructors, nor does it require optimization for each object after additional operation. The shape is inferred with the camera in the first stage, and the second stage uses the resulting spatial relationships directly.
When entering multiple viewpoints, the model must also know which image each camera represents. For this reason, GenSfM gives the image characteristics and its corresponding cameras the indication of adding the same viewpoint tag, and uses shared universal information to show clues to gather multiple images. The tag identifies the source, and the specific shooting angle still requires the model to infer itself.
Information for input images is provided by two types of features: DINOv2 is responsible for semantic clues to objects and structures, a small convoluted network adds multi-dimensional color, edge and texture information. The former helps to understand the whole, and the latter provides a basis for local differences and detail recovery.
The camera can also be written as a space representation.
Objects have a three-dimensional structure that is suitable for spatial representations such as objects. So how do we get cameras into the same generation process?
GenSfM is used for this purpose.UV-oxygen gridIt can be thought of as a projection coordinate: a two-dimensional coordinate for each location in a three-dimensional grid that is recorded in a given image (u, v).
This representation and object structure share a set of 64×64×64 physical grids, positioned one by one, and the three-dimensional scattering model can process the two together. In order to reduce redundancy in dense grids, the shared camera encoder first compresses them into a grid.The potential grid of 4x4x4When combined with noise, the model is treated as an object-shaped sub-variable and multiple compact camera sub-variables.
After production is completed, the camera sub-variable is decoded back into the UV-body grid, and the camera matrix is recovered through DLT based on three-dimensional position and two-dimensional projection to participate internally and externally. Thus, the model can be obtained from the same round of joint production of a rough structured and polygonal camera.
After finding the direction of shooting, the texture is based on location.
For example, designs on the backpack side need not only to be generated, but also to appear in the correct surface position.
If image characteristics are only injected through Attention, the relationship between image regions and three-dimensional positions is largely learned through network implicit learning. GenSfM further utilizes the first-stage camera in the second stage to provide a clear sampling position for local characteristics: projecting active organisms in a rough structure onto each input image, and then extracting characteristics from the corresponding coordinates.
These sampling characteristics are divided into semantic and textural pathways. DINOv2 provides semantic information, while the convolutional network provides local visual cues. The characteristics of different perspectives are averaged within each branch. The semantic characteristics are then converted into three-dimensional underlying spaces through a frozen SLat encoder, and are spelled with the textural characteristics into noise-free physical characteristics.
The projection camera thus links the patterns, edges and colors in the input to three-dimensional positions that need to be refined.The model can take advantage of overall image conditions and obtain local information along the projection relationship, thus improving color and surface pattern restoration.
The area not covered by the input image is still generated by pre-completion. The final resulting SLat indicates that it can be decoded as a 3D gauge or grid.
Experiments to check the quality of the object and the quality of the camera, respectively
Evaluating whether the generated objects are close to the reference results is also a matter of predicting whether the camera has recovered the correct perspective relationship.
The paper selected 400 test objects from Toys4K and GSO respectively, each using two sets of non-overlapping four-point inputs. Both test sets were not involved in the training.
Compared to the multi-point baseline TRELLIS-M, GenSfM's new point of view PSNR is improved on Toys 4K.4.53 dBIt's been a long time since I've been on GSO.3.22 dBThe PSNR measures the closeness of the rendered image to the reference image, and the F-score measures the compatibility of the surface to the reference geometry under a given distance threshold.
The higher the Toys4K, the lower the GSO; the PSNR on the left side of each group and the F-score on the right, the better.
SAM 3D [3] and TRELLIS-S are monotonic lines in which SAM 3D uses four input generations and averages four output indicators respectively. TRELLIS-M, ReconViaGen [4] and GenSfM use multiple viewpoint inputs. TRELLIS-M randomly selects an image as a condition for each de-noise update, while GenSfM allows multiple viewpoints to participate in deductions and explicitly restores the spatial relationship between viewpoints and objects.
See also the camera. The paper uses the relative relationship between viewpoints to evaluate to avoid the impact of overall coordinate selection. Under the conditions of self-predicting inputs, GenSfM's relative rotation error on Toys4K and GSO was as follows:1.60° and 1.27°FreeSplatter [5], ReconViaGen [4] and VGGT [6] are below the comparison in the chart.
On the left is Toys4K, and on the right is GSO; the lower the unit, the better. GenSfM uses predictive input. The two relative positions of the original predictive cameras are calculated according to the rules of permissible position positions, without an ICP alignment.
Go back to specific objects and see if different points of view match
The clock in the diagram below needs to distinguish between the dashboard and the back, but also to retain the top ring and bottom support. The backpack also tests the position of structures such as shoulder straps, and the restoration of lateral patterns.
The contrast contains SAM 3D [3] and TRELLIS [1]. Except for the input column, each column uses the same new perspective one by one, with reference model rendering at the far right. The GenSfM column shows 3D Gaussian rendering.
The transverse video is used to see if the angle outside the input is complete, while the input perspective re-projection is used to check whether the outline and texture already exist match the input, and the two complement each other.
GenSfM is currently focused on individual objects, with the recovery of unseen surfaces still dependent on prior generation, and the effect will also be influenced by training distribution and representation resolution. In this context, it shows a path to combine object generation with camera recovery, first inferring a three-dimensional structure with several unlabeled images, and then returning image details to three-dimensional space with the restored projection relationships.
The paper:Generative Structure from Motion with Native 3D Diffusion（NeurIPS 2026）
The main page of the project:
The full text of the paper is available on arXiv, and you are welcome to visit the project's homepage to see more reconstruction results and interaction cases, or visit the code repository to learn the details and how to implement them.
References
Jianfeng Xiang etc. Structured 3D Latents for Scalable and Versatile 3D Generation (TRELLIS). arXiv:2412.01506, 2024.
CUPID: Generative 3D Reconstruction via Joint Object and Pose Modeling. arXiv:2510.20776, 2025.
SAM 3D: 3Dfy Anything in Images. arXiv:2511.16624, 2025.
ReconViaGen: Towards Accurate Multi-view 3D Object Reconstruction through Generation. arXiv:2510.23306, 2025.
FreeSplatter: Pose-free Gaussian Splatting for Sparse-view 3D Reconstruction. arXiv:2412.09573, 2024.
VGGT: Visual Geometry Grounded Transformer. arXiv:2503.11651, 2025.
