How about replacing 10,000 tracks with an encoder?
This article is based on the Intelligent Base Aluminum Columns - Chapter 16.
Many teams are now prioritizing DINOv3 when they are working on robotic models of the world.
The same cup, the same mechanical arm, the same trajectory. Turn the white table into a wooden strip, turn off half the lights, and grab the success rate.
Is there not enough data?
There is one thing in common: the color of the tablecloth, the brightness of the light, the geometry of the object in the picture, but the judgement of the model has changed.
This indicates that it's color and texture, not shape. If this isn't solved, adding data will just make it fit on more colorful tablecloths.
The root of the disease is in the encoder: in the characteristics it produces, color and texture dominate.
In 2026, a study of robots will reveal that quite a few of the same movements occurred: holding the visual encoder up, not training it.Instead of having a more geometrical skeleton, and holding onto it, you're being pulled back into the color and texture of the data when you're not tweaking it.）。
They chose the same thing: DINOv3.
It's becoming a visual underpinning for smart phones that are default handheld.
For example, the goal of a world model is to predict the future.Changes in scenery caused by the actionInstead of producing a high-resolution, realistic image, it is more concerned with what, where and how objects change than a pixel texture. DINOv3 is characterized by a natural bias towards semantics and object structure, a stronger interference with light and texture, and is ideal for comparing states, tracking time, and cross-domain alignment.
For this reason, more and more WAMs (World-Action Model) operate simulation, emulate learning projects, and use the frozen DINOv3 as a visual encoder to send latent characteristics into the world model as sequential predictions rather than pixel rebuilding routes.
However, it's not just a reason, it's a way.
To answer this question, we have to look at something more real: what does a graph go into, what does it spit out, and why is the output the kind of structure that a robot needs to look at, rather than the appearance of a color wheel?
Basics of the DINOv3 model
Input and output
DINOv3 natively receives single frames of RGB images without the need for a target box, tag masks or depth charts.
The input size supports variable resolution and is commonly used in sizes such as 512×512×896×896, up to 4096×4096.
For example, entering an 896×896 image will be cut into 56×56 patches.
In object-oriented tasks, depth charts are often added as additional pathways, but this does not belong to the DINOv3 native input, but to the task-side expansion.
ViT is the core backbone of DINOv3, and DINOv3 is based on the ViT architecture.
Models produce two types of characteristics and are the most commonly used form of representation in the study of physical intelligence.
The following is an example of a 512×512 pixel desktop scene (the picture contains a white cup on the left, a metal head on the right):
Global token (class token):
768 dimensional vectors (such as ViT-B/16) are used to summarize the overall situation of an entire image and are suitable for judging whether the overall scenario of two images is similar.
This group of vectors can remember the cups and metal caps on the glass table, but do not remember the cups on the left or the caps on the right, and are not generally used to locate objects.
This is the first time I've ever seen a patch token.
A digital array in the form of a two-dimensional grid. The images are cut into many small blocks of 16×16 pixels, and the ViT-B/16 model gives each small block a separate output of a series containing 768 dimensional vectors.
For 512×512 input images, divided by 16 pixels according to the size of the patch, you get a total of 1024 small squares of 32×32, the output being an integer set of 32×32×768.
Each 768-dimensional vector is the model's self-learning coded vector for the characteristics of this small region of silicon structure. It does not represent color, nor does it have an artificial predefined meaning, but follows a rule:
Nearby areas of visual structure, encoded closer in high-dimensional space。
The structure described here is very similar to that of a silicon, focusing on the contours, curves and ups and downs, rather than the colours.
For example, two small squares adjacent to the side of the cylinder wall are cylindrical curves, and the two numbers are very close; even if the metallic curves in a different image are the same on the side of the cylinder, the numbers will be closer.
Even if the two regions are exactly the same color, if one is flat and the other is curved, their numbers will differ significantly.
This set of code does not have any text labels, and its use is based on similarity contrasts: it presupposes that the code of the known headboard sidewall patch is used as a template, the new patch calculates the distance from the template, and the distance is small to determine the approximation of the structure.
This allows the robot to identify which patches belong to the same object.
It's important to note that the DINOv3 native does not directly output coordinates, object masks, position capture, depth, it only output characteristic numbers.
In order to obtain coordinates and conceal these results, an additional simple network layer after DINOv3 needs to be converted.
DINOv3 Location Ability: Exact or approximate location?
This is one of the most important things to consider when using DINOv3.
Because the native DINOv3 features can only provide approximate position and object boundary perception; with a lightweight decoder/mission head, it is possible to make a pixel-level precise positioning, but the accuracy is limited, and cannot be perfected by specialized position estimation calculations.
Originally frozen bone, no mission head: only approximate location.
Patch Granularity 16 pixel determines the exact upper limit for direct matching is 16 pixel power to distinguish which area the object is in, identify the outline and approximate center, suitable for rough positioning, target retrieval and navigation, but can't distinguish small deviations, can't be directly used for fine-tuning operations like mm-level insertion, screws, etc.
Applicable scenarios include robotic rough navigation to locate the area where the target object is located; determining whether a scene has changed; and image goal matching.
The limitation is that it is impossible to distinguish between small deviations of objects and cannot be used directly for precision work such as mechanical arm millimetre level inserting and screws.
Freeze the skeleton + mount a lightweight task head: the exact position within the image plane.
On top of freezing DINOv3's density characteristics, a decoder with few parameters is trained to sample the patch characteristics back to the prototype resolution, which can predict object masks, key points, grab centers, object 2D outlines.
This was done using DINOv3 to learn the relative relationships between object structures, edges, and components, and the task leader learned to map these structural features to pixel coordinates.
But at the same time, there are two limits:
Second, it is the precise position on a 2D image plane, not the mm-scale position under 3D space. 2D pixel coordinates cannot be directly converted into real-world 3D coordinates, and require camera input, depth information, or multi-view triangulation.
Second, DINOv3 is essentially a self-monitored visual representation, not a specialized high-precision position estimator. It is suitable for desktop object grabbing, object placement, simple alignment, cm-level spatial operation; but is not suitable for independent high-precision operations such as precision assembly and small parts positioning without additional depth/point cloud assistance.
Core Principles of DINOv3
DINOv3 is Meta's third-generation DINO self-monitored visual transformer model, launched in 2025, which trains the LVD-1689M data set containing approximately 16.89 billion unlabeled Internet images.
Unlike traditional supervised learning models that require manual labeling, DINOv3 does not require the entire training process to tell the model, "Hey, this is a cup, this is a headset, and it relies on self-supervised learning to look for learning signals from the image itself".
As mentioned earlier, the characteristics of the output of DINOv3 do not have a human-readable meaning in themselves.
It's a direct match of similarities, zero training.
The digital array of the target object's reference patch is saved in advance, and the new image extracts the patch. After the array, the distance between the two arrays is calculated.
For example, if a robot is looking for headers, it can use a reference array of headers to find the matching patch headers, by comparing them one at a time across the entire feature grid, and positioning the headers in roughly the same area in the image.
This method has only rough positioning capabilities, and it is not possible to distinguish the exact categories.
Take a lightweight task-headed network and give the character a sense of meaning (a little training)
We mark a few images (mark where objects are, grab points, masks) and train a simple network. This network learns a mapping relationship: we translate the meaningless array of numbers that DINOv3 produces into the results we want, such as object masks, grab coordinates, object categories.
Let me give a simple analogy:
DINOv3 is like a color-blind painter, who can accurately discern the contours of objects, the curves, the ups and downs, but can't call the names of the objects, the painter draws a sketch (i.e. this set of numbers); we also find an interpreter (a task head), and look at the sketch to tell the name and location of the object.
The sketch itself has no words, and the interpreter is responsible for interpreting the meaning.
YOLO can be used as an interpreter for DINO to add meaning to the language.
Training
The entire training is usedDual model of teacher from distilled frameAt the same time, it integrates iBOT masking prediction, KoLeo feature deployment, Gram-defined loss, and multi-target collaboration training.
Dual model of teacher from distilled frame
When training, two different random transformations of the same original image are generated.Global view(cuts to larger areas) andLocal view(Cut a small area of the image)
The output characteristics of the teacher model serve as a soft target, guiding student model learning. The teacher model is not involved in reverse gradient updates, and its weight is only gradually smoothed up by the student model weight, ensuring the output characteristic stability.
It's like distilling beer, and the smaller the volume, the brighter it is.
IBOT camouflage predictions
In addition to being distilled, the iBOT module randomly covers a portion of the patch in the image, requiring the model to predict the corresponding characteristics of the patch according to the remaining patch that is visible. This process is similar to filling a void, forcing the model to actively learn the association between the local structure of the object, the edge continuity and the components of the object, rather than just remembering the surface color texture. This is also the key to the DINOv3 dense patch token that captures geometric information such as curves and outlines.
KoLeo Feature Spread, Gram-negative loss
KoLeo feature scattering and Gram-negative loss further constrain feature space, solving the problem of DINOv2: when image resolution increases, dense patch features tend to degrade, patch features in different spatial locations converge, and space differentiation is lost.
With these two constraints, DINOv3 still maintains the uniqueness of each patch's characteristics under high-resolution input, and patches that are adjacent but belong to different objects are not easily mixed together.
Axis rotation position encoding ((axis rotation RoPE))
At the network structural level, DINOv3 uses axis-spinning position coding (RoPE) to replace the previous DINO series of learnable absolute position coding.
The RoPE axis rotates the patch's spatial coordinate information directly into the characteristic vector. Regardless of the size of the input image, the model can understand the relative position of each patch in the image.
This feature is very important for robotic tasks: the close-up and cutting range of the camera images on board changes frequently, and the RoPE ensures the stability of spatial information and supports up to 4096×4096 inputs.
Reasoning
The model progresses to the reasoning stage, which is the process of entering a desktop photo with a patch token, where we no longer train to update the weight. Input images are split into fixed 16-pixel patches, each patch is first projected linearly, converted to a fixed-dimensional vector, and then sent to a multi-layer Transformer. The Transformer relies on a multi-faceted attention mechanism to allow each patch to exchange information with all the other patches in the image: the glass patch will interact with the adjacent rotation patch, the desktop patch generates interaction, understanding the boundary relationship between objects and the background.
Finally, two types of results are produced: a class token that represents the entire image abstract, and a full patch token that preserves the spatial grid structure.
DINOv3 architecture diagram, where AACM is an anomaly sensing calibration module in the AD-DINOv3 framework (Anomaly-Aware Calibration Module)
The most central characteristic of DINOv3 is dense semantic representation, which means that the image is divided into 16×16 patches, each patch with independent output characteristic vectors. These vectors prioritize encoding structural information such as the shape, curvature, outline, relative position of objects, and the disturbance caused by color, light, and texture changes.
This results in a characteristic structural integrity that differs from traditional supervisory training models.
Light illuminates dark changes, the surface texture of the object is worn out, and the patch characteristic similarity can be maintained as long as the object's geometry remains the same. When the object is grabbed, pushed, and the local geometry changes, the corresponding patch vector changes as well.
This is a characteristic of interchangeably-sensitive silicon, which meets the need for robots to constantly perceive changes in environmental conditions.
DINOv3 How to deal with physical and intellectual pain
The core of physical intelligence isIt's the first time I've ever been in a relationship with a woman.DINOv3 happens to target four traditional pain points:
The cost of collecting real robot data is high.
Object positioning, interaction status marking, simulation data and real-world scenarios vary widely, and the simulation training tactics are easily deployed directly to physical robots.
DINOv3 relies on mass Internet unlabeled images to learn general spatial structural knowledge beforehand, freeze the skeletal weight, require only a small amount of robot interaction data to minimize task heads, reduce the need for demonstration trajectories, and improve model cross-scenario generalization capabilities.
There is information redundancy in the pixel space.
Early world models directly predicted future bright RGB images, and the model consumed large amounts of computational power in rendering unrelated texture details.
DINOv3 provides a semantic sub-space in which world models no longer predict pixel graphics but instead predict changes in DINOv3 characteristics, concentrating computational modeling on object interactions and state changes, reducing computational expenses, and improving the ability to reason for physical factors.
The problem of visual distribution deviation.
Models such as the ResNet, based on supervised learning, are easily disturbed by color and texture. DINOv3 teaches to prioritize the extraction of information about the structure of objects, the impacts of weakened color texture, the stability of lighting and background characteristics while changing, improving the ability to migrate zero samples.
Space location modeling needs.
RoPE axis allows the DINOv3 patch token to associate itself with spatial coordinates, and the output density characteristic map can be mapped back to the original, facilitating the subsequent network to complete obstacle splitting, scraping point detection.
DINOv3 is a typical application direction in physical intelligence
It's the first time I've ever seen a robot in my life.
The visual action strategy receives camera images with the robot's own body sensing data, outputting the robot's joint action sequence.
DINOv3-Diffusion Policy is a representative program
DINOv3, which uses frozen or fine tuned DINOv3 as a visual encoder for Diffusion Policy, has a maximum success rate of about 10% compared to ResNet encoders for robot-operated benchmarks such as Push-T, Lift, Can, Square, etc. Smelting experiments show that even if DINOv3 is frozen, dense features still have excellent migration capabilities, improving the sampling efficiency and robustness of the spread strategy.
In grabbing detection tasks, the ViT-B/16 serves as a backbone, coupled with an anti-voltage sampling task head, that can output pixel-by-pixel grabbing confidence, grabbing angles, grabbing width charts. It can be used to predict grabbing positions using a single RGB image, for mechanical arm planning, and for stably performing centimeter-level counter positioning in a desktop object operating scene.
From a simulation scroll of 10,000 optimization steps, you can see that the DINOv3 model is always faster and simpler than the ResNet18 variant.
The world model in space.
World models are an important branch of physical intelligence, with the goal of allowing robots to simulate and execute post-action environmental changes for action pre-planning and risk assessment. In recent years, many world models have ceased to use RGB images as a predictive target, instead opting for DINOv3 feature spaces.
CLWM (DexWorldModel) The Causal Hidden Space World Model is a representative of this path: it aims to generate a DINOv3 characteristic that unravels the semantics of interaction with visual noise.
LiLa-WAM demonstrates the light feasibility of this route:
Frozen DINOv3 extracts visual subtractive characteristics and learns the causal model of movement to characteristic change. The model predicts the next moment DINOv3 dense characteristic graph, which simulates the changes in the state of the object after being pushed, grabbed, rather than generating images. This eliminates unrelated texture noise and reduces computation.
Most of the world's model scenes require only the tendency of objects to change position and state, without the need for pixel-level high-precision positioning, native DINOv3 dense patch features can meet the demand.
Robotic visual navigation
The image target navigation task requires the robot to move to the target location in an unknown environment based on a target reference image, which is part of a rough positioning task, and the scenario layout and geometric characteristics of DINOv3 are good enough.
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
The idea of AstraNav-Memory is very representative.
The program, based on the DINOv3 freeze, adds lightweight token compression modules, compresses visual features, inserts multi-channel historical scene information into context windows on large language models, and builds long-term scene memory. No explicit map is needed to integrate perception, memory, language instructions, and decision making into end-to-end systems.
Visual language is physically intelligent
DINOv3 dense patch token can be combined with visual language model VLM. DINOv3 is responsible for extracting the characteristics of finite particle space objects, VLM is responsible for higher semantic understanding and analysis of human natural language instructions.
Such scenarios are typically roughly positioned, requiring only the area where the target object is located.
Current restrictions
DINOv3 brings significant improvements to robotic visual perception, but it is not omnipotent, and there is still a noticeable shortcut.
The original DINOv3 belongs to the 2D visual representation and has no apparent three-dimensional geometric modeling capabilities.
While features imply some depth, spatial structural information, but not strictly three-dimensional reconstruction. Encountering object shadows, multi-layered object stacking scenarios, single RGB inputs generate spatial ambiguity, often requiring a combination of depth cameras or multiple view convergence. Even a 2D pixel-level counterpoint cannot be directly exchanged for a real three-dimensional spatial coordinate.
Large size DINOv3 (ViT-7B) is reasonably expensive.
Humane robots, mobile robots on board have limited edge computation, 7B versions are difficult to run in real time, and engineering landings are generally opted for with ViT-B/16, ViT-L/16, the cost of characteristic performance decline. Model lightweighting, quantization, scant attention is an important optimization direction.
DINOv3 pre-training data is primarily based on Internet static images and lacks dynamic robot interaction data.
Models are good at identifying static objects and scenic structures, but they are not familiar with the laws of physical dynamics such as friction, collision, and soft transformation of objects. World models require additional robot interaction data sets to learn the rules of physical change in DINOv3 feature space.
Extreme scenarios are easy to fail.
Samples such as transparent objects, mirror reflections, are low in pre-training data, and the patch characteristics of these objects are prone to drift and need to be adapted online.
Summary
DINOv3 provides a general-purpose visual description language with low-cost, high-level, and spatial information:
Robots don't have to train to identify models for each type of object individually, nor do they have to use computational rendering pixels, but instead analyze characteristic spatial changes in the object's state.
It has become the mainstream visual backbone of robotic operations, navigation, and world model research, but it is only a basic component of the perceptual layer, and the entire system also needs synergies of time-order memory, causal reasoning, physical dynamics, and motion control.
The native characteristic can only determine approximate position; the hanging task head can achieve 2D precise position within the image plane; the high-precision position in three-dimensional space must be supplemented with depth or three-dimensional sensing modules.
With the advancement of lightweight, timed modeling and three-dimensional integration, robot schemes based on this type of self-monitored expression are expected to gradually move from simulation experiments to real-world scenarios.
Editor's note: Coffee fish
I'm not sure what I'm doing.