分享好友 Global News首页 Global News分类 切换频道

Weitu AI Describes a Streaming Video Model Designed for Persistent Memory

2026-11-16 13:3010RoboInd Editorial TeamWeChat Official Account

Before going out wearing AI glasses, ask the AI ​​"Where did you put the keys just now?"; after a meeting, ask the AI ​​assistant to review "what changes have been made to the plan on the whiteboard compared to the original one"; when a maintenance technician wearing a wearable camera device was performing maintenance, he was reminded by the video brain that "the action of closing the valve just now was not completed, please check again." These daily life and industrial needs require AI to connect the scene in front of it with previous content. Weitu AI says its streaming multimodal video model is designed to help camera-equipped devices interpret their surroundings continuously and turn observations into memories that can support later recall and reasoning.

Weitu AI, a Hong Kong multi-modal artificial intelligence company, currently serves dozens of enterprises around the world and approximately 6 million individual users around the world. The team hopes to further bring multi-modal technology into daily life, office and industrial scenarios, so that multi-modal devices in various scenarios have the ability to continuously perceive, reason and make decisions.

According to the Weitu AI team, the core problem to be solved by the streaming video brain is to efficiently understand the changes in the scene as the video continues to arrive, continuously generate memories, and achieve complex reasoning and decision-making through real-time understanding and continuously updated memories. For users, this means that with the application of such streaming video brains, all various multi-modal devices that need to constantly perceive the environment and interact will first usher in the ChatGPT moment of the physical world.

Efficient learning architecture of space-time compression brings continuous multi-modal perception

In reality, videos are continuously generated, but the computing resources and memory capacity of the model are very limited. If the entire frame is retained for a long time and repeatedly involved in calculations, the processing burden will be increased; excessive frame extraction or only a short time window may be retained, and key timing changes may be missed. For example, the action of "the key is put into the side pocket of the schoolbag" only lasts for a moment. This detail becomes particularly important when the user is looking for the key ten minutes later.

Weitu AI creates a video-native multi-modal architecture. Such an architecture enables efficient compression and perceptual learning of streaming video signals. The design idea is to center the calculation around the current changes that need to be understood. When building a video-native representation and memory mechanism, the team built a Dual Sparse Memory (dual-layer sparse memory) learning architecture: first automatically filter out relevant clips from the time dimension, and then extract key spatio-temporal features from within the clips to avoid a large amount of irrelevant picture information occupying computing resources for learning.

The first step is temporal block-level sparse learning, which is temporal compressed sensing. Faced with retrospective questions such as "What was written on the whiteboard at the beginning of the meeting?", the model first focuses on the relevant historical time segments, greatly reducing the scope of subsequent calculations. The second step is sparse learning in the spatial domain: after entering these fragments, the model automatically focuses and extracts key visual details (such as the main content of the picture and its changes), rather than allowing all the information of the entire scene to participate in calculations with equal intensity.

This mechanism puts "when you saw it" and "which details are useful" into the network for learning at the same time. The goal is to control attention computation and cache overhead while retaining key features, making continuous video understanding more suitable for various hardware devices that run for a long time. For example, for various hardware systems, computing efficiency is directly related to response experience, deployment costs, and the feasibility of device-cloud collaboration solutions.

Constantly updated memory aids complex reasoning

Efficient compressed sensing and computing solve the problem of continuous understanding, while the continuously updated memory module can help the model solve more complex reasoning. In addition to understanding the current picture, Weitu AI's model also includes a Long-term Memory Module, which compresses earlier visual and spatiotemporal information into compact representations and continuously updates them as new pictures arrive.

This type of memory carries historical information that can be used by the model, including the position changes of objects, the sequence of actions, and important states in the scene. The model must not only remember that "the key was on the table", but also retain the information that "later was put into the side pocket of the bag" with time and space changes. For example, in an industrial scenario, such a capability could be used to understand the steps of past operational demonstrations.

Schematic representation of the memory architecture of the streaming video brain

The Weitu AI team introduced that the team views memory as a selected and compressed intelligent representation. The memory module needs to learn not only the compressed sensing of current information, but also information that can support future understanding and reasoning. Just like when humans think, they often find memory as information for reasoning through recall. This provides inspiration for the memory design of video models.

For example, when a user asks "Are keys and books put together?" the system needs to associate the locations of items seen at different times with storage actions, and then determine whether they are placed in the same location. This engages memory in complex reasoning and judgment. Compressed memory cannot retain all the details of the original video. The model needs to learn and continuously update within a limited capacity, so that the memory module can participate in subsequent complex reasoning as much as possible.

Not only that, the Weitu AI team also explored new scenarios of streaming video brain, such as prediction and judgment of video status, and making accurate predictions of the status of ongoing events or actions. The Weitu AI team mentioned, “For example, when the AI ​​system is watching the coffee brewing process, it needs to accurately distinguish between “water is being poured” and “this step has been completed” in order to answer questions such as the order of steps and whether the action is completed. Incorporating state changes and time relationships into memory helps to understand how an event or action occurs and when it enters the next stage. This type of problem is also a very interesting new issue in research, that is, how to predict the time boundary of unfinished actions, which is equivalent to predicting the future state.”

Showing Strength in Streaming Video and Memory Measurements

Recent evaluations show that Weitu AI's streaming video brain demonstrates significant architectural advantages in the field of continuous video understanding. On OVBench [1], an authoritative benchmark that examines the three major time dimensions of "past memory, current perception, and future prediction" and a total of 16 subtasks, the comprehensive score of Weitu AI Streaming Video Brain reached 69.9, which is 11.1 higher than the industry's well-proven general multi-modal strong baseline Gemini 3.1 Pro (58.8); at the same time, it is highly close to the Meta S-EMBER benchmark [2] that continuously perceives scenes with wearable devices such as AI glasses. (This test uses first-person perspective daily video, requiring the model to answer only based on what it saw before the time of the question), the Weitu AI Streaming Video Brain achieved a score of 58.23, which was also 3.82 higher than the baseline-aligned Gemini 3.1 Pro (54.41).

These results show that Weitu AI’s model demonstrates superiority on both streaming video understanding and memory-based benchmarks. The product value brought by this can allow more AI devices to better understand changing scenes and make reasoning and decisions through the current picture combined with memory.

The two examples in the figure demonstrate the model's "spatial and temporal memory" question-answering capabilities: in continuous video streams, the model can respond to user questions at any time, integrate current visual information and related historical memory images, and accurately answer questions.

Let continuous perception and memory enter more application scenarios

The application space of streaming multi-modal brain covers various devices and products that need to continuously understand the environment and respond. From AI glasses, wearable cameras to other visual terminals, this set of capabilities is expected to be integrated into daily life, office, education, retail and industrial scenarios, providing users with immediate help closely related to current activities.

For example, when learning origami or cooking, the AI ​​assistant can combine the steps that have been completed to prompt the next step; when shopping, the product in front of you can be compared with the styles, specifications and labels that have been seen before; after the office demonstration, it can help the user review the operation sequence and plan changes; during industrial maintenance, it can combine previous operations and the current equipment status to prompt links that require further inspection. These scenarios share the same ability requirement: connecting current observations with historical memory, understanding the progress of the task, and providing a basis for subsequent judgments.

Such capabilities can also be applied to video creation. In the process of continuous video generation and editing, the model can continuously evaluate the fit between the work and the creative goals based on the current scene, creation progress, and previous content, put forward optimization suggestions around shot splicing, narrative rhythm, and visual style, and dynamically plan subsequent generation, editing, and modification steps. By integrating continuous perception, memory and reasoning into the creative process, it is expected to reduce the time and computing costs caused by repeated generation and modification, making the work more in line with the creator's expressive intention, while maintaining content coherence and style consistency.

Provide a portal for continuous perception of the world for general large models

Advances in the capabilities of general-purpose large models are expanding the space for AI to participate in complex tasks. OpenAI's GPT-6 series has made progress in complex task processing, computer operation and cost efficiency; Anthropic, in the release of Claude Fable 5.2, demonstrated the ability of model-controlled robots to complete tasks such as unscrewing bottle caps and drawing.

According to Weitu AI, as the reasoning and task planning capabilities of general models increase, how to perceive environmental changes efficiently and in real time and bring relevant historical information into current judgments will become an important foundation for robot applications. For example, when a home robot is looking for items or tidying up a room, it needs to continuously confirm whether the items have moved, whether the operation is completed, whether the task conditions have changed, and adjust its actions based on previous observations. Continuously updated perceptions and memories can provide the necessary context for these judgments.

Weitu AI hopes to promote its streaming multi-modal brain to become a portal for various general-purpose large models to provide continuous perception of the world: the video brain continuously processes images, tracks scene states and updates memories, providing timely, compact and task-relevant visual and multi-modal information for complex reasoning and task planning. This is equivalent to providing a pair of "eyes" that can continuously observe and retain relevant memories for large general models, which is expected to reduce the burden of repeatedly processing lengthy visual storage.

This positioning also gives the streaming multimodal brain two types of application value: it can directly support specific scenarios, or it can be used as a perception and memory module to collaborate with other general models. In the future, when this set of capabilities is further combined with the robot's motion control and execution system, it is expected to help the robot continue to understand the task process in home, office and service environments, and adjust subsequent actions according to environmental changes.

Starting from video, integrating more modal information

Weitu AI introduced that the streaming video brain is the starting point for the team to explore multi-modal interactive intelligence. In the future, the team hopes to continue to integrate more modal information on this model architecture so that perception, memory and reasoning capabilities can adapt to the needs of different devices and scenarios.

For example, in industrial scenarios, video can be combined with sensor signals such as temperature, pressure, and vibration to jointly reflect equipment operation and environmental changes. Different signals provide complementary information: the screen shows the operating process and appearance changes, and the sensors reflect temperature fluctuations, pressure changes or vibration abnormalities. Models need to correlate this information in time and space, understand the connections between them, and form memories that can be used for subsequent judgments.

Continuously sensing changes, compressing important information into spatio-temporal memory, and then combining current perception and historical memory for reasoning and decision-making are the core ideas of Weitu AI in building a streaming multi-modal brain. From today's cameras, AI glasses and wearable devices to future robots and industrial multi-modal systems, the team hopes to use the scalable model architecture to enable more devices to have the ability to continuously understand the environment, and also provide general large models with an entrance to quickly and accurately perceive the world.

[1] Huang Z, Li X, Li J, et al. Online video understanding: Ovbench and videochat-online [C]//2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025: 3328-3338.

[2] Wang X, Zhao X, Rodriguez P, et al. S-EMBER: A Large-Scale Benchmark for Streaming Egocentric Memory Retrieval [J]. arXiv preprint arXiv:2607.02689, 2026.


点赞 0
举报
收藏 0
分享 1