Researchers from eight institutions including Duke, Google, USC, UPenn, and UNC jointly released a systematic review of the latest vision-language model (VLM) memory mechanisms. Based on the new 4D classification framework, the authors surveyed more than 300 related papers and nearly 70 evaluation benchmarks, covering eight hot application scenarios such as long video understanding and generation, multi-modal RAG, embodied navigation and action, and personalized assistants.

- Paper: “Memory in Vision-Language Models: Taxonomy, Mechanisms, and Applications”
- Authors of the paper: SHAO-JUN XIA, YIZHUO HE, YUNER ZHANG, YIFAN JIANG, XIAOYANG CHEN, LIANGXI LIU, CHENWEN LUO, JINBAO WANG
- Project contributors: Yifan Jiang, Yizhuo He, Yuner Zhang and Shao-Jun Xia
Project link:https://github.com/Xzcv-hub/Awesome-Memory-in-VLM
- paper :
1. Why does VLM need memory?
In recent years, VLM has developed rapidly in tasks such as image description, visual question answering, video understanding, multi-modal dialogue and embodied intelligence, and has gradually expanded to more complex, large-scale long-range tasks and real applications. These tasks require the model to continuously accumulate and call upon past information over a longer time span and more rounds of interactions. Simply expanding the context window is often difficult to meet this demand. To this end, researchers have used different methods to try to introduce memory mechanisms into VLM, such as external memory banks, hierarchical memory, long and short-term memory, etc.
However, existing designs mostly start from specific task scenarios and architectures, and the implementation and optimization of memory often focus on task-related dimensions. The terminology and designs vary, and there is a lack of a unified perspective to understand and compare these methods. On the other hand, most of the existing memory-related reviews focus on LLM, and the description of VLM only includes specific task forms, which has certain limitations.
The review proposes a new four-dimensional framework for systematically describing memory mechanisms in VLMs. Using this framework, the authors surveyed more than 300 related papers, summarized eight typical memory forms, compiled nearly 70 evaluation benchmarks, analyzed the role of memory in eight application scenarios such as long video understanding, embodied intelligence, and personalized assistants, and discussed the current challenges and future directions.

Figure 1 | Development timeline of VLM memory-related research. The figure shows representative research and its development context from four dimensions: When (time range), Where (storage location), What (memory content) and How (memory operation). The development nodes of the main basic models in the same period are marked in the middle.
2. Core Framework: 4D Memory Classification

Figure 2 | 4D classification framework of VLM memory: When (time range), Where (storage location), What (memory content), How (memory operation).
The author proposes that the perspective of memory should be expanded from a simple information storage unit to the entire life cycle: relevant information is first observed from multi-modal tasks and written into memory, is repeatedly refined and organized over time, and is retrieved and recalled when processing related tasks. Based on this perspective, the author proposed a 4D classification framework to systematically characterize the memory mechanism in VLM from four dimensions: When (time range), Where (storage location), What (memory content) and How (memory operation).
1) When: How long the memory is retained. This dimension is concerned with the duration of memory. The shortest transient memory is used for single reasoning, and episodic memory is suitable for multi-turn conversations and multi-turn interactions. The memory of both is usually retained only during the current reasoning or task interaction. The longest lasting memory will remain after the session ends and can be used for subsequent interactions.
2) Where: Where does the memory exist. There are two types of memory storage locations: internal and external of the model. Internal memory is often stored in the parameters, hidden states and KV cache of the model. This type of memory is fast to recall and more closely integrated with inference tasks, but is not conducive to modification and viewing. External memory is stored in an independent database, knowledge graph or experience library for easy updating and review, but the effect depends on the quality of retrieval.
3) What: What is the memory stored as? This dimension involves the form in which memory stores original relevant information. In addition to direct original information fragments, memory can also be organized into structured dialogue and trajectory records, abstract semantic knowledge, or compressed into potential representations within the model.
4) How: How does memory operate. This dimension is concerned with the operations that the memory undergoes throughout its life cycle, and mainly includes four categories: writing, which determines which new information is written into the memory in what form. Consolidate, organize and refine existing memories. Retrieval, finding memories related to the current task to assist in completing the task. Update and forget, update and clean information in continuous interaction to ensure the timeliness of information.
3. Eight typical memory forms

Figure 3|Eight typical memory forms and their positioning in the 4D framework.
The four dimensions can be discussed conceptually separately, but are often bound to each other in real systems. Choosing how long to keep a memory often determines where and in what form it is better suited to be stored. Based on this, the author summarized eight typical memory forms (Figure 3), each form corresponding to a typical combination of four dimensions. For example, long context memory retains tokens or their compressed representations through context windows, and is usually a transient, internal memory; lifelong agent memory requires continuous accumulation of experience in multiple sessions, and is a persistent memory. Accordingly, memory systems can be designed around five operations: writing, consolidation, retrieval, updating and forgetting.
It’s important to note that these combinations are just common design tendencies, not fixed rules. For example, although the instantaneous intermediate state cannot be retained after the reasoning is completed, its content can be organized by additional components and turned into a scenario or persistent memory, and assist in subsequent tasks. The authors hope that these combinatorial relationships will not only help researchers clarify the preferences of existing work in memory implementation, but also point out design spaces that have not been fully explored.
4. Looking at memory from the perspective of application: eight typical scenarios

Figure 4|Memory demand preferences in eight types of application scenarios.
The eight memory forms answer "how memory can be realized." In practical applications, which form to choose and how to combine them largely depends on the task itself. From an application perspective, the review further organizes nearly 70 existing VLM memory benchmarks and discusses the memory implementation and demand preferences of eight types of application scenarios:
1) Long video understanding and generation: Limited by the context of the model, which cannot accommodate massive video frames, memory implementations mostly focus on compressing visual information and selectively retaining task-related visual information.
2) Image reasoning and description: In this task, the model easily forgets important information during long reasoning, and memory implementation focuses on retaining key visual information.
3) Continuous learning: The core of this type of task is that the model retains its previous ability when learning new knowledge and skills. Memory is usually reflected in the protection of the model's original parameters and representation space.
4) Embodied navigation and action: Memory implementation mainly focuses on recording action trajectory data, task progress and environmental spatial structure.
5) Multi-modal RAG: This type of task mainly examines the construction of external knowledge bases and document images, as well as the efficiency and accuracy of retrieving relevant information according to the task.
6) First-person vision: This type of task involves long-term information lasting days or even weeks, and hierarchical organization of memory is often an important solution.
7) Personalized assistant: Since the user's exclusive requirements and preferences need to be saved across sessions, memory implementation is mostly based on external memory that can be updated at any time.
8) Virtual environment interaction: This type of task examines whether the agent can refine past interaction experiences into reusable knowledge and skills, and retrieve references when encountering new tasks.
5. Challenges and future directions
At the end of the review, the author sorts out the challenges that the memory mechanism of VLM still faces and the future directions worth exploring. First of all, longer context does not mean that the memory is effective. Even if the model receives more information, it may miss a short but critical event. Secondly, the importance of information often depends on what will be asked later. This lag makes it difficult to choose between compressing and retaining relevant memories. Existing methods often lose event sequence and visual details while retaining high-level semantic information. In addition, the retrieved memories themselves may be irrelevant, outdated, or even wrong, and models often lack the ability to verify these memories and give seemingly reasonable answers even if they are based on erroneous information, making errors difficult to detect. Finally, most existing benchmarks only test a single ability of memory, while errors in real task scenarios often involve multiple links, and a single test cannot fully reflect the performance of the memory system.
In response to these problems, the authors identify several directions for future research: drawing on human cognition to build a hierarchical memory structure, allowing the system to adaptively select storage methods based on the type of information, allowing the model to learn to judge what is worth remembering, and allowing each memory to be traced back to the original visual evidence. At the same time, the evaluation of memory systems should not only focus on accuracy. Whether users can view, modify and delete their own memories should also be considered at the beginning of the design.








