SemStream: Semantic Boundary-Aware Memory for Streaming Video Understanding

ACM MM 2026
Zixuan Tan, Mingjin Kuai, Zheqi Lv✉, Hao Fei, Wei Ji✉
Nanjing UniversityZhejiang UniversityNational University of Singapore
✉ Corresponding Authors
PaperComing soon CodeComing soon

SemStream replaces heuristic feature similarity with a learned Semantic Continuity Controller for dynamic event partitioning and a Budgeted Episode Merger for long-term memory consolidation. It achieves 78.6% on StreamingBench, 74.1% on MLVU, and 62.3% on ODV-Bench.

Comparison of cosine similarity and the proposed semantic continuity score

Abstract

Processing continuous video streams requires multimodal large language models (MLLMs) to maintain temporally coherent memory. Existing architectures predominantly rely on heuristic feature similarity, such as cosine similarity, for memory updates, making them blind to high-level semantics and susceptible to catastrophic fragmentation from camera ego-motion. To overcome this, we construct SemBound-1K, a multi-dimensional soft-labeled dataset that factorizes video boundaries into visual, action, and subject transitions while explicitly penalizing camera noise. Building upon this foundation, we propose SemStream, an intent-driven streaming architecture realized through semantic boundary-aware memory management. SemStream replaces heuristic matching with a learned Semantic Continuity Controller (SCC) for dynamic event partitioning, and employs a Budgeted Episode Merger (BEM) for long-term memory consolidation under strict token constraints. Extensive experiments demonstrate that SemStream achieves state-of-the-art performance, with accuracies of 78.6% on StreamingBench, 74.1% on MLVU, and 62.3% on ODV-Bench. By natively modeling semantic transitions, SemStream effectively prevents memory fragmentation, enabling robust long-horizon reasoning in complex, unconstrained real-world environments.

Overview of SemStream

SemStream organizes incoming visual tokens through three cooperative components: Boundary-Governed Hierarchical Memory, the Semantic Continuity Controller, and the Budgeted Episode Merger.

Overview of the SemStream architecture

The hierarchical memory maintains recent, transition, and episode buffers. SCC predicts multi-dimensional semantic continuity scores from adjacent frames to detect event boundaries, while BEM consolidates historical episodes under a fixed token budget.

SemBound-1K

SemBound-1K models event transitions with continuous, semantically factorized, and explicitly noise-aware supervision.

Three-stage construction pipeline of SemBound-1K

Candidate events are extracted from raw GEBD annotations, semantically relabeled along visual, action, subject, and camera dimensions, and cleaned through quality control to form the final dataset.

Raw typology of event boundaries
SemBound-1K multi-dimensional boundary typology
Distribution of visual, action, subject, and camera semantic scores

Experimental Results

SemStream is evaluated on general streaming, long-video, and autonomous-driving video benchmarks.

Table 1: Comparison on streaming and long-video benchmarks

SemStream establishes state-of-the-art results on StreamingBench, OVBench, OVO-Bench, and MLVU while operating online at 1 frame per second.

Table 2: Results on ODV-Bench

On ODV-Bench, SemStream achieves the best overall score and the best average results on Static Target and Event Oriented tasks.

Ablation Studies

The ablations isolate the contributions of semantic boundary modeling, budgeted memory consolidation, and multi-dimensional supervision.

Tables 3 and 4: SemStream ablation studies

BibTeX

Acknowledgements

This work is supported by State Key Laboratory of Novel Software Technology, Nanjing University (KFKT2025A22).