ARTIFICIAL INTELLIGENCE
Segmenting Video with a Memory
SAM 3.1 makes tracking many objects more efficient. Its March 2026 release shows why segmentation quality, identity and computation must be evaluated together.
Research checked
Newest cited publication
A visual-network diagram introduces learned image features; the video-memory and segmentation mechanisms discussed here require additional components.
Renanar2 · Image source · CC BY-SA 4.0
A person walks behind a pillar and appears on the other side. For a human watching the video, the interruption is ordinary. For a segmentation system, it creates several different questions: which pixels belong to the person now, whether this is the same person as before, and whether the person still matches the requested description. The Segment Anything research line increasingly treats these as connected problems. The newest verified release examined here is SAM 3.1, published on March 27, 2026. Its shared approach to tracking many objects adds an engineering development to the more visible advance of SAM 3: asking for a category in words and following the matching instances through a video.
From one clicked object to a visual concept
SAM 2, introduced in August 2024, established streaming memory as a practical ingredient for promptable video segmentation. A point, box or mask identifies a target, and information retained from earlier frames helps produce later masks. SAM 3, whose paper first appeared in November 2025 and was revised in March 2026, expands the instruction interface. A short phrase or visual example can request the objects matching a concept. Its detector and memory-based tracker share a visual backbone, while a presence component separates recognizing a requested concept from locating its individual instances.
Consider a fictional recording of a crowded bicycle rack. Clicking one bicycle and asking for every bicycle create different tasks. The first needs a persistent identity; the second also needs an inventory. A rider entering the scene may add another bicycle, while a person wearing a bicycle-shaped print should not necessarily count. This distinction explains why attractive colored masks alone reveal little about whether an instruction was followed. The system needs a definition of membership, and the evaluator needs to check both omissions and unjustified additions. A correct boundary around the wrong object remains a wrong answer.
Memory helps continuity and can preserve mistakes
The useful role of memory is easiest to see when an object changes appearance. A turned head, temporary shadow or partial occlusion can make frame-by-frame matching unreliable. Previous observations provide additional evidence about what is being followed. Yet persistence has a cost: a mistaken association can also become persistent. If two similar cyclists cross, a tracker can preserve a smooth-looking sequence while quietly exchanging their identities. The masks may remain plausible in every frame, even as the history becomes incorrect.
That suggests an evaluation exercise beyond checking a handful of screenshots. Construct a short clip with an entrance, a crossing, a full occlusion and a return. Mark the identity before the crossing, then examine the assignment after the return. Repeat with the requested concept absent entirely. These are proposed diagnostic tests, not results reported for a particular SAM checkpoint. Their purpose is to separate boundary quality, continuity and rejection. Applications have different tolerance for each error. A casual visual effect might tolerate a brief slip; an animal-counting study could turn the same slip into a false claim about population size.
The 2026 change is also about the cost of many targets
The official SAM 3.1 release notes describe Object Multiplex: objects are placed into fixed-capacity buckets and processed jointly using shared memory. The preceding pipeline processed tracked objects independently, so the object-dependent work grew with their number. The release also reduces synchronization overhead, improves compilation support and batches parts of the computation. Meta reports about sevenfold acceleration for 128 tracked objects on a single H100 graphics processor compared with its November 2025 release. That is a specific workload and device comparison, not a universal speed multiplier.
An independent analogy is a workshop receiving many identical inspection forms. Reading the shared instructions separately for every form wastes effort. Reading them together can save work, but only if the worker still records each item's answer separately. Shared computation in a vision system faces an analogous distinction between reusing information and losing individuality. For deployment, the useful question is therefore a curve: how do latency and memory change as targets are added? One headline measurement cannot reveal the cost of one target, the behavior near a bucket boundary or the delay introduced by preprocessing.
Faster does not mean every accuracy score rises
The release notes themselves contain a useful qualification. Their concept-segmentation video results are mixed across datasets, while conventional video-object segmentation improves on six of the seven listed benchmarks. For example, the displayed video concept results improve on YT-Temporal-1B but decline on some public benchmarks. This prevents a simplistic reading in which the version number guarantees uniform superiority. The original SAM 3 paper introduces its own concept-segmentation benchmark, SA-Co, while the official repository provides implementation and checkpoint instructions for reproducing the model family.
A benchmark is a sampling decision. If a collection emphasizes short clips and familiar object categories, a model can succeed there while struggling with a distant camera observing tiny insects over an hour. Conversely, a difficult occlusion dataset may say little about language ambiguity. A responsible comparison states the prompts, object counts, resolution, hardware, inclusion rules and scoring method. It should also report the correction work required after inference. A tool that saves ten minutes of computation but creates twenty minutes of manual repair has shifted the cost rather than removed it.
A useful workflow keeps a human correction path
Suppose a laboratory wants masks for reusable containers moving along a conveyor. An illustrative workflow would begin by checking a small, deliberately varied sample: empty belt, reflective lid, stacked containers, motion blur and an unfamiliar container shape. Reviewers would define what counts as a separate instance before expanding to a larger collection. They would record corrections instead of silently treating model output as ground truth. This is an operational proposal, not evidence that SAM has already been validated for that laboratory or industry.
The distinction matters when segmentation becomes training data for another model. Uncorrected omissions can teach the next system that partially visible objects do not exist. Consistently merged masks can erase the distinction between a container and its lid. Good annotation practice therefore stores the original image, the prompt, the checkpoint and the correction history. It also keeps a held-out sample that was never used to tune prompts. Otherwise the same examples gradually become both the development set and the proof of success, making an apparently improving workflow difficult to trust.
The next test is sustained, measurable usefulness
The interesting trajectory from SAM 2 through SAM 3.1 combines a richer request language with more efficient temporal processing. Neither contribution makes the meaning of a request automatic. Words such as damaged, usable or dangerous can depend on hidden properties that pixels do not establish. A segmentation model can select a visible region associated with a phrase without verifying every real-world property implied by that phrase. Its output remains a visual prediction, not a complete inspection report.
The strongest next evidence would measure complete tasks over long recordings: missed instances, identity exchanges, human corrections, end-to-end latency and the stability of those quantities under changed cameras. An open implementation makes that kind of work possible; it does not supply the answer in advance. SAM 3.1 is valuable because it makes a crowded scene less computationally awkward while retaining the concept-based interface. Understanding that specific achievement is more useful than interpreting any colorful overlay as proof that a machine understands everything it has outlined.
Sources & further reading
Primary papers, project documentation, and research releases used in this article. Publication dates below refer to the sources themselves.
- SAM 2: Segment Anything in Images and VideosMeta FAIR / arXivPublished Checked
- SAM 3: Segment Anything with ConceptsMeta / arXivPublished Checked
- Release Notes: SAM 3.1 — March 27, 2026Meta / GitHubPublished Checked
- SAM 3: Segment Anything with Concepts — official repositoryMeta / GitHubChecked
FOLLOW THE QUESTION
Study engineering foundations