ARTIFICIAL INTELLIGENCE
Gemini Robotics: From Perception to Action
Gemini Robotics ER 2 separates embodied reasoning from motor control. Its 2026 documentation reveals the practical work between seeing a scene and acting.
Research checked
Newest cited publication
A Universal Robots industrial manipulator illustrates physical robot control. It is background imagery rather than a photograph of the Gemini Robotics system.
Auledas · Image source · CC BY-SA 4.0
A robot can correctly describe a mug on a shelf and still fail to pick it up. The gap includes coordinate conversion, reachability, collision avoidance, grasp selection and recognizing whether the grasp worked. Gemini Robotics is interesting partly because its public interface makes some of those boundaries visible. As of the October 18, 2026 cutoff, Google's current developer documentation identifies Gemini Robotics ER 2 as the latest embodied-reasoning model, with a model card published July 30. The April ER 1.6 announcement is already historical context, not the current endpoint. Reading this family carefully means distinguishing a model that reasons about an action from the software and hardware that physically execute it.
Embodied reasoning is one layer in a physical system
The ER 2 model card describes a vision-language model based on Gemini 3.5 Flash. Its inputs can include text, images, video and audio, while its output is text. The developer overview describes structured spatial outputs, tool use and orchestration of robot functions or vision-language-action models. There are standard and streaming preview endpoints, with different interfaces and capabilities. These details prevent an easy category mistake: generating a coordinate or requesting a tool call is not the same thing as supplying a validated, continuously updated motor command.
An independent analogy is a person directing a crane operator. Saying which beam should move, deciding a useful destination and checking the result are substantial tasks. They do not replace the operator's low-level control or the crane's physical constraints. In a robot application, those responsibilities may be distributed across models, planners, controllers and sensors. The division is useful because each component can have a clear contract. It becomes dangerous when a plausible sentence is treated as proof that all the downstream contracts have been satisfied.
A current article needs a current model boundary
Google's April 14, 2026 ER 1.6 release emphasized pointing, multi-view understanding, success detection and instrument reading. The company also explained that some evaluations used agentic vision while others did not, and that single-view and multi-view success-detection sets contained different examples. By October, the API overview points developers to ER 2 and states that the ER 1.6 endpoint was deprecated on August 31. Those are distinct stages of the product and research history. An attractive April demonstration should not be silently relabeled as an ER 2 experimental result.
This is a broader problem in fast-moving robotics coverage. A model name, a system demonstration and an available endpoint can evolve on different schedules. A technical reader should record all three. If a reproduction changes the endpoint while retaining an old prompt and evaluation script, a difference could come from the model, its interface or its defaults. Good release documentation makes migration practical; good scientific comparison freezes the relevant version long enough to understand what actually caused an improvement.
Spatial output needs a coordinate contract
The developer overview illustrates object pointing with normalized image coordinates and labels. This creates a convenient interface for connecting perception to an application. It does not by itself establish metric distance. A point in a camera image identifies a viewing direction, and recovering a position in the robot's working space requires additional geometric information or assumptions. The documentation's examples therefore belong to a larger integration process, even when the model returns perfectly valid structured output.
For an independent calculation, take a hypothetical image 1,200 pixels wide and 800 pixels high. A point halfway across and one-quarter down corresponds to pixel coordinates x = 600 and y = 200. If an interface instead expects y followed by x, swapping the entries produces a different location while preserving valid numbers. Normalization removes dependence on pixel dimensions, but it does not remove axis conventions. Crop offsets and image rotation add more opportunities for error. Before evaluating semantic intelligence, a developer needs to verify that a known marked point survives the entire transformation from camera image to robot command.
A tutorial demonstrates sequencing, not physical success
Google's task-orchestration guide uses a mock robot interface with movement and gripper functions. The model receives an image and tool definitions, requests actions, receives function results and continues the sequence. The example is explicitly a mock interface: its functions print messages, and the loop returns success status to the model. This is a useful demonstration of communication and task decomposition. It is not an experiment establishing that a real arm picked up a real block, avoided a collision or detected a failed grasp.
The example exposes an integration question worth taking seriously. In a physical system, a function returning successfully may mean only that a command was accepted. A grasp may still fail because the object slipped. A robust application distinguishes command acceptance, completed motion and achieved task state. It can ask sensors for evidence at those boundaries rather than allowing a single success string to stand for all three. That design principle follows from inspecting the interface; it is not a claim that the tutorial was intended as a production controller.
Success detection should earn its place in the loop
Imagine an illustrative task in which a robot moves a sponge into a tray. A useful evaluation changes the sponge's color, shifts the tray, obscures part of the view and includes an empty-gripper failure. The system should determine whether it needs another observation, a recovery action or a stop. A final image alone can be misleading if the object is hidden behind the arm. Multiple views and intermediate observations can add evidence, but their value must be measured against the additional latency and complexity they introduce. Recording when each observation was captured also helps distinguish an incorrect judgment from a correct judgment about a scene that has since changed.
The ER 2 model card explicitly limits acceptable use in safety-critical settings and describes known limitations through its underlying model documentation. Those boundaries matter alongside the capabilities. A high score on a visual reasoning benchmark cannot certify an entire robotic workcell. Physical validation also depends on controller behavior, reachable space, force limits and how uncertainty changes decisions. This article is explaining a research and developer interface, not offering deployment instructions for systems where a mistaken action could injure someone.
Evaluate the complete interaction, including waiting and recovery
The ER 2 overview describes latency and performance tradeoffs, possible hallucinations and dependence on input quality. A practical comparison should therefore record elapsed task time and recovery work as well as the rate of successful final states. A model that reasons longer may improve a difficult judgment while delaying an already-moving robot. A streaming interface may reduce communication friction without eliminating the need to synchronize observations with the physical state. These are measurable properties of the integrated application, not answers supplied by the model's name.
My interpretation is that the most important change is a clearer route for placing multimodal reasoning inside an observable action loop. The strongest next evidence would compare matched tasks, hardware and sensing while separately measuring perception mistakes, planning mistakes and execution failures. It would preserve the cases that required a human intervention. Gemini Robotics ER 2 makes the reasoning layer available through a current 2026 interface. Whether that layer creates a more dependable robot remains something a carefully instrumented system must demonstrate, one complete interaction at a time.
Sources & further reading
Primary papers, project documentation, and research releases used in this article. Publication dates below refer to the sources themselves.
- Gemini Robotics ER 2 — Model CardGoogle DeepMindPublished Checked
- Gemini Robotics ERGoogle AI for DevelopersChecked
- Task orchestrationGoogle AI for DevelopersChecked
- Gemini Robotics-ER 1.6: Powering real-world robotics tasks through enhanced embodied reasoningGoogle DeepMindPublished Checked
FOLLOW THE QUESTION
Study engineering foundations