ARTIFICIAL INTELLIGENCE
From tau2 to tau3: reliable tool conversations
Tau’s 2026 knowledge and voice evaluations expose the gap between fluent support conversations and consistent, policy-correct changes to shared application state.
Research checked
Newest cited publication
An agent–environment diagram supplies background for tool use and feedback; the benchmark also imposes task-specific conversational rules.
Megajuice · Image source · CC0 1.0
Customer service can sound successful while leaving the underlying problem untouched. An agent may explain a refund clearly, misunderstand which purchase qualifies, and then change the wrong record. The tau benchmark family asks whether conversational systems can connect language, policy, and tools reliably enough to complete a task. Its 2025 dual-control extension added another complication: users can change the world too. In March 2026, tau3 expanded the research program toward knowledge retrieval and simultaneous voice interaction. A July grading repair then demonstrated why evaluator versions matter. These developments make a useful case study in agent reliability, provided their simulated environments and dated results are kept distinct from real customer outcomes.
Measure the state after the conversation
The original tau-bench paper, submitted June 17, 2024, evaluated conversations between an agent and a language-model-simulated user, with domain tools and policy guidelines. It checked the resulting database state against an annotated goal. It also introduced pass^k, a measure emphasizing success across repeated trials. The idea differs from rewarding a system for producing one good attempt among many. That distinction is central when the intended product handles requests from individual people who cannot choose retrospectively which conversation they received.
An original probability example illustrates the difference without estimating any actual benchmark score. Suppose a service succeeds independently with probability 0.8 on each interaction. The chance that all four interactions succeed is 0.8 multiplied by itself four times, or 0.4096. The chance that at least one succeeds is one minus 0.2 to the fourth power, or 0.9984. Those numbers answer almost opposite operational questions. Real task outcomes are not necessarily independent, and benchmark aggregation adds further details, but the example explains why reliable repetition should not be confused with best-of-many sampling.
The user becomes part of the control loop
The tau2 paper, submitted June 9, 2025, introduced a telecommunications domain where both the user simulator and the agent have tools that affect shared state. The agent might control account-side settings while the user controls a device. The authors model this as a decentralized, partially observable decision problem and use compositional tasks to study coordination. Their experiments isolate aspects of reasoning and communication, rather than treating every failed support conversation as a single undifferentiated error.
Imagine a connection problem that requires changing a network setting and then restarting a device. The order matters. If the user restarts first and the agent assumes the later setting is already active, both participants can perform reasonable actions without reaching the goal. The agent needs to confirm what happened and what remains. A natural-language instruction is therefore an attempted intervention, not proof of an intervention. This distinction motivates tools and observations that constrain the simulated user's behavior, while leaving open the harder question of how faithfully simulated coordination predicts human interaction.
Finding a policy is only the first step
Tau-Knowledge, submitted March 4, 2026, extends the setting toward actions that depend on a large collection of unstructured documents. Its banking support domain couples roughly seven hundred knowledge documents with account-update tools. In that paper's experiments, even high-reasoning-budget frontier systems remained limited, with the abstract reporting approximately 25.5% single-trial success for its evaluated setting. This is a dated research result. It should not be read as the current leaderboard value after the later grading changes or as a measured success rate for commercial banking deployments.
Retrieval and policy application can fail independently. An agent may retrieve the correct transfer policy but apply a general rule while missing an exception for a specific account status. Another may reason correctly from the wrong document. A third may explain the correct action but populate the tool arguments incorrectly. In an original diagnostic design, evaluators would separately record the governing document, the relevant condition, the proposed action, and the actual mutation. That evidence would locate whether improvement came from search, reasoning, or execution rather than attributing every gain to a larger language model.
Voice adds timing to the task
Tau-Voice, submitted March 14, 2026, evaluates full-duplex agents that can listen and speak simultaneously. Its framework combines grounded support tasks with simulated voices, accents, audio conditions, and turn-taking behavior. Across 278 tasks, the authors reported lower voice completion rates under realistic audio conditions than under clean ones, and a substantial gap from the text baseline in their setup. These comparisons concern the tested systems and simulator. They do not establish that every voice product loses the same proportion of its text capability.
Timing introduces errors that a transcript-only evaluation can overlook. A user might interrupt a confirmation with a corrected date while the agent is already issuing the tool call. The final transcript could contain both dates without making clear which one controlled the action. A useful evaluation preserves audio timing and the sequence of state-changing operations. It can then ask whether the agent waited for a completed confirmation, handled an interruption, or continued speaking over a critical correction. Natural speech quality and correct tool execution are related qualities, but neither guarantees the other.
The July repair changes what comparison is valid
The official version 1.0.1 release was published July 22, 2026. It corrected banking_knowledge task errors and regraded affected leaderboard submissions. The maintainers explicitly warn that results from earlier grading are not comparable with version 1.0.1 or later on that domain; other domains are unaffected. The release also describes replaying old trajectories against fresh task definitions. This is an important example of a score changing because the measuring instrument changed, even when the recorded agent behavior stayed the same.
For readers, the appropriate response is to identify the version rather than erase the earlier work. The March knowledge paper establishes what was reported under its original evaluation. A later regrade establishes what those same trajectories receive under repaired rules. Neither justifies silently combining the old denominator with the new score. For developers, preserved trajectories are valuable because they permit this correction without rerunning the conversation and introducing fresh sampling variation. They also make disagreement inspectable: one can examine whether the agent was wrong, the expected action was wrong, or the policy left genuine ambiguity.
Reliability needs more than a fluent answer
The combined research suggests a concrete test for a support agent: give it a request whose relevant facts are distributed across a user conversation, tool observations, and policy documents, then introduce a correction before the final action. Require an auditable result that preserves unrelated state. Repeat the task with controlled variations and inspect the failures. This is a proposed stress test derived from the problems above, rather than a claim that the tau papers already establish every deployment requirement.
A production decision would still need evidence beyond these benchmarks. Real people express goals differently from a simulator, account data can be incomplete, and actions may have consequences that cannot be undone by resetting a test database. Yet the 2026 extensions make the evaluation target clearer. Useful conversational intelligence is the ability to obtain the right evidence, coordinate with another actor, follow the applicable policy, and execute the correct action consistently. A pleasant conversation is part of that experience. The final state and the path to it determine whether the promised help actually happened.
Sources & further reading
Primary papers, project documentation, and research releases used in this article. Publication dates below refer to the sources themselves.
- τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World DomainsarXivPublished Checked
- τ²-Bench: Evaluating Conversational Agents in a Dual-Control EnvironmentarXivPublished Checked
- τ-Knowledge: Evaluating Conversational Agents over Unstructured KnowledgearXivPublished Checked
- τ-Voice: Benchmarking Full-Duplex Voice Agents on Real-World DomainsarXivPublished Checked
- τ-bench 1.0.1 — banking_knowledge Grading FixesSierra ResearchPublished Checked
FOLLOW THE QUESTION
Study engineering foundations