Amazon's ConvAgent Beats ChatGPT and Claude at Multi-Turn Conversational Search
A new agentic framework called ConvAgent uses reinforcement learning with three specialized rewards to interleave search and reasoning across multi-turn conversations.
Takeaways
- ConvAgent trains a single LLM policy to interleave reasoning, search, clarification, and answering across conversation turns via GRPO reinforcement learning.
- Three-part reward decomposition: outcome F1/EM, information gain from retrieved passages, and mixed-initiative action classification with a -0.5 penalty for spam.
- The 7B Qwen-2.5 model beats Search-R1, AgenticLM, ChatR1, ChatGPT, and Claude on TopiOCQA, QReCC, and CORAL F1.
- Information gain reward supervises query rewriting using only ground-truth answers, removing the need for gold rewritten queries.
- Mixed-initiative reward lifts clarification accuracy from 3.67% to 83.19% and no-answer accuracy from 1.42% to 11.51%.
- Achieves better answers than AgenticLM with roughly one-third the reasoning tokens; no public code release yet.
Multi-turn conversational search has always been the awkward middle child of retrieval-augmented generation. Single-turn deep search agents like Search-R1 can reason and hit a search engine iteratively, but they assume the question is self-contained. Real conversations rarely are. A new paper introduces ConvAgent, an agent trained with reinforcement learning to handle the messy realities of dialogue: pronouns that refer three turns back, topic switches, ambiguous queries that need clarification, and questions that should be refused outright.
The team behind it comes from Amazon, the University of Montreal, UMass Amherst, Notre Dame, and Northeastern. Their pitch is straightforward. Existing conversational RAG systems break the task into a static rewrite -> retrieve -> generate pipeline where each module is optimized separately. Agentic deep-search models fuse those steps through reasoning but were built for one-shot questions. ConvAgent tries to combine the best of both worlds.
Where the pipelines fall apart
The static rewrite-retrieve-generate workflow trains separated models with different task-specific supervision signals, including rewritten queries, relevance judgments, and ground-truth answers for each turn. That is a lot of labeled data to collect for every domain you want to deploy in.
A conversational agent also has to do things a single-turn agent never worries about:
- Generate a self-contained search query by reading the conversation history, not just decomposing a self-contained question.
- Decide when a question is actually ambiguous and needs a clarifying follow-up.
- Refuse to answer when retrieved evidence is insufficient, rather than hallucinating.