Skip to main content
Back

ACE-SQL Beats GPT-4 on Hard SQL Benchmarks Using 70% Fewer Tokens

ACE-SQL uses reinforcement learning to jointly train schema retrieval and SQL generation, hitting 65.3% on BIRD Dev with 70% fewer tokens than competing methods

  • ACE-SQL jointly trains schema retrieval and SQL generation using RL, achieving 65.3% execution accuracy on the BIRD Dev benchmark — best among open-source 7-8B models.
  • The key innovation is an empirical column-set pool: retrieval targets are derived from which columns the generator actually used in correct SQL, not static gold annotations.
  • ACE-SQL uses only 0.93k output tokens per query, roughly 70% fewer than SQL-R1 (3.10k), while scoring 1.6 points higher on BIRD Dev.
  • Joint RL training of two roles in one model requires two stabilizers: PCGrad (to resolve conflicting gradients) and a generator-weight warmup schedule.
  • The approach generalizes beyond Text-to-SQL as a framework for co-optimizing any retriever-generator pipeline using downstream execution feedback.
  • Code is open-source at github.com/xbchen1/ACE-SQL, trained on ~3k RL examples with 4x A100 GPUs.

Getting an LLM to write correct SQL is harder than it looks. The model doesn't just need to understand your question , it needs to navigate a database schema that might have hundreds of tables and thousands of columns, figure out which handful are actually relevant, and then write a query that executes correctly. ACE-SQL, a new paper from researchers at Harbin Engineering University and Harbin Institute of Technology, attacks this problem with a reinforcement learning framework that trains both steps together, and the results are striking: state-of-the-art accuracy on the BIRD benchmark while using roughly 70% fewer output tokens than the nearest competitor.

The schema linking bottleneck

Schema linking , the step where a model decides which tables and columns are relevant to a question , is the silent killer of Text-to-SQL accuracy. Replacing full schemas with gold-column inputs improves execution accuracy by up to +15.4 points, and even a model trained on only 4k gold-column samples can surpass a 2.5M-sample model under gold-column inputs. That's a massive gap, and it's almost entirely attributable to the quality of schema selection upstream.

Bar charts showing execution accuracy gap between full schema and gold column inputs across model types

There are two standard approaches to schema linking, and both have a fundamental flaw. The first is full-schema generation: just dump the entire database schema into the prompt and let the model figure it out. This works poorly because the model has to do implicit schema selection inside a huge search space. The second approach uses a separate retriever model to pre-select relevant columns, then passes those to a SQL generator. This is better, but the retriever is typically trained against static "gold" column annotations , the columns the benchmark says are correct.

The problem with static gold targets is subtle but important. From 19.1% to 42.3% of execution-correct full-schema predictions rely on non-gold column sets, particularly on harder queries. SQL often has multiple valid routes to the same answer. A retriever trained to always pick the annotated columns will penalize the model for finding alternative, equally valid paths , paths the generator may actually prefer based on what it has learned.