UIUC's FOXGLOVE Catches What AI Writing Feedback Gets Wrong
A new dataset of 2,340 essay comments reveals where LLMs and human writing instructors agree, diverge, and why longer comments inflate AI quality scores
Takeaways
- New dataset: FOXGLOVE contains 2,340 essay feedback comments from 14 human instructors and 4 frontier LLMs on 69 high school essays.
- Models tested: GPT-5.2, Claude Sonnet 4.5, Llama 3.3 70B, and Qwen3-Next 80B, all under a shared feedback protocol.
- Key finding: Humans and LLMs agree on which essay goals to address, but diverge significantly on which specific sentences to target.
- LLM quality scores inflated: LLMs rated higher on most quality dimensions, but the advantage largely traces back to writing longer comments, not deeper insight.
- Style gap: LLMs write more prescriptive, complex feedback; human instructors use more questions to prompt student reflection.
- Open source: Dataset is publicly available on GitHub with sentence anchors, goal labels, urgency rankings, and expert quality ratings.
Every time a student submits an essay and an AI tool spits back feedback, a fundamental question goes unanswered: is that feedback actually doing what good writing instruction is supposed to do? A team from the University of Illinois Urbana-Champaign decided to find out, and the results are more nuanced than the usual "AI is better" or "humans are better" narrative.
The researchers built FOXGLOVE, a carefully structured dataset that puts human writing instructors and frontier LLMs side by side on the exact same essays, under the exact same instructions, and measures them on the dimensions that writing research actually cares about.
The gap nobody was measuring
There was no systematic comparison of LLM and expert feedback on the dimensions that writing research identifies as central to revision: goal-orientation, anchoring to specific sentences, and prioritization. That absence is striking given how widely AI writing tools are now deployed in classrooms.
To understand why this matters, it helps to know what these three dimensions mean in practice:
- Goal-orientation: Does the feedback target a specific argumentative goal -- like strengthening a claim, adding evidence, or addressing a counterclaim?
- Sentence anchoring: Is the feedback tied to a specific sentence in the essay, or is it vague and floating?
- Prioritization (urgency): Does the feedback signal what the student should fix first?
Answering this question at scale requires data that is presently unavailable: there are no datasets that capture trained instructor and LLM generated feedback along the same protocol, anchored to specific text spans, labeled with the goals the feedback addresses, and ranked by urgency. FOXGLOVE fills that gap.
What they built
FOXGLOVE collects anchored, goal-labeled feedback comments from trained human instructors (696 comments) and four models -- GPT-5.2, Claude Sonnet 4.5, Llama 3.3 70B, and Qwen3-Next 80B -- produced under a shared protocol. The essays themselves are 69 twelfth-grade argumentative essays, giving a total of 2,340 feedback comments across the dataset.