Easy-First Training Makes AI 2.23x Better at Learning Exceptions
A new study shows that teaching simple rules first helps both people and neural networks because curricula force them to unlearn overly compressed representations.
Takeaways
- New paper argues curricula help by forcing learners to unlearn overly compressed representations, not just learn faster.
- Both humans (n=207) and matched neural networks showed simplicity bias on a rule-plus-exceptions task.
- Simple-to-complex curriculum made participants 2.23x more likely to adopt the correct rule+exceptions strategy.
- Exception accuracy correlated with hidden-layer participation ratio at r = +0.70 across networks.
- Pretraining that compresses representations (1-D rule) amplified curriculum benefit; XOR pretraining did not.
- No code or models released; study is mechanistic rather than a benchmark result.
Curriculum learning orders training examples from easy to hard, a strategy used in education and machine learning. A new paper from the Summerfield and Saxe labs offers a computational explanation for its benefits. In controlled classification experiments, easy-first schedules helped humans and neural networks escape an overly simple rule by expanding internal representations to encode exceptions.
The simple rule crowds out exceptions
People and gradient-trained networks often adopt the simplest rule that fits most examples, even when the task contains exceptions. A network expresses this bias by compressing its hidden activations along the dimension that represents the dominant rule. Features needed to distinguish unusual cases then become difficult to recover.
An easy-first curriculum initially teaches the dominant rule, then presents the exceptions. Once the rule is learned, those exceptions account for the remaining errors and force the learner to restore dimensions that separate them. When every example appears from the start, the more numerous rule-following items can dominate training and preserve the compressed representation.
Eight stimuli, three schedules
The researchers tested 207 people on a standard bug-classification task containing eight stimuli. Most followed a simple classification rule, while two required an exception strategy. They also trained one shallow neural network per participant, using that participant’s exact sequence of trials.
| Schedule | Training order |
|---|---|
| Simple to complex | Rule-following items first, then exceptions |
| Complex to simple | Exceptions first, then rule-following items |