2025-08-18 Jamie Lord
Every textbook showed the same inexorable curve: small models underfit, optimal models generalise, large models catastrophically overfit. End of story.
…
In 2019, a group of researchers committed the ultimate sin: they ignored the warnings and kept scaling anyway. Instead of stopping when their networks achieved perfect training accuracy—the point where theory screamed “danger”—they pushed further into the forbidden zone.
What happened next shattered 300 years of learning theory.
The models didn’t collapse. After an initial stumble where they appeared to memorise their training data, something extraordinary occurred. Performance began improving again. Dramatically.
…
The lottery ticket hypothesis crystallised: large networks succeed not by learning complex solutions, but by providing more opportunities to find simple ones. Every subset of weights represents a different lottery ticket—a potential elegant solution with random initialisation. Most tickets lose, but with billions of tickets, winning becomes inevitable.
During training, the network doesn’t search for the perfect architecture. It already contains countless small networks, each with different starting conditions. Training becomes a massive lottery draw, with the best-initialised small network emerging victorious whilst billions of others fade away.