Harvard paper reveals new scaling axis for generative models

For years, the playbook for making AI models better has been straightforward: more parameters, more data. A new paper from researchers at Harvard and the University of Illinois at Urbana-Champaign argues there’s a third lever that’s been hiding in plain sight: exploration. The paper, titled “Explorative Modeling: Unlocking a Third Pretraining Axis and End-to-End Generation,” introduces what the authors call Explorative Modeling, or XM. The core idea is that increasing the number of candidate generations a model explores during training produces outsized improvements in efficiency and output quality. The numbers behind the claim Expanding the exploration mode, referred to as K, yielded a 4.1x increase in FLOP efficiency, a 6.2x boost in sample efficiency, and a 47% improvement in parameter efficiency. In practical terms, that means models can achieve the same quality outputs while burning through far fewer computational resources. Those gains also appear to compound as you scale. The paper reports that efficiency improvements grew from 7% to 36% as the amount of training data increased, and from 13% to 23% as model sizes expanded. On the benchmark front, the XM approach achieved an ...