Microsoft, Cornell University unveil Free Pause Tokens method for efficient language model training

Researchers from Microsoft and Cornell University have published a new training technique that makes large language models measurably better at predicting the next token, all while keeping inference speed essentially unchanged. The method, called Free Pause Tokens, represents one of those rare improvements in AI research where you get something for almost nothing. The paper, titled “Free Pause Tokens” (also referred to as “Almost Free State Prediction Separation”), was submitted to arXiv on September 3. It describes a parallel prediction stream that shares weights with a model’s main state stream, adding extra computation near the end of training without the usual penalties that come with architectural changes. What Free Pause Tokens actually do Previous research from 2023 and 2024 tried to address transformer models’ fixed computation constraints by inserting explicit “pause tokens” into the input sequence. These gave the model extra steps to think, but they came with a cost: longer context lengths, bigger key-value caches, and slower inference. The Free Pause Tokens approach sidesteps these problems entirely. Instead of adding tokens to the input sequence, it creates a lightweigh...