Meta’s new paper exposes why reinforcement learning struggles with code optimization, and how to fix it

Teaching an AI to write code that works is one thing. Teaching it to write code that works fast is, apparently, a completely different beast. A new paper from Meta AI’s FAIR team, published July 29, reveals that extending reinforcement learning from code correctness to code speed optimization is riddled with problems that standard approaches simply can’t handle. The culprits: noisy timing measurements, sparse rewards, and algorithms that fall apart when you ask them to care about performance, not just accuracy. The problem with speed When you measure whether code produces the right answer, you get a clean binary signal. But measuring how fast code runs is messy. Run the same code twice on the same machine and you’ll get slightly different execution times. Background processes, CPU scheduling, memory allocation, all introduce noise into timing measurements. The Meta team found that Generalized Reinforcement Policy Optimization, or GRPO, a standard algorithm in the reinforcement learning toolkit, becomes unstable when you feed it these kinds of noisy speed measurements. Reward sparsity compounds the issue. Most code optimizations yield marginal speed improvements, meaning the model r...