IBM introduces BenchDrift to quantify wording effects in LLM benchmarks

If you’ve ever aced a test only to bomb a retake with slightly reworded questions, you already understand the problem IBM just put a number on. The company’s research division has published a paper introducing BenchDrift, a tool designed to measure how much large language model benchmark scores shift when test prompts are rephrased without changing their meaning. What BenchDrift actually measures The core idea is deceptively simple. Take a benchmark question, rewrite it in a way that preserves its meaning and correct answer, then see if the model still gets it right. BenchDrift does this systematically across four distinct axes: linguistic, referential, pragmatic, and structural. BenchDrift generates these variations at scale, then tracks what researchers call “bidirectional correctness flips,” instances where a model’s answer switches from right to wrong or wrong to right based solely on phrasing. The research team, led by Shailja Thakur, Sungeun An, Chad DeLuca, and Hima Patel at IBM Research, tested eight models across three widely used benchmarks: GSM8K (grade-school math), MMLU (massive multitask language understanding), and MATH-Hard (advanced mathematical reasoning). Drift s...