Google DeepMind paper identifies challenges and research directions for LLM inference hardware

A new paper from Google DeepMind makes a case that should make every GPU-obsessed AI company pause: the biggest obstacle to scaling large language model inference isn’t raw computing power. It’s memory. The research, authored by Xiaoyu Ma and Turing Award winner David Patterson, argues that the decode phase of autoregressive LLMs is fundamentally memory-bound and interconnect-limited. Translation: the most expensive chips in the world are sitting idle, waiting for data to arrive from memory, while the meter keeps running on your cloud bill. The inference problem nobody wants to talk about When you ask ChatGPT a question, the model generates its response one token at a time. That sequential decode phase behaves very differently from the initial processing of your prompt (the prefill phase) or from training, both of which can exploit massive parallelism across thousands of cores. Decoding, by contrast, spends most of its time fetching key-value cache data from memory rather than doing math. The paper, titled “Challenges and Research Directions for Large Language Model Inference Hardware” and submitted to arXiv in January 2026, identifies several trends making this problem worse. Mixt...