OpenAI board member Zico Kolter says AI cyber capabilities demand new safety measures

Zico Kolter, an OpenAI board member and Carnegie Mellon University professor, says the AI industry’s growing awareness of what its models can do in cyberspace is forcing a rethink of how security works. The comments carry weight because of where Kolter sits. He chairs OpenAI’s Safety and Security Committee, the internal body that can hold back a model launch until its risks are dealt with. What Kolter is actually saying Kolter’s core point is that AI systems now present a threat landscape traditional software security was never designed to handle. In his view, that landscape includes AI-enhanced cyberattacks and even help with bioweapon design. Kolter has singled out two risks in particular: prompt injection and agent autonomy. He describes both as problems with no real equivalent in traditional security. Prompt injection is when someone hides instructions inside content an AI reads, such as a webpage or an email, and tricks the model into following them. Agent autonomy is the related worry about AI systems that take actions on their own, like browsing, executing tasks or touching other software. The more freedom an agent has, the more damage a single successful trick can cause. Ko...