Microsoft introduces ThinkingBox to assess AI agent reliability

Microsoft has released ThinkingBox, an open-source sandbox framework designed to test whether AI agents can actually be trusted to handle real business tasks. The tool, detailed in a Microsoft Command Line blog post on August 19, 2026, by Principal Machine Learning Engineer Liang-Chun Tsai, takes a fundamentally different approach to evaluating AI agents. Instead of grading them on what they say they did, ThinkingBox checks what they actually changed in a database. The discovery-reliability gap ThinkingBox comes paired with a benchmark called ThinkingBox-Bench, and the initial results are sobering. Microsoft tested 12 different proprietary and open-weight models across 507 specific tasks spanning five business domains. Each task was run through 20 separate trials, creating a rigorous statistical picture of how these agents perform under repeated conditions. The best-performing model managed a 65.36% pass@1 rate, meaning it completed a task correctly on its first try roughly two-thirds of the time. That sounds reasonable until you see the pass^20 number: 25.25%. Pass^20 measures whether a model can succeed across all 20 trials of the same task. Microsoft calls this the “discovery-re...