DeepSeek’s V4 Flash struggles with real-world tasks despite topping AI leaderboards

DeepSeek’s V4 Flash has been called a “total monster” by developers since its July 31 launch. The model shot to the top of multiple AI leaderboards, and its pricing, at $0.14 per million input tokens and $0.28 per million output tokens, undercuts comparable models by roughly tenfold. In practice, the monster has a limp. When testing firm Composio ran V4 Flash through a battery of real-world agent tasks, the model managed a 53.8% pass rate. Out of 240 total runs spanning 30 deliberately difficult, multi-step workflows, only 129 passed. Just six of the 30 workflows were completed successfully by every agent harness tested. What the Composio tests actually measured Composio’s evaluation tested V4 Flash across eight different agent harnesses, including Claude Code, Codex, and OpenCode. The 30 tasks involved live tools that developers actually use every day: Gmail, GitHub, Slack, and Google Sheets. Results varied dramatically depending on which harness was used. Pi Agent emerged as the strongest performer, completing 20 out of 30 tasks. The same underlying model can look brilliant or mediocre depending entirely on how it’s integrated into a workflow. The benchmark-to-reality gap persist...