CoinPhoton

A recent re-evaluation by BlockSec challenges the growing narrative that AI can fully automate smart contract auditing. While earlier results from EVMBench—developed by OpenAI, Paradigm, and OtterSec—suggested rapid progress, BlockSec’s findings paint a more grounded picture.
TL;DR (including Zhou’s statement)
“EVMBench says AI can exploit 72% of smart contract vulnerabilities, and the industry started talking about fully automated auditing. We re-tested with more configurations and 22 real-world attack incidents. Exploit success: 0%. AI auditing has real value, but replacing humans is not close. The right direction is human-AI collaboration.” — Yajin Zhou
A promising benchmark — but with limitations
EVMBench marked a major step forward by introducing a standardized benchmark for evaluating AI agents in smart contract security. It tested 120 vulnerabilities across 40 Code4rena audit repositories, covering tasks like detection, patching, and exploitation.
The headline results were striking:
45.6% detection rate72.2% exploit success rate
These numbers led many to believe that fully automated AI auditing was within reach.
Re-evaluating with real-world conditions
BlockSec’s study, “ReEVMBench,” aimed to test whether those conclusions hold under more realistic conditions.
Two major improvements were introduced:
Expanded configurations: Increased from 14 to 26 combinations across multiple model families and execution frameworksNew dataset: 22 real-world security incidents occurring after February 2026, ensuring no overlap with AI training data
This approach addressed two critical issues in the original benchmark:
Potential bias from limited model–framework combinationsRisk of data contamination from known vulnerabilities already included in training datasets
Key finding: real-world exploit success is 0%
The most important result is clear:
110 agent-incident tests0 successful end-to-end exploits
Despite earlier claims of 72% exploit capability, AI agents failed entirely when faced with real-world attack scenarios.
However, detection performance remained relatively strong:
Top models identified up to 65% of real-world vulnerabilitiesCommon patterns (e.g. overflow, access control issues) were consistently detected
The gap appears in execution. AI struggles to:
Understand cross-contract dependenciesChain multi-step attack logicAdapt to complex, real-world protocol behaviors

AI excels at patterns, not reasoning
The study highlights a fundamental limitation:
AI is strong at pattern recognitionWeak at deep reasoning and adversarial thinking
Simple, known vulnerabilities are reliably detected. But complex exploits—especially those requiring protocol-level understanding—remain out of reach.
In one case, all 26 tested agents failed to detect a vulnerability in a signature validation system, even when directly analyzing the relevant code.
Human guidance dramatically improves results
One of the most revealing insights comes from “hint” experiments:
Exploit success increased from 62.5% to 95.8% when human hints were provided
This suggests that AI is not incapable—but lacks direction.
AI can execute tasks effectively once guided, but cannot independently determine attack strategies in complex environments.
The real takeaway: collaboration, not replacement
BlockSec concludes that AI will not replace human auditors anytime soon. Instead, the future lies in collaboration:
AI handles breadth: large-scale scanning and pattern detectionHumans handle depth: protocol understanding and adversarial reasoning
Each compensates for the other’s weaknesses.
Conclusion
AI auditing is already valuable—but far from autonomous.
Detection ceiling: ~47.5%Real-world exploit success: 0%Performance varies significantly across setups
Smart contract security is fundamentally unforgiving: missing even one critical vulnerability can lead to catastrophic losses. In this context, partial automation is not enough.
The question is no longer whether AI can replace humans—but how humans and AI can work together effectively.
The answer, for now, is clear: human-in-the-loop systems are the only viable path forward.
https://x.com/yajinzhou/status/2034572010030862349