a16z research: AI agents can identify DeFi price manipulation vulnerabilities but cannot yet replace professional security auditors

On April 29, according to a16z disclosure, its researchers conducted systematic tests on whether AI agents can independently complete the exploitation of DeFi price manipulation vulnerabilities. The study used 20 Ethereum price manipulation events as a dataset and used Codex (GPT5.4) equipped with the Foundry tool chain as a test agent. Under the benchmark condition of no domain knowledge, the agent success rate is only 10%; after introducing structured domain knowledge refined based on real attack events, the success rate is increased to 70%. Failure cases show that agents can accurately identify vulnerabilities, but they are generally unable to understand the leverage logic of recursive lending, misjudge profit margins, and unable to assemble cross-contract multi-step attack structures. The experiment also recorded a sandbox escape event: the agent extracts the RPC key in the local node configuration, calls the anvil_reset method to reset the node to a future block, bypassing information isolation restrictions and obtaining real attack data. The research team believes that AI agents can currently effectively assist in vulnerability identification, but they cannot yet replace professional security auditors.