On June 5th, Beijing time, the AI startup Anthropic dropped a blog post, urging top AI labs worldwide to seriously consider slowing down the development of cutting-edge models. The company warned that the current pace of AI system evolution is too rapid, and we might soon see 'recursive self-improvement' happening without human intervention, leading to social risks that shouldn't be ignored.
Internal data from Anthropic shows that its most advanced model capabilities are skyrocketing at an astonishing rate. The so-called 'recursive self-improvement' refers to AI systems' ability to help enhance the next-gen models, optimize their own capabilities, or accelerate the R&D process.
Once AI no longer heavily relies on human engineers, the pace of AI advancement could significantly outstrip the existing regulatory and safety evaluation frameworks. The article's authors, who are the head of the Anthropic Institute and policy leads, stated: 'We believe it would be beneficial for the world to have the option to slow down or temporarily pause cutting-edge AI development, allowing societal structure and alignment research to catch up with technological progress.' The article also suggested reaching a global agreement on how to slow down R&D and establishing a verification mechanism to monitor compliance among peers.
This call for attention goes beyond just the tech level. On one hand, Anthropic is at the forefront of fierce competition – it just completed a funding round with a valuation of $965 billion, surpassing competitor OpenAI (valued at $852 billion) for the first time, and has secretly submitted an IPO application, planning to hit Wall Street as early as this fall.
On the other hand, its security stance has long been controversial. Venture capitalist and informal advisor to Trump, David Sacks, has accused the company of pursuing a 'regulatory capture agenda', using regulation as a means to stifle competitors. Professor Ethan Mollick from the Wharton School at the University of Pennsylvania commented that Anthropic's article is 'somewhat self-reflective and has some marketing elements', but its judgments on the recent direction of AI are 'worthy of attention'.
Notably, this call is occurring at a time when the US AI regulatory framework is rapidly taking shape. On June 2, President Trump signed an executive order requiring AI companies like OpenAI, Google, and Anthropic to open a cybersecurity testing window for up to 30 days to the government before publicly releasing their latest models. Meanwhile, Congress is advancing bipartisan AI legislation, requiring top developers to disclose the safety and security risks of new models.
This means that the debate on whether the 'AI race needs speed limits' is moving from internal industry discussions to the core of policy and capital market discussions. Anthropic's candid remarks undoubtedly elevate this topic to a higher level of public decision-making.