Anthropic urges global freeze on AI as it warns of losing control

The world’s most valuable artificial intelligence company has called for a global freeze on the development of the most powerful AI systems, warning that humanity is approaching a point where it could lose control of the technology it is racing to build. In a blog post published on Thursday (Friday AEST), Anthropic, the San Francisco company behind the Claude chatbot, said it would be willing to suspend work on more capable systems if it could be confident that its rivals would do the same. “If it were possible to effectively slow the development of this technology to give ourselves more time to deal with its immense implications, we think that would likely be a good thing,” executives wrote. Anthropic’s Claude Mythos Preview has been too powerful to release to the general public. BloombergAnthropic has every commercial reason to keep building. It recently overtook ChatGPT maker OpenAI to become the world’s most valuable AI firm, at a valuation of about $US965 billion ($1.3 trillion). It has also withheld its most powerful system, known as Mythos, from public release over fears it could be used to carry out devastating cyberattacks, and it said this week it would extend access to Mythos to Australian organisations for the first time, as part of a staggered rollout meant to fix security flaws before the product reaches hackers. The company’s argument turns on a concept it calls “recursive self-improvement”: the point at which AI systems become capable of designing and building their own successors with little human input. Anthropic head of research Marina Favaro and its president Jack Clark said the industry was edging towards that threshold, and that crossing it could trigger an explosion in capabilities that outpaces society’s ability to manage the consequences. Anthropic chief executive Dario Amodei has called for regulation of the technology. BloombergTo make the case, the company published internal data on how quickly Claude is improving, and it said more than 80 per cent of new code had been written . In one instance, the company said Claude had resolved more than 800 software bugs in April that it estimated would have taken a human engineer four years to fix. Favaro and Clark called the situation an “arms control problem”. They said there was limited time to act. The industry remains split over how close today’s systems are to milestones such as recursive self-improvement and artificial general intelligence, a level of machine intelligence broadly comparable to a human’s.