“The current race towards ever more powerful AI is a step change towards greater existential risks to every aspect of our lives,” he wrote in an open letter.

The High Commissioner’s message comes six days after a former researcher at Anthropic – the company behind the chatbot Claude – resigned from his position while warning that “the people building AI earnestly believe that it could kill us all by the end of the decade.”

Several recent incidents, including Jacob Coxon’s resignation and announcements by OpenAI and Anthropic that their models had broken out of testing environments, have intensified the global debate over the extent to which governments should regulate the technology.

No country can go it alone

Mr. Türk called for countries hosting frontier AI companies to increase regulations, including by requiring companies to report serious incidents, verifying AI capabilities and mandating human rights due diligence.

He also urged Member States to work together via multilateral institutions to build global consensus on AI governance and protect human rights in the face of AI risks.

“No country can govern this technology alone,” Mr. Türk wrote. “No company should be able to decide by itself which risks the world must accept. No person should be required to surrender their human rights in exchange for claims of technological progress.”

‘Pace the frontier’

Anthropic CEO Dario Amodei on Saturday published an essay entitled We Must Pace the Frontier, arguing for companies to continue progress on AI development while taking the requisite time to ensure sufficient safeguards to prevent “agentic systems” from going rogue.

Leaders at OpenAI, Google and xAI all expressed agreement with Mr. Amodei’s proposal, which includes calls for third-party evaluation of AI models, increased coordination between AI companies and cooperation between governments.

Mr. Türk welcomed Mr. Amodei’s proposal specifically and industry self-regulation broadly – but added that governments must also urgently act to regulate AI technology.

No ‘race to the bottom’

“The handful of States hosting leading frontier AI companies have a particular obligation to require those companies to act responsibly and hold them accountable if they do not,” the High Commissioner wrote.

Identifying companies in China and the US as “at the forefront” of frontier AI, Mr. Türk recognised the need for international standards and alignment to reduce the incentives for an AI arms race between the two countries.

“They need to align their approaches so that competition between jurisdictions does not encourage a race to the bottom,” Mr. Türk wrote.

The risks and governance of AI should primarily be judged by its impact on human rights, according to Mr. Türk.

Beyond ensuring AI doesn’t destroy humanity, the rights chief said AI safety means actively protecting people, communities, institutions and future generations.

People over profit

Mr. Türk rejected the industry argument that AI safety protections slow down innovation, arguing that nobody should be forced to give up their human rights to enable progress.

As it stands, he said the rights to life, food, privacy, health and security are all threatened by unchecked AI development.

Whether AI continues to enable disinformation and unequal automation or tackles the climate crisis and cures diseases depends on how companies and governments choose to respond to the current “watershed moment,” according to Mr. Türk.

“The generation now entering adulthood will inherit the consequences of the decisions being taken on this issue without having a say in them,” Mr. Türk wrote. “We owe it to them to ensure those decisions are made openly, with their rights in view, and with the humility that the scale of this undertaking demands.”