What AI extinction fears mean for business leaders

It’s been a dramatic few days in the AI world. In case you missed it, Jacob Coxon, former researcher at Anthropic, took to X early September to announce his resignation, claiming that the people building AI fear that the technology “could kill us all”. 

Evan Hubinger, Alignment Science Lead at Anthropic, supported Coxon’s statement, estimating there was a greater than 10% chance of AI pushing humanity to extinction within the next decade, giving new weight to fears about artificial intelligence. 

While I believe these claims have been a bit sensationalist, and dare I say, fear mongering, even still, anyone developing technology this powerful should be thinking hard about where it could lead. For business leaders, the immediate challenge is how to keep moving with AI when the people they need to bring with them may trust it less than ever.

AI adoption is facing its biggest trust problem yet

Even though AI has neared ubiquity – embedded across offices, search results, our phones, kitchens and cars, and the apps we use every day – public confidence is not keeping up. According to KPMG’s “UK attitudes to AI” report:

• 78% of people are concerned about negative outcomes from AI
• 80% of the UK public believe AI regulation is required
• 91% want laws and action to combat AI-generated misinformation

Crucial to know is that this report was conducted in 2025; with Coxon and Hubinger adding fresh fuel to concerns around job security, opaque decisions and, indeed, human extinction risk, I wouldn’t be surprised if those figures have dramatically increased this month.

It would be easy and even logical for business leaders to dismiss those fears as melodrama and carry on with their AI strategies as normal – though I think that would be a serious mistake. If leaders underestimate how uneasy people feel about AI, they risk damaging the trust they will need to introduce or champion it successfully.

AI transformation depends on buy-in 

AI strategy has often been framed around capability: what can the technology do and where can we create value?

But today’s public debate has added another requirement. Leaders now need a credible answer for why employees, clients, customers and stakeholders should trust the way their organisation is using AI. 

Securing that buy-in means explaining why AI is being used, where the boundaries are, and who remains accountable when something goes wrong. A leader who only talks about productivity gains will struggle with an audience worried about redundancy, while a leader who reacts to every alarming headline by slowing investment will struggle to build a competitive business.

Good leadership lives in those who balance conviction with restraint; who make the case for useful AI investment while still taking concerns seriously. Leaders who can explain the benefits, set clear limits and bring people with them will be far better placed to turn AI plans into something people will actually support.

The leadership brief has become harder

From an executive search perspective, companies should pay close attention to what they now need from senior leaders responsible for AI adoption. Technical fluency carries obvious value, but credibility will depend more on how that fluency translates into human judgement:

• Can a CEO explain an AI strategy to a sceptical workforce? 
• Can a CTO set boundaries and defend them when commercial pressure rises?
• Can a Chief Product Officer win the confidence of clients who are uneasy about AI being embedded in the products or services they rely on?

Those capabilities are harder to assess than knowledge of a particular platform, and they have a direct bearing on followership because people support change when they trust the people leading it.

Executive hiring therefore needs to look beyond a candidate’s record of technology delivery and ask how they create legitimacy around change. Boards need leaders who can explain difficult trade-offs without hiding behind technical language while retaining enough commercial conviction to keep useful transformation moving.

Responsible ambition with beat panic

I have no idea if AI carries a 10% extinction risk, and I would be suspicious of anyone who claimed certainty either way. 

Regardless, pausing AI plans because the rhetoric has become frightening would hand away useful ground. At the same time, charging ahead while treating scepticism as ignorance would damage trust when companies need it most. Both scenarios demonstrate why the quality of leadership around AI deserves as much attention as the technology itself.

For boards planning senior appointments, it’s worth remembering that leaders who create real value from AI will be those who can pair ambition with restraint and give people credible reasons to follow them.

As the debate around AI becomes louder, companies will discover that successful adoption depends on something deeply human: trust in the person asking everyone else to come with them. 

If you have anything you’d like to add to this debate, or if you’d like guidance on your next AI leadership appointment, I’d love to speak with you.