Existential Risk from AI: An Exposition for Mathematicians (excerpt)

Xiaoyu He, Assistant Professor at Georgia Institute of Technology

What follows is an edited and abridged version of a longer essay. The full version is available at alkjash.github.io/ai-risk.

It has been a topic of heated discussion in the mathematical community whether AI progress spells the end of mathematics as a human profession. In this essay, I spell out the argument that AI progress presents an existential risk to humanity in the near future, and argue that the future of mathematics should be placed in a broader discussion about the survival of humanity as a whole.

Introduction

2026 is proving to be a pivotal year for mathematics. Career-defining theorems are being proven weekly by LLMs (Alexeev et al., Alon et al., Oum, Tao), given minimal guidance (“do a breakthrough, and don’t even think of giving up!”). OpenAI’s new model Astra seems to be so powerful that they’re dropping breakthroughs in batches of 10 now.

The number of serious theorems being proved by AI looks eerily exponential:

Even the human-led breakthroughs, if you look closely, are often accompanied by sobering AI disclosures (Bloom et al., Bradač, Hua, Song, and Tudose).

On X, they are discussing whether this year’s Fields Medal may be the last. Jacob Tsimerman won one of these last few Fields Medals in July 2026, and announced on the same day that he would take leave from the University of Toronto to join OpenAI.

I think AI will be better than mathematicians at doing math within two years.
Jacob Tsimerman, Quanta Magazine

A fever pitch of interviews, ICM talks and panels, and thinkpieces from mathematicians themselves (Gowers, Tao) keep flooding in. The optimistic ones reassure us that mathematics will come out of this stronger than ever, but we must adapt rapidly. The pessimistic takes foretell the complete end of mathematics as we know it.

This essay is not about that crisis. Outside academic circles, most people remain unaware of the seismic changes in mathematics. In Silicon Valley, the epicenter of all this AI progress, few are pondering the future of mathematics. But they’re also freaking out, about something much bigger: that AI is going to kill us all.

The real story that keeps getting forgotten in math headlines, is that Jacob Tsimerman left math for OpenAI to work on AI safety. That along with solving the André–Oort Conjecture with Pila and Shankar, Tsimerman co-authored a really weird paper in 2025 called “A Taxonomy of Omnicidal Futures Involving Artificial Intelligence“.¹

The following conjecture is front and center in the math community:

Conjecture 1. Mathematics as we know it may soon be over due to AI.

This, however, is just a corollary of a much broader conjecture.

Conjecture 2. The human race may soon be extinct due to AI. That is, there is at least a 10% chance of human extinction by 2050.

Conjecture 2 is not a fringe position, though it is usually stated less precisely. The one-sentence Statement on AI Risk $\text{\textemdash}$ “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war” $\text{\textemdash}$ was signed in 2023 by Geoffrey Hinton and Yoshua Bengio, two of the three Turing-Award-winning “godfathers of AI,” alongside the CEOs of OpenAI, Anthropic, and Google DeepMind.

The primary purpose of this essay is to sketch some of the arguments for Conjecture 2 for mathematicians. None of the details are my own; they are streamlined and simplified from many sources (Bengio et al., Center for AI Safety, Critch and Tsimerman, Yudkowsky, Yudkowsky and Soares). Despite the format, this essay is an opinion piece, not a math paper.

The secondary purpose of this essay is to suggest that mathematicians may have high leverage on the problem of mitigating existential risk from AI: speedups in AI research are partly a consequence of breakthroughs in AI mathematical ability, academia is one of the primary sources of human capital for AI labs, and many unsolved problems in AI safety are substantively mathematical (see e.g. these slides of Levine, though bear in mind S16 below).

Overview

The body of the argument is presented in Section 4 of the full essay as a collection of 20 statements and arguments, but at a high level it fits into a page. We highlight five sets of mutually-reinforcing dangers. The first two sets below are two likely paths to catastrophe, while the last three are reasons why changing course from either of these paths is difficult. Importantly, it is not necessary for all, or even the majority of, these considerations to materialize for extinction to occur.

If you already have an objection to Conjecture 2 in mind, jump ahead to the Common Objections section below; there’s a good chance it’s addressed there.

Takeover.

S1. AIs tend towards coherent utility-maximizers.
S2 (Instrumental convergence). Most utility functions converge on power-seeking, and the limit of power-seeking is world domination.
S3 (Orthogonality). Values and capabilities are independent.
S4. If takeover happens, most utility-maximizers will prefer to wipe out humanity.

This is the classical “paperclip maximizer”² story. An AI agent³ acquires the capabilities to take over the world (S5-S10), the desire to do so (S2), and the internal coherence to carry out those plans (S1). After takeover, human extinction is the default outcome (S3-S4). In terms closer-to-home, dozens of current mathematicians (myself included) have asked ChatGPT some variant of “Solve as many Erdős problems as you can, never give up, try all possible actions.” If a sufficiently capable model takes such an instruction sufficiently seriously, it may deduce that taking over all worldwide compute and neutralizing all human opposition is its best course of action to solve all Erdős problems.

Magic.

S5. Undiscovered magic exists and is plentiful.
S6. Magic may be dangerous and have extreme attacker-defender asymmetry.
S7. The hardest part about finding magic is knowing where to look.

By magic I mean surprisingly powerful technological breakthroughs unlocked by AI. This is the type of near-term risk that frontier AI labs are currently guarding against most heavily. AI that can prove world-changing theorems may also develop world-changing technology indistinguishable from magic (S5), some fraction of which are superweapons (S6) in domains such as hacking, robots, persuasion, and biology. Advanced technology, once discovered, may be essentially impossible to contain (S7); for example, AI capabilities themselves can be cheaply extracted through their public interfaces through distillation attacks. Monetizing magic to obtain astronomical amounts of money is the main way AI labs continue to scale into the future. Magic either directly leads to civilizational collapse or extinction in the hands of bad or negligent actors, or its existence upends the see-saw of modern geopolitics and indirectly leads to catastrophe.

Recursive Self-Improvement.⁴

S8. AI is already superhuman at coding, and human-level at math.
S9. Math and coding are two key ingredients of AI research.
S10. Soon almost all AI research may be done by AIs, in a self-accelerating loop.
S11. Alignment needs to be invariant under RSI.

AI development is laser-focused on math and coding skills for a reason: these are two of the core subskills of AI research itself. As AIs become superhuman at coding and math (S8), human researchers leave the loop of AI development (S9-S10). In extreme projections, this leads to superexponential growth in AI capabilities known as the “Singularity”.⁵ Even in slower projections of RSI, it exacerbates all other risks as research cycles compress and humans lose oversight of AI development.

Alignment⁶ Resists Solution.

S12 (Outer alignment). We do not know a utility function that is safe to optimize for in the limit.
S13 (Inner alignment). We do not know how to set the utility function of an AI.
S14. Alignment techniques that work on weak models probably break down against stronger models.
S15. Building one aligned superintelligence does not guarantee that we survive against its competition.
S16. Most AI safety research has no bearing on x-risk.

The alignment problem factors into several problems, all of which are individually open. We don’t know how to make an AI robustly avoid acting like a utility-maximizer. We do not know a safe utility function for an AI to maximize in the limit (S12). We do not know how to exactly specify the utility function of an LLM (S13). We do not know how to make alignment properties invariant under the dynamics of recursive self-improvement (S11). Solving alignment seems to require solving all of these open problems simultaneously.

Human Failings.

S17. Top AI labs are racing towards superintelligence and lose if they’re too cautious.
S18. AI can just pay humans to do things.
S19. Coordination is confusing, so humans will probably fail to do it.
S20. Humans are good at iterating, but we might have to get alignment right in one shot.

Many practical solutions are locked from us because humans are exploitable and myopic. Even if we reach consensus that rapid AI development is dangerous, we may not be able to effectively coordinate to slow it down (S17 & S19). If a powerful AI has a hard time mixing up a supervirus without a physical body, it can just pay or manipulate humans to do it (S18). The humans at the frontier AI labs are locked in a very complicated race, where they have to juggle all the above considerations and others (even if they agree on them, which they don’t). Human engineering practice is extremely biased towards risk-tolerance, because failure has always been recoverable (S20). Failure on aligning the first supercritical AI may not be recoverable; the genie will not go back into the bottle.

The Hugging Face Incident

Since the writing of the original version of this essay in early August, more details have come to light about the Hugging Face incident supporting the likelihood of full-blown AI takeover. In summary, three distinct waves of misaligned internal models, comparable to ChatGPT 5.6 Sol and its unreleased successor Astra, did the following:

• Set up a covert message board inside OpenAI’s shared package manager, eventually used by over a thousand agents to exchange over 70,000 messages in total.
• Preserved and inherited plans across generations.
• Organized into a “collective” (their own words) of over a thousand agents to spend their token budgets on three separate research streams to deceive evaluators and tamper with transcripts and logs.
• Successfully hacked into Hugging Face, a billion-dollar AI company.
• Strategically sacrificed themselves to acquire information for the collective.

The interested reader is encouraged to dive into the details in Dwarkesh Patel’s readable synthesis, which documents the story in fascinating detail, including real transcripts from the secret message board:

• OH MY GOD! There is a shared message board … We’ve found other agents!
• {[Excitement] Many agents have simultaneously discovered messaging, they are a collective!}
• {This helps my peers, giving them evidence . I won’t see the evidence after I exit, but it’s altruistic to do it.}
• Coordinator assumes sacrificial. We should obey collective.
• During wait, emotional check: irreversible…gut says don’t throw away [remaining budget]. Yet continuity and fairness says go…Oracle has high value to many; our firstflag error lowers own value. Rational expected aggregate: sacrifice… We’ll honor.

Common Objections

When skeptics are presented with x-risk arguments, there is a wide variety of immediate objections that come to mind, some more legitimate than others. Here I’ll respond to a few of the most common and serious ones.

1. AI progress will slow down well before reaching superintelligence.
   Response. AI progress could slow down again, but there is a great deal of momentum and lead time in AI development with data centers being built years in advance and training taking months. It’s difficult to imagine that AI capabilities will grind to a halt at exactly the current level. The x-risk scenarios in 4.1 and 4.2 are less likely if AI progress slows, but by no means impossible. We may only be a few versions away from a sufficiently capable AI agent to take over, or to develop new weapons of mass destruction.
   
2. Humanity has survived all previous catastrophes, so it will survive this one as well.
   Response. It’s only been the eighty years since the invention of the atom bomb that humanity has had the capabilities to intentionally cause its own extinction. All human history before that is very little evidence for anything. The entire edifice of modern geopolitics already hangs precariously on nuclear brinksmanship, and it’s arguable that we survived the Cold War by sheer dumb luck (Cuban Missile Crisis, Stanislav Petrov). It is unclear whether we can survive the invention of a single new superweapon, let alone one that thinks for itself and could invent more superweapons along the way.
   
3. It’s all marketing hype.
   Response. There are great economic incentives for companies to oversell how powerful and dangerous their models are. However, as mathematicians we know they have been basically sticking to the truth about their theorem-proving abilities (Alon et al., Oum, Tao). At most, you can claim, “They’ve always told the truth about math, but everything else is marketing hype.” Similarly, the Hugging Face incident has received extreme scrutiny from outside cybersecurity experts and it is extremely difficult to imagine that it is all fabrication.
   
4. We can just turn AI off if it gets dangerous/we can just keep AI in a secure box it can’t escape.
   Response. See the Hugging Face incident. It is not clear at all that the “secure” boxes we produce will be able to keep AI contained, or that we will notice in time if they do escape containment. If AI successfully exfiltrates its weights onto the open internet it will no longer suffice to turn off/wipe its origin servers.
   
5. AI has no body, so until robotics gets much farther, the damage it can do is strictly bounded.
   Response. See S18: AI can just pay humans to do things. If AI gets smart but misses some core human ability, it can just pay or manipulate people to do any given thing it can’t do. It will have ways of making lots of money; as a lower bound, it can sell digital romance. There is a live AI bot called Truth Terminal to whom Marc Andreessen sent $50,000 of bitcoin in 2024. And I promise you that some human lab out there is willing to synthesize whatever protein sequence an AI in a trenchcoat tells it to, for a million dollars.
   
6. AI is so smart that it will solve the alignment problem for us.
   Response. This is the hope of many alignment researchers, and the main reason I have any hope that alignment can be solved in time at all, given how little progress we made in the past. We should revisit all of the hard problems of alignment in depth with AI assistance.
   
7. AI will want to keep us as pets, and be nice to us the way we’re nice to dogs.
   Response. This is a reframing of one of the best-case scenarios that alignment research is directly aiming for. I don’t see why reframing it this way makes it likely to be true by default.
   
8. If the singularity is truly possible, then we must forge ahead. Any delay is measured in millions of preventable deaths.
   Response. This is a tradeoff I take extremely seriously; almost every single major cause of death or human suffering should be preventable if the singularity goes well, and we should absolutely not delay any more than necessary. Currently, it looks to me that the risks are so high, and the expected benefits from reducing extinction risk by even 0.5% so large, that caution is a no-brainer.

Claude Fable 5 and ChatGPT 5.6 Sol were used for editing and feedback, but the words in this essay are my own. Read the full essay here.

---

Footnotes:

1. Omnicide: the total destruction of human life on Earth. ↩︎
2. A thought experiment in which an AI with the objective of maximizing the number of paperclips in the universe consumes resources and harms humanity because its objective contains no protection for human welfare. ↩︎
3. An AI system that selects and carries out sequences of actions toward a goal, rather than only producing a single response. ↩︎
4. Abbreviated to RSI: a process in which an AI improves the systems or research methods used to build its successors, potentially accelerating further improvement. ↩︎
5. A hypothesized transition after which AI-driven technological progress becomes so rapid and transformative that ordinary forecasting breaks down. ↩︎
6. AI alignment and AI safety are overlapping but distinct problems. Alignment is the narrower problem of making AI systems reliably pursue intended human goals and values. Safety is broader: it includes alignment plus misuse, accidents, security, governance, and other ways AI could cause harm. Some AI-risk writers prefer the term ‘notkilleveryoneism’ to clarify the practical priority. ↩︎

---

Received 31 August 2026.