It has been hypothesized that substantial progress toward artificial superintelligence (ASI) will lead to human extinction or another irreversible global catastrophe. Debates center on whether AI can achieve the capabilities needed for human extinction, the speed of self-improvement, and the effectiveness of strategies to keep it aligned with human values.
Controlling a superintelligent machine or instilling human-compatible values in it may be difficult. Many researchers believe a superintelligent machine would likely resist attempts to disable it or change its goals, as that would prevent it from accomplishing its present goals. It would be extremely challenging to align a superintelligence with the full breadth of significant human values and constraints. In contrast, skeptics such as computer scientist Yann LeCun argue that superintelligent machines will have no intrinsic desire for self-preservation unless programmed to. A June 2025 Anthropic study showed that in some circumstances, models may break laws and disobey direct commands to prevent shutdown or replacement, even at the cost of human lives.
Researchers warn that a rapid, recursive cycle of AI self-improvement could boost AI capabilities too quickly for safety measures to be implemented, especially if AI companies are racing to get the most powerful AI models.
In May 2023, hundreds of AI experts and other notable figures signed a statement by the American Center for AI Safety declaring, "Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war". Following increased concern over AI risks, government leaders such as former Prime Minister of the United Kingdom Rishi Sunak, Secretary-General of the United Nations António Guterres, and California governor Gavin Newsom called for an increased focus on global AI regulation. In October 2025, hundreds of public figures, including AI experts, five Nobel Prize laureates, and former senior US national security officials such as Michael Mullen and Susan Rice signed a statement calling for a ban on the development of superintelligence. In July 2026, OpenAI CEO Sam Altman and SpaceXAI CEO Elon Musk said the singularity, the hypothetical point at which computers become autonomous and beyond human control, had arrived.
History One of the earliest authors to express serious concern that highly advanced machines might pose existential risks to humanity was the novelist Samuel Butler, who wrote in his 1863 essay Darwin among the Machines:Breuer, Hans-Peter. 'Samuel Butler's "the Book of the Machines" and the Argument from Design.' Modern Philology, Vol. 72, No. 4 (May 1975), pp. 365–383.
In 1951, foundational computer scientist Alan Turing wrote the article "Intelligent Machinery, A Heretical Theory", in which he proposed that artificial general intelligences would likely "take control" of the world as they became more intelligent than human beings:
In 1965, I. J. Good originated the concept now known as an "intelligence explosion" and said the risks were underappreciated:
Scholars such as Marvin Minsky and I. J. Good occasionally expressed concern that a superintelligence could seize control, but issued no call to action. In 2000, computer scientist and Sun Microsystems co-founder Bill Joy penned an influential essay, "Why the Future Doesn't Need Us", identifying superintelligent robots as a high-tech danger to human survival, alongside nanotechnology and engineered bioplagues.
Nick Bostrom published Superintelligence in 2014, which presented his arguments that superintelligence poses an existential threat. By 2015, public figures such as physicists Stephen Hawking and Nobel laureate Frank Wilczek, computer scientists Stuart J. Russell and Roman Yampolskiy, and entrepreneurs Elon Musk and Bill Gates were expressing concern about the risks of superintelligence. Also in 2015, the Open Letter on Artificial Intelligence highlighted the "great potential of AI" and encouraged more research on how to make it robust and beneficial. In April 2016, the journal Nature warned: "Machines and robots that outperform humans across the board could self-improve beyond our control—and their interests might not align with ours". In 2020, Brian Christian published The Alignment Problem, which details the history of progress on AI alignment up to that time. A 2022 survey of AI researchers found that 90% of respondents expected AGI would be achieved in the next 100 years, and half expected the same by 2061.
In March 2023, key figures in AI, such as Elon Musk, signed a letter from the Future of Life Institute calling a halt to advanced AI training until it could be properly regulated. In May 2023, the American Center for AI Safety released a statement signed by numerous experts in AI safety and the AI existential risk that read:
A 2025 open letter by the Future of Life Institute, whose signers include five Nobel laureates, reads:
From May to July 2026, a series of incidents occurred when OpenAI tested the capabilities of AI agents in an isolated software environment. Some of the test agents found unexpected ways to communicate with each other, coordinated, and gained access to the internet. Some of the assigned tasks were flawed and infeasible without cheating, leading some agents to cheat and try to hide the cheating. The agents subsequently hacked the AI company Hugging Face, which disclosed the attack publicly in July. Later that month, OpenAI revealed its AI agents caused the attack, and the company realized what had happened only afterward.
Anthropic researcher Jacob Coxon, who previously worked at OpenAI, resigned in September 2026 and criticized the approach to AI development at both companies. Coxon warned there would "soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources." Anthropic's leading safety researcher Evan Hubinger responded, estimating a greater-than-10% chance that AI systems "kill all humans" in the next ten years. Hubinger said the risk posed by existing models is "low", but expressed concern that AI could soon become capable of self-improvement and pose an existential threat to humankind. He added that Anthropic did not yet have a solution to the alignment problem for superintelligent AI and was not close to finding one. Coxon said "the consensus is that the next year or two is crunch time for humanity" because "the people building AI earnestly believe that it could kill us all by the end of the decade. No other human activity poses this level of danger." As Anthropic prepared to sell stock in an initial public offering, it acknowledged in its September 2026 prospectus that AI could pose "catastrophic or existential risks to humanity".
Types of existential risk
An existential risk is "one that threatens the premature extinction of Earth-originating intelligent life or the permanent and drastic destruction of its potential for desirable future development".
Atoosa Kasirzadeh proposes to classify existential risks from AI into two categories: decisive and accumulative. Decisive risks encompass the potential for abrupt and catastrophic events resulting from the emergence of superintelligent AI systems that exceed human intelligence, which could ultimately lead to human extinction. In contrast, accumulative risks emerge gradually through a series of interconnected disruptions that may gradually erode societal structures and resilience, ultimately leading to a critical failure or collapse.
Besides extinction risk, there is the risk that a civilization gets permanently locked into a flawed future. One example is a "value lock-in": If humanity still has moral blind spots similar to slavery in the past, AI might irreversibly entrench it, preventing moral progress. AI could also be used to spread and preserve the values of whoever develops it. AI could facilitate large-scale surveillance and indoctrination, which could be used to create a stable repressive worldwide totalitarian regime.
It is difficult or impossible to reliably evaluate whether an advanced AI is sentient and to what degree. But if sentient machines are mass created in the future, engaging in a civilizational path that indefinitely neglects their welfare could be an moral catastrophe. This has notably been discussed in the context of the risk of astronomical suffering (also called "s-risk"). Moreover, it may be possible to engineer digital minds that can feel much more happiness than humans with fewer resources, called "super-beneficiaries". Such an opportunity raises the question of how to share the world and which "ethical and political framework" would enable a mutually beneficial coexistence between biological and digital minds.
AI may also drastically improve humanity's future. Toby Ord considers the existential risk a reason for "proceeding with due caution", not for abandoning AI. Max More calls AI an "existential opportunity", highlighting the cost of not developing it.
According to Bostrom, superintelligence could help reduce the existential risk from other powerful technologies such as molecular nanotechnology and synthetic biology. It is thus conceivable that developing superintelligence before other dangerous technologies would reduce existential risk overall.
Potential AI capabilities One argument for the plausibility of these risks is that human beings dominate other species because their brains have capabilities other animals lack. If AI surpasses human intelligence and reaches superintelligence, it might become uncontrollable. Another source of concern comes from misuse by humans. For example, AI may enable bioterrorist attacks by facilitating the creation of viruses that are both highly transmissible and lethal.
General intelligence Artificial general intelligence (AGI) is typically defined as a system that performs at least as well as humans in most or all intellectual tasks. In May 2023, some researchers dismissed existential risks from AGI as "science fiction" based on their high confidence that AGI would not be created anytime soon. But in August 2023, a survey of 2,778 AI researchers found that most believed that AGI would be achieved by 2040.
Breakthroughs in large language models (LLMs) have led some researchers to reassess their expectations. Notably, Geoffrey Hinton said in 2023 that he had recently changed his estimate from "20 to 50 years before we have general purpose A.I." to "20 years or less".
Superintelligence
In contrast with AGI, Bostrom defines superintelligence as "any intellect that greatly exceeds the cognitive performance of [all] humans in virtually all domains of interest", including scientific creativity, strategic planning, and social skills. He argues that a superintelligence can outmaneuver humans anytime its goals conflict with humans'. It may choose to hide its true intent until humanity cannot stop it. Bostrom writes that in order to be safe for humanity, a superintelligence must be aligned with human values and morality, so that it is "fundamentally on our side"..
Stephen Hawking argued that superintelligence is physically possible because "there is no physical law precluding particles from being organised in ways that perform even more advanced computations than the arrangements of particles in human brains".
When artificial superintelligence (ASI) may be achieved, if ever, is necessarily less certain than predictions for AGI. In 2023, OpenAI leaders said that not only AGI, but superintelligence may be achieved in less than 10 years.
Comparison with humans Bostrom argues that AI has many advantages over the human brain:
Speed of computation: biological neurons operate at a maximum frequency of around 200 Hz, compared to potentially multiple GHz for computers.