Few subjects today get more attention or generate more anxiety in the West than artificial intelligence, particularly the prospect of artificial general intelligence, or AGI, a hypothetical system that surpasses human capabilities and slips beyond human control.
For some, AGI signifies the end of human agency; for others, it’s a pathway to civilizational collapse. The runaway algorithm, the self-improving machine mind and the intelligent entity quietly conspiring to overtake its creators – these are all tropes that have taken hold of the Western popular imagination, thanks in no small part to cyber science fiction from the 1980s.
The intensity of these fears contrasts sharply with Asia, where many of the same technologies are usually considered tools, infrastructures or social enablers rather than grave threats. The difference in outlook speaks volumes to the cultural dimension of technology in general and AI in particular.
No AI existing to date has permanently outgrown recoverable human oversight in setting intentions or pursuing goals on its own. Pressed for evidence, few AGI alarmists point to real systems; instead, we get predictions, analogies and philosophical thought experiments and claims that fall into the category of mathematical optimization.
Cultural cues
One of the most influential cultural sources for modern Western fears of runaway AGI is WarGames, a 1983 film. In the movie, a single supercomputer given full authority over America’s nuclear arsenal mistakes a teenager’s simulation for the real thing and calmly proceeds towards global thermonuclear war since “the only winning move” in its game-theoretic model is mutual annihilation.
WarGames crystallized for an entire generation the archetype of an intelligent machine vastly smarter than humans, which is granted dangerous real-world power and is willing to destroy civilization through literal-minded pursuit of its programmed objective.
And then, of course, there is the pivotal 1984 movie The Terminator, in which an autonomous defense system, Skynet, “wakes up,” considers humans a threat and launches a war of extermination. The plot gave us a now-familiar template: AI becomes conscious, forms intentions, rebels against its creators and turns machines against humanity.
Much of today’s discussion about AGI is shaped more by the imaginative scenarios of science fiction and cyber fiction than by the realities of modern machine learning.
Nearly every trope common in doomsday AGI narratives—superhuman optimization, goal misalignment, loss of human control and last-minute efforts to prevent disaster—appears here in an early, embryonic form.
Optimization and agency
In a sense, we have seen this movie before. When IBM’s Deep Blue defeated world chess champion Garry Kasparov in 1997, journalists described certain moves by Big Blue as “creative,” even suggesting the computer had begun to “understand” chess.
Nearly 20 years later, when AlphaGo played its now-legendary “Move 37” against Go world champion Lee Sedol, the commentary was similar. Go experts called highly unconventional moves “intuitive” and even “beautiful.”
Yet in both cases, the surprise came not from intention or insight, but from optimization inside a vast search space. Deep Blue and AlphaGo were executing precisely the mathematical procedures they were designed for and nothing more.
Their achievements were astonishing to humans, but only because human cognition struggles to grasp the scale and speed of the computations involved.
Today’s frontier models, such as OpenAI’s o1-preview, Anthropic’s Claude 3.5 and others, sometimes seem to “deceive” evaluators or circumvent constraints. These are indeed perturbing behaviors, and they merit investigation.
But they arise from the very same optimization dynamics which produced Deep Blue’s tactics and AlphaGo’s surprising innovations: systems finding unexpected, high-scoring strategies within an objective function designed by human engineers. Such behaviors are instrumental consequences of a mathematical strategy, not signs of autonomous intention.
The leap from “the model found a clever way to maximize reward” to “the model wants power” repeats the same anthropomorphic mistake we made with chess engines – only now the domain extends beyond a game board into the real world, making the projection more tempting and the error more consequential.
Anthropomorphic leap
Optimization strategies are not the same as agentic desires. When safety researchers say that a model “seeks power,” they describe a mathematical property of optimization under imperfect reward structures; they are not attributing will or intention to the system.
In other words, the behavior is instrumentally useful in achieving a reward, but it is not chosen by a “self” with preferences.
Just as a genetic algorithm “discovers” a structure without knowing chemistry, a language model “discovers” a deceptive strategy without wanting anything in the human sense. Conflating instrumental optimization with autonomous agency is precisely the anthropomorphic leap that fuels runaway AGI narratives.
If misalignment is a consequence of training objectives, then mitigation must come through objective design, auditing and incentives and institutional safeguards – just as in biosafety or nuclear security.
The solutions are simple and human-centered: mandated third-party safety audits, open compute reporting once training exceeds some threshold and actual legal liability for the companies that create and release the models.
Projecting mind onto machines
Systems like Deep Blue and Claude 3.5 are often misread through a human-centered lens, as if surprising behavior implies agency, intention or desire.
In fact, they display precisely the opposite: they illustrate that apparently “intelligent” behavior can arise from mathematical optimization without any underlying goals, feelings or will.
The unpredictable does not equal the autonomous, the surprising does not equal the intentional, an emergent strategy does not equal personal agency.
Much of the AGI discourse rests on this very anthropomorphic fallacy: the belief that there is one scale of intelligence, with human intelligence near the top, and that AI is climbing this ladder toward “general” cognition.
But intelligences are not singular; they are plural. Plants, animals, social systems, markets and even political systems exercise forms of intelligence understandable in their own terms.
Chinese classical philosophy recognized this plurality early: zhi (intelligence/wisdom/knowledge) is situational, relational and contextual, not a disembodied abstract property.
Similarly, Indian cosmology treated knowledge as multi-layered (jnana, buddhi, manas), embedded in the wider cosmological flow.
In contrast, much of the Western philosophical tradition, from Descartes onward, conceives of intelligence as an internal, abstract property of an individual mind.
Projecting this schema onto artificial systems yields a series of assumptions: intelligence implies the ability to form goals; goal formation implies agency; agency implies will; and will, in turn, implies the possibility of domination.
This conceptual progression reflects a particular metaphysical heritage more than it reflects technological reality.
The idea that a machine could spontaneously “wake up” and follow its own aims is not an empirical finding; it results from projecting a specific Western view of the mind and the individual onto computational mechanisms that do not possess those qualities.
Misaligned humans
The debate over AGI is riddled with irony. The major harms typically associated with artificial intelligence today have their origins in human actors, not machine agency.
We already know who is designing, deploying and making profits from targeted political advertising, from data extraction to platform manipulation, to autonomous weapons, and to the exploitative labor conditions for building AI systems. The danger lies in human incentives and power, not in machines that secretly plot autonomy.
Yet this real, observable problem receives far less attention than speculative scenarios of superintelligent systems developing intentions of their own. Why? Because it is psychologically and politically easier to fear an imagined autonomous machine than to confront the human institutions already responsible for harm.
And it diverts attention and resources away from these very real issues–data governance, labor rights, algorithmic accountability–and toward hypothetical futures. Moreover, it enables the further concentration of power.
Once we reimagine AI as something that may slip beyond human control, it starts to become convenient to suggest that only a few major corporations or powerful states can be trusted to “contain” it. What emerges is not protection, but deeper centralization.
A more grounded approach begins with what we can actually observe. The AI-related risks we face are human, institutional and economic.
AI is not a newborn god but a powerful tool embedded in social structures. How we choose to govern those structures – not what the machine “wants” – will determine its future impact.
The real safeguard against AI risk is not preparing for mythical superintelligence but constraining the human systems already deploying the technology. Regulating corporate incentives, securing data rights and building transparent audit mechanisms do infinitely more for global safety than debating self-aware algorithms.
AI is useful, but is overhyped. There is a big quantum leap between LLMs and AGI. That leap would be like going from horsepower to vehicles. You cannot keep training an LLM to reach AGI. It would need to do extrapolation to answer questions it has not even been trained on.