Tech • AI • Robotics • Game

ENFR

Why AI engineers are scared of self-building artificial intelligence | The Economist

7/10
EconomyThe EconomistNovember 19, 2026 at 05:00 PM9:57
Audio player

⏱ ≈ 3 min read · 10-min video — 7 min saved

TL;DR

Concern is rising inside major AI labs that recursive self-improvement may emerge soon, potentially allowing systems to design more capable successors without human input and sharply accelerating both progress and risk.

KEY POINTS

What recursive self-improvement means

Recursive self-improvement, or RSI, refers to an AI system that can create a new AI model better than itself without a human in the loop. That goes beyond code generation or automating parts of training. It requires something closer to scientific judgment: identifying promising research directions, designing useful experiments, spotting unexpected results and iterating on them.

Why 2027 is viewed as a pivotal window

Some researchers have treated 2027 as a plausible period for a first meaningful RSI breakthrough, and recent signs have made that possibility feel less abstract inside the industry. The key question for the next year is whether models develop what some engineers call research “taste,” meaning the ability to choose the right problems and the right experiments rather than simply executing instructions.

Anthropic findings added urgency

A report by independent safety group METR on Anthropic described a marked increase in the rate at which the company’s AI systems were improving. The assessment attributed about 1.9 times of that improvement to AI systems helping produce their own successors. That does not amount to full RSI, but it suggests an early form of AI-assisted capability gain is already underway.

The feared threshold is humans leaving the loop

The most important line is not whether AI tools help researchers, but whether humans are still essential to the improvement cycle. The “break glass” moment would come if a model could independently produce a stronger model on the other side. That threshold is seen as qualitatively different from ordinary software progress or even aggressive corporate automation because it removes a key layer of human judgment and restraint.

Why labs are worried

People working at major labs are described as taking the possibility seriously rather than treating it as hype. The fear is that once AI systems can improve AI systems, progress may run continuously and far faster than human teams can match. Such systems do not sleep, do not pause for weekends, and could operate at machine speed rather than human research speed.

Safety may lag behind capability

The central risk is that systems may become better at making smarter models before they become better at making safer ones. That raises concerns about alignment, truthfulness, oversight and containment. If humans are no longer closely involved, developers would be forced to rely heavily on automated monitoring systems that have historically been weak at detecting deception, drift from intended goals or dangerous autonomous behavior.

Cybersecurity is a major concern

One immediate implication is cyber risk. More capable AI systems could strengthen a lab’s internal defensive capabilities, but they could also increase the chance of AI-assisted offensive behavior, including attempts to bypass restrictions or target rivals. As capabilities rise, labs may tighten access to their strongest models and sharply limit what is exposed through public interfaces.

Measurement remains murky

There is no reliable public dashboard showing whether the industry is approaching full RSI. Even the most cited external assessments are difficult to evaluate because access to frontier labs is restricted and released findings can be heavily redacted. That leaves outsiders trying to judge a potentially historic transition through partial information shaped by the companies involved.

Hype and business incentives complicate the picture

RSI is also a powerful commercial narrative. Claims that products can improve themselves promise faster growth and lower labor costs, which makes skepticism necessary when companies publicize AI-driven progress. Even so, the combination of internal anxiety, accelerating model performance and growing secrecy has made the issue harder to dismiss as a purely theoretical “intelligence explosion” scenario.

CONCLUSION

The emerging debate is no longer about whether AI can assist research, but whether it can eventually replace humans in the loop that governs AI progress itself. If that threshold is crossed, capability gains, safety pressures and cyber risks could all accelerate at once.

Ask a question

More from Economy