Tech • AI • Robotics • Game

ENFR

This physical limit will bring down Nvidia

AIUnderscore_November 20, 2026 at 04:11 PM45:02
Audio player

⏱ ≈ 3 min read · 45-min video — 42 min saved

TL;DR

Nvidia dominates AI data centers through a mix of software lock-in, supply power and commercial guarantees, but a new wave of specialized chips is exposing limits in GPU design for fast token generation and opening space for challengers.

KEY POINTS

How Nvidia built its lead

Nvidia’s advantage began in the 2000s when graphics chips designed for pixel shading were repurposed for massively parallel computing. The key turning point came with CUDA around 2006-2007, which turned an experimental practice into a serious software platform. Crucially, CUDA worked across consumer and enterprise cards, letting students, researchers and startups learn on gaming hardware and later scale inside companies.

Software inertia became market power

AI frameworks such as PyTorch grew up around CUDA, reinforcing Nvidia’s position. That created a self-feeding loop: developers optimized for Nvidia because that was what they had, and data centers bought Nvidia because the software stack was already there. The result is not only technical dominance but cultural familiarity across the industry.

Data center economics now favor Nvidia by default

The company’s strength is no longer just software compatibility. Operators say Nvidia can effectively guarantee demand for new clusters, including commitments around utilization that de-risk multibillion-dollar deployments. In a market still constrained by compute shortages, the easiest chips to source at scale are often Nvidia’s, which further entrenches the status quo.

The AI bottleneck is shifting from compute to memory

For large language models, generating a token often requires reading the full model’s weights. That makes memory bandwidth a central limit, especially during decoding. A dense model with 100 billion parameters stored in 8-bit precision is roughly 100 GB; with 1 TB/s of memory bandwidth, the physical ceiling is about 10 tokens per second before software overheads. For frontier-scale models, the pressure is far greater.

Why agents make latency more important

In chatbot use, token speed mainly had to satisfy human reading speed. With autonomous agents, the bottleneck becomes machine-to-machine loops: generate, act, retrieve, generate again. In that setting, token latency directly affects how quickly tasks complete, making decoding performance much more valuable than it was in simple chat interfaces.

GPUs remain strong, but not everywhere

GPUs are highly effective at the prompt processing stage, where many operations run in parallel. They are less naturally suited to the decoding phase, where memory access dominates and compute units can sit idle waiting for data. That is where Nvidia’s general-purpose design starts to look less optimal than chips built specifically for inference throughput or low-latency token generation.

Groq and Cerebras attack the memory problem differently

New challengers are trying to beat GPUs by moving much faster data closer to compute. Groq packs large amounts of SRAM onto its chips, while Cerebras pushed the idea much further with wafer-scale processors that use an entire silicon wafer as one giant chip. A recent Cerebras design combines about 900,000 compute cores with roughly 44 GB of on-chip SRAM and around 150 TB/s of memory bandwidth, enabling extremely high token rates.

The trade-off is cost and complexity

These architectures deliver striking performance, but they are expensive and difficult to deploy. Systems can cost on the order of tens of millions of dollars to run a single large model, and they bring major engineering challenges in manufacturing yield, signal integrity and cooling. That makes them attractive for specific premium use cases rather than an immediate mass-market replacement for Nvidia.

A French effort is targeting the lock-in layer

French startup ZML is taking aim at a different barrier: software dependence on one vendor. Its machine learning stack is designed to run across multiple chip families, including Nvidia, AMD, Google TPU, AWS Trainium, Intel, Apple Metal, Vulkan and Tenstorrent. The goal is to preserve high performance while lowering the cost of switching hardware.

Breaking Nvidia requires demand, not just better chips

Faster or cheaper silicon alone may not be enough. Even when alternatives offer better price-performance, buyers face operational inertia, vendor relationships and supply risks. The likelier path to change is a broader demand for sovereignty, lower costs, better efficiency and more supplier freedom, combined with software layers that make those alternatives practical to use.

CONCLUSION

Nvidia remains the default supplier for AI infrastructure, but its dominance is no longer unchallenged at the architectural level. The next battle will hinge not only on who builds the fastest chip, but on who can pair performance with availability, software compatibility and buyer confidence.

Ask a question

More from AI