
Tech • AI • Robotics • Game
⏱ ≈ 3 min read · 45-min video — 42 min saved
Nvidia dominates AI data centers through a mix of software lock-in, supply power and commercial guarantees, but a new wave of specialized chips is exposing limits in GPU design for fast token generation and opening space for challengers.
Nvidia’s advantage began in the 2000s when graphics chips designed for pixel shading were repurposed for massively parallel computing. The key turning point came with CUDA around 2006-2007, which turned an experimental practice into a serious software platform. Crucially, CUDA worked across consumer and enterprise cards, letting students, researchers and startups learn on gaming hardware and later scale inside companies.
AI frameworks such as PyTorch grew up around CUDA, reinforcing Nvidia’s position. That created a self-feeding loop: developers optimized for Nvidia because that was what they had, and data centers bought Nvidia because the software stack was already there. The result is not only technical dominance but cultural familiarity across the industry.
The company’s strength is no longer just software compatibility. Operators say Nvidia can effectively guarantee demand for new clusters, including commitments around utilization that de-risk multibillion-dollar deployments. In a market still constrained by compute shortages, the easiest chips to source at scale are often Nvidia’s, which further entrenches the status quo.
For large language models, generating a token often requires reading the full model’s weights. That makes memory bandwidth a central limit, especially during decoding. A dense model with 100 billion parameters stored in 8-bit precision is roughly 100 GB; with 1 TB/s of memory bandwidth, the physical ceiling is about 10 tokens per second before software overheads. For frontier-scale models, the pressure is far greater.
In chatbot use, token speed mainly had to satisfy human reading speed. With autonomous agents, the bottleneck becomes machine-to-machine loops: generate, act, retrieve, generate again. In that setting, token latency directly affects how quickly tasks complete, making decoding performance much more valuable than it was in simple chat interfaces.
GPUs are highly effective at the prompt processing stage, where many operations run in parallel. They are less naturally suited to the decoding phase, where memory access dominates and compute units can sit idle waiting for data. That is where Nvidia’s general-purpose design starts to look less optimal than chips built specifically for inference throughput or low-latency token generation.
New challengers are trying to beat GPUs by moving much faster data closer to compute. Groq packs large amounts of SRAM onto its chips, while Cerebras pushed the idea much further with wafer-scale processors that use an entire silicon wafer as one giant chip. A recent Cerebras design combines about 900,000 compute cores with roughly 44 GB of on-chip SRAM and around 150 TB/s of memory bandwidth, enabling extremely high token rates.
These architectures deliver striking performance, but they are expensive and difficult to deploy. Systems can cost on the order of tens of millions of dollars to run a single large model, and they bring major engineering challenges in manufacturing yield, signal integrity and cooling. That makes them attractive for specific premium use cases rather than an immediate mass-market replacement for Nvidia.
French startup ZML is taking aim at a different barrier: software dependence on one vendor. Its machine learning stack is designed to run across multiple chip families, including Nvidia, AMD, Google TPU, AWS Trainium, Intel, Apple Metal, Vulkan and Tenstorrent. The goal is to preserve high performance while lowering the cost of switching hardware.
Faster or cheaper silicon alone may not be enough. Even when alternatives offer better price-performance, buyers face operational inertia, vendor relationships and supply risks. The likelier path to change is a broader demand for sovereignty, lower costs, better efficiency and more supplier freedom, combined with software layers that make those alternatives practical to use.
Nvidia remains the default supplier for AI infrastructure, but its dominance is no longer unchallenged at the architectural level. The next battle will hinge not only on who builds the fastest chip, but on who can pair performance with availability, software compatibility and buyer confidence.
Ask a question