AI Boom Drives DRAM Shortage & Skyrocketing Memory Prices | IEEE Spectrum
- The relentless advance of artificial intelligence is creating a ripple effect throughout the technology industry, and nowhere is that more acutely felt than in the market for computer...
- While the largest AI hardware companies have reportedly secured chip supplies well into the future – some as far out as 2028 – the rest of the industry,...
- At the heart of this supply crunch lies high-bandwidth memory, or HBM.
The relentless advance of artificial intelligence is creating a ripple effect throughout the technology industry, and nowhere is that more acutely felt than in the market for computer memory. Demand for DRAM, the type of memory crucial for powering GPUs and other AI accelerators in data centers, has surged to unprecedented levels, diverting supply from other sectors and driving prices sharply higher. DRAM prices have risen 80-90 percent this quarter, according to Counterpoint Research.
While the largest AI hardware companies have reportedly secured chip supplies well into the future – some as far out as – the rest of the industry, including PC manufacturers and consumer electronics producers, are facing a scramble to secure scarce resources at inflated prices. The situation isn’t simply a matter of increased demand; it’s a collision of the DRAM industry’s historically cyclical nature and an AI infrastructure build-out unlike anything seen before.
The Rise of High-Bandwidth Memory (HBM)
At the heart of this supply crunch lies high-bandwidth memory, or HBM. Developed as a way to circumvent the slowing pace of Moore’s Law, HBM utilizes advanced 3D chip packaging technology. Each HBM chip consists of up to 12 layers of thinned-down DRAM dies, interconnected by thousands of vertical connections called through-silicon vias (TSVs) and microscopic solder balls. This stacked structure, roughly 750 micrometers thick, is then placed close to the GPU or AI accelerator, linked by as many as 2,048 micrometer-scale connections.
The goal of this tightly integrated design is to overcome what’s known as “the memory wall” – the bottleneck in energy and time required to deliver the massive amounts of data needed to run large language models (LLMs). Memory bandwidth is a critical factor limiting the speed at which LLMs can operate.
HBM technology has been evolving for over a decade, with DRAM manufacturers continually pushing its capabilities. However, its increasing importance to GPUs, driven by the growth of AI models, has come at a significant cost. SemiAnalysis estimates that HBM generally costs three times as much as other types of memory and now accounts for 50 percent or more of the total cost of a packaged GPU.
A History of Boom and Bust
The DRAM industry is known for its cyclical patterns of boom and bust. Building new fabrication plants (fabs) requires massive investments – upwards of $15 billion – making companies hesitant to expand capacity unless during periods of high profitability. Once a new fab is operational, it typically takes 18 months or more, often resulting in a surge of supply that floods the market and depresses prices.
The current cycle’s origins can be traced back to the chip supply panic surrounding the COVID-19 pandemic. Hyperscalers – large data center operators like Amazon, Google, and Microsoft – aggressively purchased memory and storage to avoid supply chain disruptions and support the shift to remote work, driving up prices. However, as supply normalized and data center expansion slowed in , prices plummeted, continuing into . Some companies, like Samsung, even cut production by as much as 50 percent in an attempt to stabilize prices – a rare and desperate measure.
Following a recovery that began in late , memory and storage companies were wary of reinvesting in new production capacity. “Thus there was little or no investment in new production capacity in and through most of ,” explains Thomas Coughlin, a storage and memory expert and president of Coughlin Associates.
The AI Data Center Boom
This lack of investment is now colliding with a massive surge in demand fueled by the construction of new data centers. Globally, nearly 2,000 new data centers are currently planned or under construction, representing a potential 20 percent increase in the existing global supply of around 9,000 facilities. McKinsey predicts that companies will spend $7 trillion by , with $5.2 trillion dedicated to AI-focused data centers.
Nvidia has been the primary beneficiary of this boom. Its data center revenue soared from barely $1 billion in the final quarter of to $51 billion in the quarter ending . This growth has driven demand not only for more DRAM but also for an increasing number of DRAM chips per GPU. Nvidia’s recently released B300 GPU uses eight HBM chips, each containing a stack of 12 DRAM dies, a trend mirrored by competitors like AMD.
HBM is becoming an increasingly significant portion of DRAM manufacturers’ revenue. Micron reported that HBM and other cloud-related memory went from 17 percent of its DRAM revenue in to nearly 50 percent in .
Micron predicts the total market for HBM will grow from $35 billion in to $100 billion by – exceeding the entire DRAM market size in . CEO Sanjay Mehrotra stated that the company now expects to reach that figure two years earlier than previously anticipated, and that demand will substantially outstrip supply “for the foreseeable future.”
Future Outlook and Technological Advancements
Addressing the supply issues requires both innovation and increased manufacturing capacity. Mina Kim, an economist with Mkecon Insights, explains that while DRAM scaling has become more challenging, the industry is turning to advanced packaging techniques – essentially using more DRAM – as a workaround.
Micron, Samsung, and SK Hynix are all investing in new fabs and facilities. Micron is building an HBM fab in Singapore, expected to be operational in , and retooling a fab in Taiwan, also slated for production in the second half of . It also broke ground on a DRAM fab complex in New York, but full production isn’t expected until . Samsung plans to start production at a new plant in South Korea in , and SK Hynix is building HBM and packaging facilities in Indiana and an HBM fab in Cheongju, both expected to be complete in and respectively.
Despite these investments, Intel CEO Lip-Bu Tan recently stated at the Cisco AI Summit that “there’s no relief until .”
Beyond capacity expansion, incremental improvements in yield, tighter coordination between memory suppliers and AI chip designers, and advancements in packaging technologies will be crucial. Technologies like advanced packaging, hybrid bonding, and SK Hynix’s MR-MUF process are aimed at improving heat conduction and enabling higher-density stacking of DRAM dies. The HBM4 standard, capable of accommodating 16 stacked DRAM dies, represents a further step in this direction.
However, even with these advancements, economists caution against expecting a rapid decline in prices. “In general, economists find that prices come down much more slowly and reluctantly than they go up,” Kim notes. “DRAM today is unlikely to be an exception to this general observation, especially given the insatiable demand for compute.”