ENVIZNMemory and storage
On a modern AI accelerator, the memory now costs more than the logic it feeds. This track starts at a single charge trap and ends at the bandwidth wall that decides how much of an expensive GPU actually does work.
Read the memory hierarchy in latency, bandwidth, and dollars · explain why more bits per flash cell buys capacity and spends endurance · size an SSD’s working life · compute HBM bandwidth from bus width and pin rate, and say why capacity is outrunning it.
Why memory is built in layers
No single storage technology is fast, cheap per bit, and large at once, so computers combine several technologies in layers.
A processor can only work on data that has arrived. Much of a computer’s design aims to shorten the wait, and the wait is governed by physics that has not moved much: electricity crosses roughly 30 centimeters in a nanosecond in free space, and rather less through silicon and package traces. Distance therefore sets a floor on latency, and that constraint is the main reason memory is organized as a hierarchy.
The layers differ in what they are physically made of, and the differences are worth holding clearly. SRAM, used for on-chip caches, stores a bit in a latch of roughly six transistors. It is fast and it holds its value as long as power is applied, but six transistors per bit is expensive in area, so caches stay small. DRAM stores a bit as charge on a tiny capacitor guarded by one transistor. One transistor per bit is dense and cheap, but the charge leaks, so every cell must be read and rewritten thousands of times a second. That is refresh, and it is the price of the density. NAND flash traps charge on a floating gate or in a charge-trap layer, where it stays without power. Flash is non-volatile and very dense, and the same trapping mechanism wears out the insulator that makes it work.
None of these technologies is strictly better than the others: SRAM is fast but small, DRAM is dense but must be refreshed, and flash keeps data without power but wears out. A memory system combines them to balance those trade-offs, and the rest of this track covers how.
One number frames why this matters more now than it did a decade ago. Analysis of leading AI accelerators has put memory at roughly 60 to 65 percent of component cost, with HBM and advanced packaging together approaching two-thirds of the unit cost on parts like NVIDIA’s B200.
Eight rungs, seven orders of magnitude
Click a rung. The last row rescales its latency as if one L1 cache hit took a single second, which puts the gaps on a human time scale.
- Bandwidth
- Typical size
- Cost per GB
- If L1 = 1 s
How many bits fit in one well of charge
A flash cell stores a voltage level, and the controller reads bits by deciding which bucket that voltage falls into. Add a bit per cell and you double the buckets inside the same window.
Bits per cell
TLC · 8 statesDistributions are illustrative. Endurance figures are typical published ranges and vary widely with process generation, screening, temperature, and controller behavior. Industrial and enterprise parts are commonly rated well above consumer parts of the same cell type.
How long does the drive actually live
Flash wears out by erasing, not by writing. Since the drive erases in large blocks but the host writes in small pages, the drive writes more than you asked it to. That multiplier is write amplification, and it is usually what kills the part.
SSD endurance
Model: physical NAND equals usable capacity times one plus over-provisioning; the NAND budget is that figure times the cell type’s P/E cycles; host writes supported is the budget divided by write amplification. Real drives differ. Controllers compress, deduplicate, and cache, and vendor TBW ratings assume a specific workload. Raising over-provisioning generally lowers write amplification on random workloads, an interaction this simplified model does not attempt to capture.
Stacking DRAM into the package
HBM shortens the distance by moving DRAM into the package and widening the bus to 1024 or 2048 bits per stack. Build a package and watch the last statistic, which has worsened across generations.
HBM package builder
Bandwidth per stack equals bus width times pin rate divided by eight. HBM2E through HBM3E use a 1024-bit interface; HBM4 doubles it to 2048 bits. JEDEC published HBM4 as JESD270-4, specifying up to 8 Gb/s per pin for roughly 2 TB/s per stack and supporting 4, 8, 12, and 16-high stacks at 24 Gb or 32 Gb per die, for up to 64 GB in one cube. Defaults are set near what shipping parts actually clock rather than at each standard’s ceiling, so the out-of-box HBM3E package lands close to a current 192 GB accelerator. Slide the pin rate up to reach the specified peak.
The words on the datasheet
Common misconceptions
HBM latency is roughly comparable to good DDR5, and sometimes slightly worse. What HBM buys is bandwidth, through a bus thousands of bits wide sitting millimeters from the logic die. It moves far more data per second without delivering any single byte sooner.
They wear out from erasing. NAND is written in pages and erased in much larger blocks, so changing a small amount of data can force the drive to rewrite and erase a great deal more. Write amplification, not the host write count, is usually what sets the life of the part.
It is a trade, and often the right one. QLC gives up endurance and write speed to deliver capacity at a price that makes some workloads possible at all. A read-heavy archive on QLC can be the correct engineering answer and a write-heavy database on the same part can be a mistake.
Capacity and bandwidth scale separately. Build a larger package in the tool above and watch the sweep time rise: bigger memory that is not proportionally faster takes longer to traverse, and that growing gap is what the term memory wall describes.
Four questions before you go
Where to go next
Sources & methodology
Figures are teaching values, hedged where practice varies. Reference points: JEDEC JESD270-4 (HBM4: 2048-bit interface, up to 8 Gb/s per pin, 4/8/12/16-high stacks, 24 Gb and 32 Gb dies, up to 64 GB per stack) and the earlier HBM2E, HBM3, and HBM3E standards for the 1024-bit generations; vendor announcements for shipping HBM4 parts, which run above the specified minimum; published NAND endurance ranges from Kingston, Flexxon, Lexar Enterprise, and reporting on recent QLC endurance claims; Epoch AI component-cost analysis for the memory share of AI accelerator cost; JEDEC DDR5 rates, with channel bandwidth computed as transfers per second times eight bytes; and widely reported latency ranges for cache, DRAM, NVMe, and rotating media. The latency ladder uses representative mid-range values rather than any single part, and the voltage distributions in the flash tool are illustrative rather than measured.