ENVIZNMemory and storage
On a modern AI accelerator, the memory now costs more than the logic it feeds. This track starts at a single charge trap and ends at the bandwidth wall that decides how much of an expensive GPU actually does work.
Read the memory hierarchy in latency, bandwidth, and dollars · explain why more bits per flash cell buys capacity and spends endurance · size an SSD’s working life · compute HBM bandwidth from bus width and pin rate, and say why capacity is outrunning it.
Memory is an argument about distance
Every storage technology is fast, cheap, or large. Engineering picks two and arranges the rest in layers.
A processor can only work on data that has arrived. Everything else in a computer exists to shorten the wait, and the wait is governed by physics that has not moved much: electricity crosses roughly 30 centimeters in a nanosecond in free space, and rather less through silicon and package traces. Distance is latency. That single fact produces the whole hierarchy.
The layers differ in what they are physically made of, and the differences are worth holding clearly. SRAM, used for on-chip caches, stores a bit in a latch of roughly six transistors. It is fast and it holds its value as long as power is applied, but six transistors per bit is expensive in area, so caches stay small. DRAM stores a bit as charge on a tiny capacitor guarded by one transistor. One transistor per bit is dense and cheap, but the charge leaks, so every cell must be read and rewritten thousands of times a second. That is refresh, and it is the price of the density. NAND flash traps charge on a floating gate or in a charge-trap layer, where it stays without power. Flash is non-volatile and very dense, and the same trapping mechanism wears out the insulator that makes it work.
Notice that nothing here is strictly better. SRAM is fast and small. DRAM is dense and forgetful. Flash is permanent and mortal. A memory system is the arrangement of those compromises, and the arrangement is what the rest of this track is about.
One number frames why this matters more now than it did a decade ago. Analysis of leading AI accelerators has put memory at roughly 60 to 65 percent of component cost, with HBM and advanced packaging together approaching two-thirds of the unit cost on parts like NVIDIA’s B200. The logic die, the thing the chip is named after, is no longer the expensive part.
Eight rungs, seven orders of magnitude
Click a rung. The last row rescales its latency as if one L1 cache hit took a single second, which is the only way most people ever feel the gaps.
- Bandwidth
- Typical size
- Cost per GB
- If L1 = 1 s
How many bits fit in one well of charge
A flash cell does not store a bit. It stores a voltage, and the controller decides which bucket that voltage falls into. Add a bit per cell and you double the buckets inside the same window.
Bits per cell
TLC · 8 statesDistributions are illustrative. Endurance figures are typical published ranges and vary widely with process generation, screening, temperature, and controller behavior. Industrial and enterprise parts are commonly rated well above consumer parts of the same cell type.
How long does the drive actually live
Flash wears out by erasing, not by writing. Since the drive erases in large blocks but the host writes in small pages, the drive writes more than you asked it to. That multiplier is write amplification, and it is usually what kills the part.
SSD endurance
Model: physical NAND equals usable capacity times one plus over-provisioning; the NAND budget is that figure times the cell type’s P/E cycles; host writes supported is the budget divided by write amplification. Real drives differ. Controllers compress, deduplicate, and cache, and vendor TBW ratings assume a specific workload. Raising over-provisioning generally lowers write amplification on random workloads, an interaction this simplified model does not attempt to capture.
Stacking memory on top of the problem
HBM solves distance by moving DRAM into the package and making the bus absurdly wide. Build a package and watch the last statistic, which is the one that has been getting worse.
HBM package builder
Bandwidth per stack equals bus width times pin rate divided by eight. HBM2E through HBM3E use a 1024-bit interface; HBM4 doubles it to 2048 bits. JEDEC published HBM4 as JESD270-4, specifying up to 8 Gb/s per pin for roughly 2 TB/s per stack and supporting 4, 8, 12, and 16-high stacks at 24 Gb or 32 Gb per die, for up to 64 GB in one cube. Defaults are set near what shipping parts actually clock rather than at each standard’s ceiling, so the out-of-box HBM3E package lands close to a current 192 GB accelerator. Slide the pin rate up to reach the specified peak.
The words on the datasheet
Common misconceptions
HBM latency is roughly comparable to good DDR5, and sometimes slightly worse. What HBM buys is bandwidth, through a bus thousands of bits wide sitting millimeters from the logic die. It moves far more data per second, not each byte sooner.
They wear out from erasing. NAND is written in pages and erased in much larger blocks, so changing a small amount of data can force the drive to rewrite and erase a great deal more. Write amplification, not the host write count, is usually what sets the life of the part.
It is a trade, and often the right one. QLC gives up endurance and write speed to deliver capacity at a price that makes some workloads possible at all. A read-heavy archive on QLC can be the correct engineering answer and a write-heavy database on the same part can be a mistake.
Capacity and bandwidth are separate purchases. Build a larger package in the tool above and watch the sweep time rise: bigger memory that is not proportionally faster takes longer to traverse, which is precisely the wall people mean.
Four questions before you go
Where to go next
Sources & methodology
Figures are teaching values, hedged where practice varies. Reference points: JEDEC JESD270-4 (HBM4: 2048-bit interface, up to 8 Gb/s per pin, 4/8/12/16-high stacks, 24 Gb and 32 Gb dies, up to 64 GB per stack) and the earlier HBM2E, HBM3, and HBM3E standards for the 1024-bit generations; vendor announcements for shipping HBM4 parts, which run above the specified minimum; published NAND endurance ranges from Kingston, Flexxon, Lexar Enterprise, and reporting on recent QLC endurance claims; Epoch AI component-cost analysis for the memory share of AI accelerator cost; JEDEC DDR5 rates, with channel bandwidth computed as transfers per second times eight bytes; and widely reported latency ranges for cache, DRAM, NVMe, and rotating media. The latency ladder uses representative mid-range values rather than any single part, and the voltage distributions in the flash tool are illustrative rather than measured.