TPU7x (Ironwood) memory hierarchy¶
How high-bandwidth memory (HBM) and host memory are organized on a TPU7x (Ironwood) chip.
High-bandwidth memory¶
Each chip has 192 GiB of HBM, with bandwidth of approximately 7.37 TB/s. The large capacity allows large batch sizes, which can improve throughput. HBM can still be a bottleneck for memory-bound vector operations or inefficient data access patterns.
Dual-chiplet design¶
Each chip has two chiplets. Each chiplet is a self-contained unit with one TensorCore, two SparseCores and 96 GiB of HBM.
Host memory¶
The host's main memory connects to the chip with much lower bandwidth than HBM. You can still use it to offload activations or optimizer states and free HBM, which helps with memory pressure in large models.
Parent topic:TPU7x (Ironwood)