Skip to main content

TPU system architecture

Exam guide§2.1

Every Cloud TPU system is built up from one component: the TPU chip, a specialized processor designed from the ground up for the matrix math of machine learning. This page works from the inside out - what sits inside a single chip, then how chips scale into cubes, pods, and slices, how Multislice stretches a job across the data-center network, and how ICI resiliency keeps the interconnect reliable at scale.

What's inside a TPU chip?​

Each TPU chip is engineered from a small set of specialized components that together accelerate ML workloads.

TPU chip internals - two TensorCores each with scalar unit, VPU/VMEM, and MXUs, sharing HBM and SparseCores, connected to the host over PCIe and to other chips over ICI
Inside a TPU chip - one or more TensorCores (each with a scalar unit, vector unit, and MXUs) share HBM and SparseCores, talk to the host over PCIe, and to neighboring chips over ICI.
What it doesTensorCoreThe primary processing unit that does the bulk of the compute. A chip has one ormore TensorCores; the exact number varies by chip version (v3, v4, v5e, v5p).Scalar unitHandles control flow, calculates memory addresses, and manages other essentialhousekeeping operations.Vector unitThe Vector Processing Unit (VPU) plus Vector Memory (VMEM) - handles generalcomputations that aren't matrix multiplies, such as activation functions andsoftmax.Matrix-multiply units (MXUs)The workhorses providing the bulk of the compute power. Each is a systolic array:thousands of multiply-accumulators wired directly into one physical matrix,optimized for multiply-accumulate ops.High Bandwidth Memory (HBM)The chip's fast-access memory interface, feeding data to the TensorCores at highbandwidth.SparseCoreA dataflow processor specialized for sparse computations, acceleratingembedding-heavy models like recommendation systems. TPU v5p has four per chip;Trillium has two.Inter-chip interconnect(ICI)A proprietary Google technology that enables seamless, high-speed communicationdirectly between TPU chips.
The specialized components inside a single TPU chip.
NumbersMXU facts
  • Array size - TPUs prior to Trillium used 128x128 systolic arrays; Trillium uses larger 256x256 arrays.
  • Throughput - each MXU performs up to 16,000 multiply-accumulate operations per cycle.
  • Precision - MXUs accept bfloat16 inputs for the multiplies but perform all accumulations in higher-precision FP32, balancing speed and accuracy.

Scaling up: from chips to pods and slices​

A single chip is powerful, but real-world models often need far more compute. That is where cubes, pods, and slices come in - progressively larger groupings of interconnected chips.

TPU cube​

A cube is a physical unit of 64 chips arranged in a 4x4x4 topology of interconnected TPU chips. Cubes apply only to 3D topologies (starting with TPU v4) and are also called a rack.

A TPU cube - a 4x4x4 building block of 64 TPU chips wired in a 3D torus with optical interconnect and liquid cooling, and a superpod of many cubes
One cube is a 4x4x4 building block of 64 TPU chips (a 3D torus with optical interconnect and liquid cooling); a superpod is many cubes wired together.

TPU pod​

A TPU pod is a collection of TPUs physically grouped together and connected by a specialized high-speed network. The total number of chips in a pod varies by TPU version.

Slice​

A slice is a subset of chips within a single pod, all connected by fast Inter-Chip Interconnects (ICI) so they communicate at very low latency. A slice can range from 4 chips up to the size of a superpod.

GotchaA slice is abstract, a pod and cube are physical

The key distinction: a slice is a flexible, abstract unit - any group of chips from 4 up to a superpod. A rack (cube) and a pod are fixed physical units of measurement. Slices still have physical properties, but the term names a conceptual grouping, not a piece of hardware.

Spatial relationship between pod, slice, and cube - a pod is a 64-rack superpod, a slice is a contiguous group of racks, and a cube is a single rack's 4x4x4 chips
The spatial relationship: a pod is a superpod of 64 racks, a slice is a group of racks within it, and a cube is a single rack (a 4x4x4 block of chips).
FactsHow slices are described
  • Slices are measured in either number of chips or number of TensorCores, depending on the TPU version.
  • Chip shape and chip topology describe how the chips within a slice are arranged and interconnected.

Beyond a single slice: Multislice​

Sometimes a single slice isn't enough for the largest models. Multislice extends TPU connectivity beyond the high-speed ICI links inside one slice, leveraging the broader data-center network (DCN) to move data between slices.

Multislice - two TPU v4 4x4x16 slices, each a stack of 4x4 cubes connected over the ICI model-parallel dimension
Multislice spans multiple slices - here two TPU v4 4x4x16 slices, each built from 4x4 cubes stacked along the ICI dimension.

How it works:

  • Within each slice, data travels over the super-fast ICI.
  • Between slices, data is transmitted over the DCN.
Two pods, each with a slice of TPU chips wired by ICI; the two slices are connected across pods over the data-center network (DCN); hosts sit above each pod
Inside a pod, chips are wired by ICI; across pods, slices connect over the data-center network (DCN). Hosts attach above each pod.

Why Multislice?​

This hybrid ICI-within / DCN-between connectivity enables parallelism across slices, letting a single job use far more TPU cores than one slice can provide - essential for training truly massive models. You choose whether to run a job on a single slice or across multiple slices, depending on scale and requirements.

What it meansMassive scalabilityScale far beyond a single slice. Google used Multislice for what is believedto be the largest publicly disclosed LLM training job - over 50,000 Cloud TPUv5e chips, across multiple data centers.Near-linear performance scalingAdding more chips yields a near-proportionate increase in training speed, soextremely large models finish in a reasonable time with efficient hardwareuse.Enhanced developer productivityScaling a distributed job often takes just an extra sharding dimension.Compiler optimizations and a streamlined stack let developers focus on themodel.Cost efficiencyEfficient scaling maximizes cost-effectiveness - TPU v5e with Multisliceoffers up to 2x higher performance-per-dollar for training LLMs than itspredecessor.Robustness and reliabilityAuto-repair of individual slices and fast recovery let long-running jobsresume quickly after interruptions or failures.Overcoming bottlenecksStrategies like distributed data loading and broadcasting checkpoints from asingle pod cut overhead in large-scale training.
The six benefits of Multislice training.

Multislice is a full-stack solution - optimized network, runtime, scheduler, frameworks, and orchestration - so cutting-edge LLMs too large for a single device train efficiently, cost-effectively, and with a good developer experience.

Multislice full stack - JAX and PyTorch frameworks over multi-controller SPMD, over the XLA compiler + scheduler + runtime, over TPU slices connected by DCN, with GKE and Compute Engine Queued Resources for orchestration
The Multislice stack - JAX/PyTorch frameworks over multi-controller SPMD, over XLA compiler + scheduler + runtime, over DCN-connected slices, orchestrated by GKE and Compute Engine Queued Resources (gang-scheduling, auto-repair).

Ensuring reliability: ICI resiliency​

The ICIs are vital for TPU performance, so Cloud TPU offers ICI resiliency to improve their fault tolerance. It improves the reliability of the optical links and optical circuit switches (OCS) that connect TPUs between different cubes - ICI links within a cube typically use more robust copper. When an OCS or optical ICI fault occurs, ICI resiliency lets connections be rerouted around it.

GotchaThe reroute trade-off

ICI resiliency improves scheduling availability - you're more likely to get the TPU resources you need even during minor network issues. The trade-off: when a reroute happens there may be a temporary, minor degradation in ICI performance.

NumbersICI resiliency availability
  • Enabled by default for Cloud TPU v4 and v5p.
  • Applies to slices that are one cube or larger (for example v5p-128 or a 4x4x4 configuration).

Recap​

Cloud TPUs are Google's custom-built ASICs, designed with specialized TensorCores to accelerate demanding ML computations. Systems build up from individual chips - each with MXUs, HBM, and SparseCores - into cubes (64 chips, 4x4x4), pods (collections of chips), and slices (flexible subsets within a pod), then across data centers with Multislice over the DCN. High-speed ICI ties chips together at every level, and ICI resiliency keeps that interconnect reliable at scale.

0%0 of 147 pages studied