Accelerated Compute
The performance-optimized, purpose-built hardware behind efficient AI has three core components: networking, storage, and compute. Google packages them as the AI Hypercomputer - a stack that pairs that hardware with open software and flexible ways to pay for it.
The rest of this page breaks down the compute layer: the three accelerator types, how each one works, and why architecture - not just clock speed - decides which is fastest for a given workload.
CPU: central processing
Central processing units (CPUs) are not specialized for the matrix multiplications at the heart of deep learning the way GPUs and TPUs are, but they still earn a place in AI workloads. As general-purpose processors they handle data preprocessing, control flow in ML programs, and less compute-intensive inference - especially low-cost inference scenarios.
How a CPU works. At its core a CPU is a general-purpose processor built on the von Neumann architecture: it works with software and memory sequentially - load values from memory, perform a calculation, store the result back to memory.
- Strength - flexibility. A CPU can execute any kind of software for a huge range of applications, from word processing to controlling rocket engines, processing bank transactions, or classifying images with a neural network.
- Limitation - the von Neumann bottleneck. Sequential memory access is slow relative to the calculation itself, which caps overall throughput.
A CPU spends much of its time moving data between memory and the processor, and that memory access is far slower than the arithmetic. This sequential memory-access bottleneck - not raw ALU speed - is what limits a CPU on the large, repetitive matrix math of deep learning.
GPU: parallel processing
Graphics processing units (GPUs) were built to render game graphics, but their real superpower is doing a massive number of calculations at once. That makes them ideal for the heavy lifting in deep learning, especially the complex matrix multiplications.
How a GPU works. A GPU reaches high throughput by packing thousands of arithmetic logic units (ALUs) into one processor - modern GPUs commonly have tens of thousands - so it can run thousands of multiplications and additions simultaneously. Compare the chip real estate: a CPU spends most of its area on control logic and cache with a few large ALUs, while a GPU is almost all ALUs.
- Strength - massive parallelism. For workloads with huge parallelism, like the matrix operations in neural networks, GPUs deliver an order of magnitude higher throughput than CPUs on deep-learning training.
- Limitation - still general-purpose. Because a GPU must support many kinds of instructions and applications, each of its thousands of ALUs still has to frequently read/write registers or shared memory to fetch data and store intermediate results. Newer architectures optimize this, but the general-purpose overhead remains.
TPU: tensor processing
Tensor Processing Units (TPUs) are application-specific integrated circuits (ASICs) designed by Google to accelerate machine learning. Their speed comes from the architecture - specifically the systolic array.
How a TPU works. The systolic array wires thousands of multiply-accumulators directly together into one large physical matrix, purpose-built for the multiply-accumulate operations at the center of matrix math. A single Trillium TPU has two matrix multiplication units (MXUs) plus a scalar unit.
- Data streams from the host into an infeed queue, then into the TPU's High Bandwidth Memory (HBM).
- To multiply, the TPU loads parameters and data from HBM into the MXU.
- As each multiplication happens, the result is passed directly to the next multiply-accumulator - no repeated memory access during the core matrix multiply.
- Final results move to an outfeed queue for the host to retrieve.
- Eliminating memory access mid-multiply is what gives TPUs exceptionally high throughput on neural-network math.
Recap
For the exam-facing angle - GPU vs. TPU selection, attaching GPUs to VMs, quota and zonal availability caveats - see GPUs & TPUs.