Skip to main content

GPU Options

Exam guide§2.1

GPUs are the workhorses behind most modern AI, from image recognition to natural language processing. Their ability to run a massive number of calculations at once suits them to the compute-heavy math at the heart of machine learning. This page covers the GPU machine families on Google Cloud, how they are wired together into clusters, and which family fits which workload.

How GPUs run AI workloads​

GPUs accelerate the two phases of an AI model's life:

Trainingthe learning phaseInferencethe serving phaseWhat happensFeed the model vast data and iterativelyadjust its internal parameters to minimizethe gap between predictions and the groundtruth.The trained model applies its learnedknowledge to make predictions on new,unseen data - often in real time.Why the GPU mattersRuns the many complex calculationsconcurrently, cutting the time to buildand optimize the model.Executes calculations fast enough forinstant, responsive answers (a chatbotreply, a car's split-second decision).
The two phases where GPUs shine - training the model, then serving predictions from it.

Compute and network architecture​

Before the machine types, here is the high-level compute and network architecture of the top-end A4 Ultra and A3 Ultra machines. The design is hierarchical and hyper-connected - it prioritizes high-speed communication at every level so data flows efficiently across the whole system.

GPU cluster network architecture - Cluster containing Blocks, each with Subblocks, Nodes, and GPUs
A cluster is built from blocks, each block from subblocks, each subblock from nodes, and each node from tightly coupled GPUs - all interconnected by high-bandwidth networks.
NumbersCluster hierarchy, top down
  • Cluster - composed of multiple blocks, all interconnected by high-bandwidth networks.
  • Block - subdivides into several subblocks.
  • Subblock - contains multiple nodes.
  • Node - the most granular level; typically houses multiple GPUs (the figure shows 8 GPUs per node).
  • GPU communication - GPUs within a node are tightly coupled for extremely fast internal communication.

This layered, low-latency, high-bandwidth networking - from GPUs within a node up through subblocks, blocks, and the whole cluster - is what makes it possible to train the largest models, which need massive distributed compute and seamless data synchronization.

GPU machine families​

Google Cloud provides GPU support across accelerator-optimized machine types (A4, A3, A2 for high performance; G2 for inference), plus the general-purpose N1 series with attachable GPUs. Each family targets a different point on the performance/cost curve.

A4A3A2G2N1 + GPUNVIDIA GPUB200H100 familyA100L4T4, V100, P100, P4Built forNewest, most powerful- training andserving foundationmodels.Peak performance andscalability forlarge-scale training,including LLMs.Versatile - trainingand high-throughputinference.Cost-optimizedinference; also somegraphics workloads.General-purposecompute withattachableolder-generationGPUs.NotesHigh GPU memory(HBM3e) and networkbandwidth. Provisionwith Cluster Directoror Spot VMs.High GPU count per VMand ultra-fastGPU-to-GPUnetworking. Hassub-families (below).Strong compute/memorybalance. Offers acompact placementpolicy. Ultra (A10080GB), High (A10040GB).Serve predictions atscale without thepremium oftraining-focusedGPUs.Flexible CPU/GPUmixing; useful forolder GPUs not in theA/G families.
The GPU machine families - each family (column) described by its NVIDIA GPU, target workload, and key notes.

The A3 family splits into several sub-families:

A3 UltraA3 MegaA3 HighA3 EdgeNVIDIA GPUH200H100 80GBH100 80GBH100Best forFoundation model trainingand serving.Large model training andmulti-host inference;often GKE or Slurmclusters.Large model inference andfine-tuning.Serving workloads;available in limitedregions.
The A3 sub-families - each (column) with its NVIDIA GPU and best-fit workload.
GotchaA4 needs dense provisioning

To use A4 machine types effectively, provision with Cluster Director or Spot VMs. Cluster Director reserves densely allocated machines with topology-aware scheduling: host machines are physically close, provisioned in resource blocks, and connected by a dynamic ML network fabric that minimizes network hops for the lowest latency.

GotchaOut of scope for this course

Google Distributed Cloud (GDC) and specific NVIDIA software offerings (for example the NVIDIA NeMo framework) are out of scope here. Worth exploring separately for a fuller picture, but not part of this material.

Using GPU options​

The predominant approach is to deploy an accelerator-optimized VM with attached GPUs (A4, A3, A2, or G2). However, some older GPUs can still be used by attaching them to an N1 general-purpose VM - the flexible path when you want a specific CPU/GPU balance or an older GPU model not offered in the A and G families.

Recap​

Choosing a family comes down to the workload:

DECISIONWhich GPU machine type?
Pre-training models→A4, A3 Ultra, A3 Mega, A3 High, A2
Fine-tuning models→A4, A3 Ultra, A3 Mega, A3 High, A2
Serving inference→A4, A3 Ultra, A3 Mega, A3 High, A2
Graphics-intensive workloads→G2, N1 + T4
High performance computing→Any accelerator-optimized series - best fit depends on how much computation is offloaded to the GPU
Pick this when: A-series for training/inference of foundation & large models; G2 or N1+T4 for graphics; any accelerator-optimized series for HPC

GPUs accelerate both the intensive training phase and the responsive inference phase through parallel processing. Google Cloud's GPU-accelerated families - A4, A3, A2, G2, and the flexible N1 option - are each tuned for a specific task, from foundation-model training to large-scale distributed training to cost-effective inference.

For the exam-facing angle - attaching GPUs to VMs, quota, and zonal availability - see GPUs & TPUs.

0%0 of 147 pages studied