GPUs and TPUs on Google Cloud

How GPUs and TPUs multiply matrices

Matrix multiplication dominates the cost of many transformer layers. GPUs and TPUs both accelerate it, but they organize the hardware differently and run different software stacks.

A GPU is divided into streaming multiprocessors (SMs); an H100 SXM has 132 of them. Each SM contains arithmetic units, registers, shared memory, and Tensor Cores, which are small fixed-function matrix units. The SM runs warps, groups of 32 threads that issue instructions together; divergent control flow can leave some threads inactive. A CUDA kernel launches many blocks of threads. When one warp stalls waiting for data from memory, the SM's scheduler issues instructions from another warp that is ready, which keeps the arithmetic units working while memory requests are outstanding.

A TPU chip has one or two TensorCores. Each TensorCore contains one or more matrix-multiply units (MXUs), a vector unit for operations such as activations and softmax, and a scalar unit for control flow and memory addresses.Google, TPU system architecture. v5p and TPU7x also have four SparseCores per chip for embedding lookups and other gather-scatter work; v6e has two. XLA, the compiler behind JAX and PyTorch/XLA, compiles and partitions the program for the TPU before it runs. Performance depends on tensor shapes, sharding, memory movement, and compiler support for the operations used.

An MXU is a systolic array: a square grid of multiply-accumulate cells, 128 × 128 before v6e and 256 × 256 in v6e and TPU7x. Operands enter at the edges in a regular pattern. On every clock cycle each cell multiplies the values it receives, adds the product to a running sum, and passes values on to its neighbors, so partial sums accumulate as data moves through the array without returning to memory in between.

The array works at full rate only once operands reach every cell. Each row starts one cycle after the row above it, so the array fills along a diagonal. When it is full, every cell completes one multiply-accumulate per cycle: 16,384 per cycle for a 128 × 128 MXU.Google: “Each MXU is capable of performing 16K multiply-accumulate operations per cycle.” Multiplies take bfloat16 inputs; accumulation is in FP32.

A matrix too small to cover the array leaves cells idle. XLA pads it with zeros to fit, which wastes both arithmetic and on-chip memory.Google, Introduction to Cloud TPU: “When a matrix computation cannot occupy an entire MXU, the compiler pads tensors with zeros.” The same guide recommends batch and feature dimensions that are multiples of 8, with one of them a multiple of 128, and warns that a program dominated by non-matrix operations such as reshapes and concatenations will likely not keep the MXUs busy.

The accelerator lineup in September 2026

Google Cloud sells GPUs as part of machine series, such as A4 for the B200 or G2 for the L4, and sells TPUs by generation. Figure 3 lists every model offered on 11 September 2026, grouped by architecture.

Each column answers a different question. Memory per chip decides whether the weights, activations and KV cache fit. Datacenter parts use HBM, stacked DRAM placed beside the die; the L4, RTX PRO 6000, T4 and P4 use GDDR. Bandwidth is how fast the chip can read that memory, and bounds any step that moves more bytes than it computes on. Dense peak is the vendor's maximum matrix throughput, which bounds compute-heavy kernels.“Dense” means without 2:1 structured sparsity. NVIDIA quotes sparse peaks that are twice the dense figure; this page divides them by two. All throughput and bandwidth figures are vendor peaks and approximate; real kernels reach a fraction of them. The fast domain is the largest group of chips joined by the accelerator's own interconnect, such as NVLink or ICI; traffic beyond it crosses the slower datacenter network. For PCIe-only GPUs, the table instead gives the maximum GPU count in one VM; sharing a VM does not provide NVLink bandwidth.

The peak column uses FP8 where the chip has it, otherwise INT8 (A100, T4, P4, v4, v5e, v6e), FP16 (V100, P100) or BF16 (v2, v3, which have no faster mode). The small label beside each number names its precision.

Two more TPUs are announced but not yet rentable.Google, TPU 8t and TPU 8i technical deep dive, Cloud Next, April 2026. TPU 8t is a training system: 9,600 chips, about 2 PB of aggregate HBM, and 19.2 Tb/s of ICI bandwidth per chip. TPU 8i is an inference system with 384 MB of on-chip SRAM, a Collectives Acceleration Engine, and a Boardfly interconnect. Google says Cloud availability for both is forthcoming.

Two speed limits: compute and memory bandwidth

A chip has two speed limits: how fast it can compute, and how fast it can read bytes from memory. Which one a workload hits depends on its arithmetic intensity, the number of arithmetic operations it performs per byte moved to or from device memory. The roofline model draws both limits on one chart.

Attainable throughput is the smaller of two numbers: peak compute, and bandwidth multiplied by intensity. The two meet at the ridge point, peak compute divided by bandwidth. For an H200, about 1,979 TFLOPS ÷ 4.8 TB/s puts the ridge at about 412 FLOP per byte. Work to the left of the ridge is memory-bound; work to the right is compute-bound.

For a dense linear layer, applying a weight uses one multiply and one add: two operations per weight. This count excludes attention, nonlinearities and other model work. With one-byte FP8 weights, decoding one sequence at a time performs about 2 FLOP per byte read. A batch of 64 sequences reuses each weight 64 times, about 128 FLOP per byte, and a 1,024-token prefill about 2,048.Simplified: these figures count weight traffic only. Reading the KV cache lowers the intensity of long-context decoding, while cache reuse and kernel fusion raise the intensity of real kernels. The presets use these values for one-byte FP8 or INT8 weights. They halve the intensity for a chip whose displayed peak uses two-byte FP16 or BF16 weights. Stored precision and the peak arithmetic mode must agree for this comparison.

Every chip has its own ridge point. Figure 5 plots peak compute against bandwidth for the whole lineup, so the diagonal lines are lines of equal ridge point. A chip above the 1,000 line, such as the TPU v6e, needs more than 1,000 operations per byte of reuse to reach its peak. Chips near the 100 line, such as the V100 and TPU v3, reach theirs with far less reuse, but their peak is lower.

Capacity, bandwidth and compute answer different questions. Capacity limits the model weights, activations and KV cache a chip can hold. Throughput bounds compute-heavy kernels. Bandwidth, and the interconnect between chips, still determine much of real training and inference performance. The usual remedies for a memory-bound workload are bigger batches, quantized weights, or a chip with more bandwidth per operation.

Fast domains: how chips connect

A model spread over many chips exchanges activations, gradients, parameters and routing traffic between them. The fast-interconnect domain constrains which parallel layouts can keep that traffic off the slower datacenter network.

On NVIDIA systems, the A3 and A4 eight-GPU VMs connect their GPUs through NVSwitch, with 900 GB/s of bidirectional NVLink bandwidth per Hopper GPU or 1.8 TB/s per Blackwell GPU. Larger jobs use the VM network: up to 3.6 Tbps of GPUDirect RDMA on A3 Ultra and A4, and up to 1.8 Tbps on A3 Mega. A4X and A4X Max connect 18 four-GPU instances as one 72-GPU NVLink domain.

TPUs link chips directly to their neighbors over ICI, Google's inter-chip interconnect. TPU v2, v3, v5e and v6e use 2D torus topologies; v5p and TPU7x use 3D tori. TPU v4 has a 3D mesh that can form a torus for supported slice shapes. Optical circuit switches configure the links between 4 × 4 × 4 cubes of chips and can route around some ICI faults.Google defines a cube as “a 4x4x4 topology of interconnected TPU chips,” used from TPU v4 on. A slice is the block of a pod rented as one unit; slice shapes have generation-specific constraints. Parallelism performs best when it maps cleanly onto the physical topology.

How many chips a model needs

The first question for any deployment is whether the model fits. Weights need parameters × bytes per parameter: a 70-billion-parameter model in FP8 takes about 70 GB. Serving also needs room for the KV cache and activations, which the calculator adds as a percentage allowance rather than computing it. A mixed-precision full fine-tune with the Adam optimizer often needs about 16 bytes per parameter before activation memory.2 bytes of BF16 weights, 2 of BF16 gradients, and 12 of FP32 optimizer state (a master copy of the weights plus Adam's two moment estimates): the accounting in Rajbhandari et al., “ZeRO” (2019).

The calculator treats 90% of each chip’s memory as usable and assumes the memory load can be divided evenly across chips. Its chip count is a memory lower bound. A deployable configuration must also match VM or TPU slice sizes, supported numerical formats and the model’s sharding constraints. Being below a domain’s chip limit does not establish that a matching allocation exists.

Buying capacity and finding it

Price and capacity depend on the provisioning model, region, duration and machine series. In this snapshot, several newer GPU series have no on-demand offering. Check the current pricing and availability pages before budgeting a workload.

Where each chip is offered

New accelerators are available in fewer locations than mature GPU series.Approximate zone counts from Google's GPU regions and zones page on 11 September 2026. Availability and quota change; verify the machine shape in the target zone before planning a deployment.

Glossary

Arithmetic intensity

Arithmetic operations per byte moved to or from device memory; integer operations and floating-point operations are labeled separately. Often low for small-batch decoding and higher for training, prefill and large batches.

Dense and sparse FLOPS

NVIDIA quotes peaks with 2:1 structured sparsity that are double the dense figure. This page uses dense figures throughout.

HBM

High Bandwidth Memory: stacked DRAM placed beside the accelerator die. Capacity limits what fits; bandwidth sets the speed of memory-bound kernels.

ICI

Inter-Chip Interconnect, Google's TPU-to-TPU links. Topology and optical reconfiguration depend on the TPU generation.

MXU

Matrix-multiply unit: the systolic array at the core of a TPU TensorCore.

Multislice

Running one job across several TPU slices joined over the datacenter network.

NVLink and NVSwitch

NVIDIA's GPU-to-GPU fabric inside a server (8 GPUs) or, with NVL72, a rack (72 GPUs).

RoCE and GPUDirect

RDMA over Ethernet between VMs. On A3 Ultra, A4 and A4X it runs on Google's Titanium offload NICs.

Slice

A rectangular block of a TPU pod rented as one unit, for example 4 × 4 × 4 = 64 TPU7x chips.

SparseCore

A TPU co-processor for embedding lookups and other gather-scatter work: four per chip on v5p and TPU7x, two on v6e.

vWS

The virtual-workstation license variant of a GPU (L4, RTX PRO 6000, T4, P4) for remote graphics.