Google designed TPUs from the ground up to be exceptionally efficient for machine learning workloads, delivering top-tier performance per total cost of ownership (TCO). Because TPUs trade general-purpose hardware scheduling for wide systolic arrays, dedicated SparseCore engines, and custom interconnect fabrics, adhering to specific architectural best practices is essential to achieve maximum performance. To help developers get the most out of TPUs, we are launching the open-source AI-Hypercomputer/tpu-codesign guide. This post highlights its key principles—spanning shape alignment, architectural choices like LatentMoE, and parallelism strategies—to accelerate LLMs, Mixture-of-Experts (MoE) models, and other hardware-intensive workloads on TPUs.

Performance tuning starts with model definition. Architectural choices made before the first training step set permanent ceilings on MFU (Model FLOP Utilization), cost, and serving latency. This guide gives concrete, hardware-grounded design rules for teams building or adapting models for TPUs, organized into three groups:.

How to use this guide. The shape rules apply to anyone designing a upcoming model. If you are adapting an existing model rather than designing one, the Expressing Parallelism section applies without touching the checkpoint. The general rules hold for any large-matmul architecture (dense LLMs, multimodal, and world models), and the companion AI-Hypercomputer/tpu-codesign repository grows model- and workload-specific sections over time.

Eighth-generation TPUs. These principles carry forward to TPU 8t and TPU 8i and to the newly unveiled TPU 8tx and TPU 8i Max. The reasoning stays the same and the thresholds move with the hardware. TPU 8t and TPU 8i both add native FP4. For training, TPU 8t also brings more balanced Vector Processing Unit (VPU) scaling. For serving, TPU 8i adds the Boardfly topology, a Collectives Acceleration Engine (CAE), and 3x more on-chip SRAM. We note the effects inline in Principles 5–8, and follow-up posts will cover eighth-generation designs in depth.

It has been revealed that measured evidence. Alongside the published sources, notes marked Measured carry results from our own Ironwood profiling program: open 20–30B MoEs (a Qwen3-30B-derived LatentMoE variant; GPT-OSS-20B) plus dense controls, single slice (except for the 512-chip TPU v7x and 1,024-chip TPU v4 ablations in Principle 4), BF16, training shapes up to 32K sequence, all against tuned-systems baselines. Treat the trends and rankings as transferable, the exact percentages as config-specific.

It has been revealed that TPUs are purpose-built custom ML ASICs designed from the ground up to deliver state-of-the-art performance per total cost of ownership (TCO). While their architectural choices—such as wide 256×256 systolic arrays, software-managed VMEM, dedicated SparseCore engines, and direct multi-dimensional ICI fabrics—enable exceptional efficiency, they inherently create specific structural constraints and programming patterns that must be carefully followed to unlock maximal performance. Reaching peak efficiency requires aligning model shapes, routing, and parallelism directly with these underlying hardware characteristics:.

→ Read original article