Long-context training β including pretraining and post-training, has become a foundation of frontier AI models. Longer native sequence lengths give large language models (LLMs) the working memory to read a whole code repository or ingest hours of video, and reason through multi-step agentic tasks. In three years, context windows have grown from 128Kβ200K tokens to 1Mβ2M, and models trained toward 10M tokens are arriving. Training at those lengths, not just serving them, is where the system limits show up.
In particular, efficient training at multi-million-token sequence lengths runs into the following bottlenecks:.
Today, we are excited to announce long-context training support in MaxText, Google Cloud's open-source JAX training framework. MaxText enables this by supporting four context-parallel strategies on Cloud TPU: All-gather, A2A (DeepSpeed Ulysses), P2P (Ring Attention), and the A2A + P2P hybrid (Unified Sequence Parallelism, or USP), combined with load balancing and sequence packing. All four use the Tokamax Splash Attention kernel. Combined with the 3D inter-chip interconnect (ICI) ( supported on TPUs such as TPU 5p or TPU 7 (Ironwood) ), this lets one sequence span an entire slice while communication overlaps compute and memory per chip stays flat. In our benchmarks, Llama 3.1 8B (25 head dim, 16 query heads) demonstrates exceptional long-context efficiency, achieving 70% Model FLOPs Utilization (MFU) on TPU 5p for sequence lengths up to 4 million tokens. Across both TPU 5p and TPU 7, context can be seamlessly extended to 14 million tokensβwith TPU 7 consistently maintaining 50% MFU across all sequence scales below 6 million tokens.
This functionality is available today in MaxText v0.2.4(doc).
It has been revealed that context parallelism splits a long sequence across chips, so no chip holds the activations of the whole sequence. Context parallelism can be combined with data and tensor parallelism, and the smallest context-parallel size that fits your sequence in HBM is usually the right one. This also holds for sparser attention variants which cut attention compute but not the activations of every other layer.
Attention is the one global operation where every token needs the keys and values of every preceding token, so each context parallel strategy has to bring those keys and values back together. Each makes a different tradeoff between memory (how much KV a chip holds), communication (how much moves per layer, and whether it overlaps compute), and model shape (how many attention heads to split).
Industry observers note that all-gather: Each chip keeps its chunk of the sequence and, at every layer, gathers the keys and values from every other chip. It is the simplest option. However, because causal masking causes later tokens to attend to larger prefixes, All-gather requires balancing compute load across devices.
