Packing Sequences.
Fill every slot of every sequence with real text — and decide whether a token may look back into somebody else’s document.
A training batch is a rectangle: B sequences of exactly T tokens. Documents are not: most web pages are short, some books are enormous. The naive layout puts one document per sequence and fills the rest with padding, which costs compute and teaches nothing. The standard layout packs documents end to end, separated by an end-of-text token, and cuts the stream into length-T pieces: almost no padding, and long documents simply continue into the next sequence.
Packing has a side effect. Under the ordinary causal mask, a token near the end of a sequence can attend to the tail of a previous, unrelated document. Most large runs accepted this — the end-of-text token teaches the model that what came before is unrelated — but some, such as Llama 3, add a document mask that blocks attention across boundaries. The workshop computes both costs for documents of realistic, skewed lengths.
Below: left, the share of positions wasted on padding and the share of attention pairs that cross a document boundary, for padding and packing, as the sequence length changes. Right, a 24-token sequence packed with three documents: the causal mask (cross-document pairs shaded) and the document mask.
- Compare padding with packing.
- Measure how much compute padding wastes.
- Count attention pairs that cross document boundaries.
- Describe the document mask.
No empty slots
With documents of median length 120 tokens and 2 048-token sequences, padding wastes 91% of all positions — about nine forward and backward computations in ten spent on nothing. Packing wastes 0.93%, essentially only the final partial sequence. Its cost is in attention: 80% of the causal attention pairs connect a token to a different, unrelated document. With longer documents (median 1 200 tokens) that share falls to 28%, because each sequence holds fewer boundaries.
Whether cross-document attention matters is an empirical question, and the answer has shifted. Early large runs packed without a document mask and relied on the end-of-text token; the Llama 3 report describes masking attention across documents within a sequence, and found it important when training on very long sequences. Modern attention kernels support such masks at little cost. Either way, packing itself is universal: no large run pays for padding.
Terms
- Padding
- Filler positions that make a short document fill a fixed-length sequence.
- Packing
- Concatenating documents, separated by end-of-text tokens, and cutting the stream into fixed-length sequences.
- Document mask
- An attention mask that also blocks attention across document boundaries within a sequence.
- Cross-document attention
- A token attending to a token of a different document packed before it.