Attention Is All You Need Attention Is All You Need The paper that changed everything. A landmark contribution from Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser & Polosukhin at Google Brain and Google Research — published in 2017, it introduced the Transformer: a pure attention-based architecture that made recurrence and convolution obsolete for sequence transduction. What followed was the foundation of every large language model, every AI breakthrough, and every intelligence revolution you see unfolding today. Landmark ResearchGoogle Brain · Google Research · University of Toronto The Authors Behind the Revolution Eight researchers — equal contributors, listed in random order — collectively redesigned the architecture of machine intelligence. Their work at Google Brain, Google Research, and the University of Toronto represents one of the most consequential collaborations in the history of artificial intelligence. Ashish Vaswani Google Brain. Designed and implemented the first Transformer models; crucially involved in every aspect of the work. Noam Shazeer Google Brain. Proposed scaled dot-product attention, multi-head attention, and the parameter-free position representation. Niki Parmar Google Research. Designed, implemented, tuned, and evaluated countless model variants in the original codebase and tensor2tensor. Jakob Uszkoreit Google Research. Proposed replacing RNNs with self-attention and initiated the effort to evaluate this transformative idea. Llion Jones Google Research. Responsible for the initial codebase, novel model variants, efficient inference, and visualizations. Aidan N. Gomez University of Toronto. Co-designed various parts of tensor2tensor, massively accelerating research (work performed at Google Brain). Lukasz Kaiser Google Brain. Spent countless days implementing tensor2tensor, replacing the earlier codebase and greatly improving results. Illia Polosukhin Independent. Co-designed the first Transformer models alongside Ashish Vaswani; foundational to the earliest implementations. The Abstract: One Paragraph That Rewrote History "The dominant sequence transduction models are based on complex recurrent or convolutional neural networks... We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely." In a single paragraph, the authors announced that everything the field had built — every LSTM, every GRU, every convolutional encoder-decoder — was about to be superseded. The Transformer is simpler, faster to train, more parallelizable, and achieves state-of-the-art BLEU scores on both WMT 2014 English-German (28.4) and English-French (41.8) translation benchmarks. It trained to world-best quality in 3.5 days on 8 GPUs — a fraction of what prior architectures required. Chapter I: The Problem with Sequential Thinking Background Before the Transformer, the field of NLP was dominated by recurrent neural networks — specifically LSTMs and GRUs. These architectures process sequences one token at a time: the hidden state ht is a function of the previous state h{t-1} and the current input. This sequential dependency is elegant in theory, but catastrophic in practice at scale. The Parallelization Problem Because each step depends on the previous, recurrent models cannot be parallelized within a single training example. At long sequence lengths, memory constraints limit batching across examples. Modern GPU hardware is wasted — thousands of cores sitting idle while computation proceeds one token at a time. The Long-Range Dependency Problem Signals between distant positions must traverse O(n) sequential steps in a recurrent network. The longer the sequence, the harder it becomes to learn that the word "making" at position 12 completes the phrase "making...more difficult" at position 20. Gradient flow degrades. Information vanishes. The Convolutional Alternative's Limits Models like ByteNet and ConvS2S replaced recurrence with convolutions, enabling parallelism but requiring O(log n) or O(n/k) operations to relate distant positions. The fundamental bottleneck of path length remained — just in a different mathematical form. The Insight: Attention Is Sufficient The Transformer's core hypothesis is deceptively simple and radically powerful: you do not need recurrence or convolution to model sequences. Attention mechanisms alone — applied globally and in parallel — can capture every dependency, at every distance, in constant time. This is not an incremental improvement. It is a paradigm shift. Global Dependencies Every position attends to every other position simultaneously. No information bottleneck. No vanishing gradient over distance. O(1) path length between any two tokens. Full Parallelism With no recurrent dependency, the entire sequence is processed in parallel. Training on modern hardware becomes orders of magnitude more efficient. 12 hours to state-of-the-art, not weeks. Elegant Simplicity The architecture eliminates LSTMs, GRUs, dilated convolutions, and positional RNNs. What remains is stacked self-attention and feed-forward layers — clean, interpretable, and composable. Chapter II: The Transformer Architecture Model Architecture The Transformer follows the canonical encoder-decoder structure. The encoder maps an input sequence (x1, \dots, xn) to a sequence of continuous representations \mathbf{z} = (z1, \dots, zn). The decoder then generates an output sequence (y1, \dots, ym) autoregressively — consuming previously generated symbols as input at each step. Both halves are built from identical stacked layers of attention and feed-forward computation. The Encoder Stack N = 6 identical layers. Each layer contains two sub-layers: (1) multi-head self-attention, and (2) a position-wise feed-forward network. Each sub-layer is wrapped with a residual connection and layer normalization: LayerNorm(x + Sublayer(x)). All sub-layers and embedding layers produce outputs of dimension dmodel = 512. The Decoder Stack N = 6 identical layers with three sub-layers: (1) masked multi-head self-attention, (2) multi-head attention over the encoder output, and (3) a position-wise feed-forward network. Masking prevents positions from attending to subsequent positions, preserving the autoregressive property. Predictions for position i can only depend on known outputs at positions less than i. Figure 1 The complete Transformer model architecture. Left: the Encoder stack (N=6 layers). Right: the Decoder stack (N=6 layers) with masked self-attention to preserve autoregressive generation. Both stacks use positional encoding added to input embeddings, enabling the model to utilize sequence order without any recurrent structure. The final decoder output passes through a linear projection and softmax to produce output token probabilities. Chapter III: Scaled Dot-Product Attention Core Mechanism Attention is formally described as a function that maps a query and a set of key-value pairs to an output — all of which are vectors. The output is a weighted sum of the values, where each weight is determined by a compatibility function between the query and the corresponding key. The authors call their specific variant Scaled Dot-Product Attention. Given queries packed into matrix Q, keys into K, and values into V: \text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{dk}}\right)V Why Scale by √dk? For large values of dk, dot products grow large in magnitude, pushing the softmax into saturation regions with extremely small gradients. Dividing by √dk stabilizes training. If q and k are independent random variables with mean 0 and variance 1, their dot product has variance dk — scaling restores unit variance. Why Dot-Product Over Additive Attention? Additive attention uses a single-hidden-layer feed-forward network to compute compatibility. While theoretically similar in complexity, dot-product attention is dramatically faster and more space-efficient in practice — it is implemented via highly optimized matrix multiplication routines (BLAS/cuBLAS), yielding significant wall-clock speedups. Optional Masking In the decoder, future positions are masked by setting their attention logits to −∞ before the softmax. This ensures that during autoregressive generation, position i cannot attend to positions i+1, i+2, … — preserving the causal structure required for valid sequence generation. Figure 2 Left: Scaled Dot-Product Attention — the atomic unit of the Transformer. The query-key dot product is computed, scaled by 1/√dk, optionally masked (in the decoder), passed through softmax, then multiplied by the values. Right: Multi-Head Attention — h parallel instances of scaled dot-product attention, each operating on linearly projected subspaces of Q, K, V. Their outputs are concatenated and projected back to dmodel dimensionality. This allows the model to jointly represent information from multiple distinct representation subspaces simultaneously. Chapter IV: Multi-Head Attention Core Mechanism Rather than applying a single attention function over the full dmodel-dimensional space, the Transformer linearly projects queries, keys, and values h times with different learned projections. Each projection operates in a lower-dimensional subspace, runs attention in parallel, and the results are concatenated and re-projected. This is Multi-Head Attention. \text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}1, \dots, \text{head}h)W^O\text{where } \text{head}i = \text{Attention}(QWi^Q, KWi^K, VWi^V) h = 8 Heads The paper uses 8 parallel attention heads. Each head projects to dk = dv = dmodel/h = 64 dimensions. Total computational cost is similar to single-head attention at full dimensionality — but expressiveness is dramatically higher. Multiple Subspaces A single attention head is forced to average information from all subspaces, inhibiting its ability to specialize. With h heads, each head learns to attend to a different type of relationship — syntactic, semantic, co-referential — simultaneously. Parameter Matrices Projection matrices: Wi^Q ∈ ℝ^(dmodel × dk), Wi^K ∈ ℝ^(dmodel × dk), Wi^V ∈ ℝ^(dmodel × dv), and W^O ∈ ℝ^(hdv × dmodel). These are all learned end-to-end during training. Three Ways Attention Is Used in the Transformer Multi-head attention appears in three distinct roles within the architecture, each serving a structurally different purpose. This tripartite deployment is one of the most elegant aspects of the design — a single mechanism solving three distinct problems. Encoder-Decoder Attention Queries come from the previous decoder layer; keys and values come from the encoder output. Every decoder position can attend to every encoder position. This is the classic cross-attention that allows the model to "read" the full source sequence at every generation step — analogous to the attention mechanism in sequence-to-sequence models. Encoder Self-Attention All queries, keys, and values come from the same place: the output of the previous encoder layer. Each position in the encoder can attend to every other position in the previous layer. This allows the encoder to build rich, globally-contextualized representations of the input sequence. Decoder Masked Self-Attention Each decoder position attends to all decoder positions up to and including itself. Future positions are masked (set to −∞) before softmax, preserving the autoregressive property. This allows the decoder to model output context while remaining causally consistent during training and inference. Position-wise Feed-Forward Networks In addition to attention sub-layers, each encoder and decoder layer contains a fully connected feed-forward network applied identically and independently to each position. This is not a global operation — it is position-local, operating on each token's representation separately. \text{FFN}(x) = \max(0, xW1 + b1)W2 + b2 Architecture Two linear transformations with a ReLU activation in between. Input and output dimensionality: dmodel = 512. Inner-layer dimensionality: dff = 2048 — a 4× expansion that provides significant representational capacity. This can equivalently be viewed as two 1×1 convolutions. Role in the Architecture While attention layers enable communication between positions — aggregating context globally — the feed-forward network processes each position's representation independently. It can be understood as the "thinking" step: after gathering context via attention, each position processes what it has gathered through a dedicated nonlinear transformation. Parameters differ layer-to-layer but are shared across positions within a layer. Embeddings, Softmax & Weight Tying The Transformer uses learned embeddings to convert input and output tokens to vectors of dimension dmodel = 512 — a standard practice in sequence transduction. What is less standard is the weight-tying strategy employed by the authors, borrowed from Press & Wolf (2016). Shared Weight Matrix The same weight matrix is used across three places: the input embedding layer, the output embedding layer, and the pre-softmax linear transformation in the decoder. This parameter sharing reduces the total parameter count and has been shown empirically to improve model quality — three roles, one matrix. √dmodel Scaling In the embedding layers, weights are multiplied by √dmodel. This prevents the embedding vectors from being overwhelmed by the positional encodings that are subsequently added to them — balancing the two sources of input representation at the correct relative magnitude. Softmax Output The decoder output is converted to next-token probability distributions via a learned linear transformation (the shared weight matrix transposed) followed by a softmax. During inference with beam search (beam size 4, length penalty α = 0.6), this distribution is used to select output tokens autoregressively. Positional Encoding: Injecting Order Without Recurrence Because the Transformer contains no recurrence and no convolution, it has no built-in notion of token order. Without positional information, "the cat sat on the mat" and "the mat sat on the cat" would produce identical representations. The solution: add a positional encoding directly to the input embeddings. PE{(pos, 2i)} = \sin(pos / 10000^{2i/d{\text{model}}})PE{(pos, 2i+1)} = \cos(pos / 10000^{2i/d{\text{model}}}) Sinusoidal Design Each dimension of the positional encoding corresponds to a sinusoid at a different frequency. Wavelengths form a geometric progression from 2π to 10000·2π. This continuous, deterministic representation encodes both absolute position and relative offsets — for any fixed offset k, PE{pos+k} can be expressed as a linear function of PE{pos}. Extrapolation Advantage The sinusoidal encoding was chosen over learned positional embeddings specifically because it may allow the model to generalize to sequence lengths longer than those seen during training. The authors validated that learned embeddings (Table 3, row E) yield nearly identical results on the training distribution — making the sinusoidal version strictly preferable for robustness. Chapter V: Why Self-Attention? Theoretical Motivation The authors rigorously compare self-attention to recurrent and convolutional layers along three critical dimensions: per-layer computational complexity, parallelizability (minimum sequential operations), and maximum path length between any two sequence positions. The results make the case for self-attention unambiguous. The Complexity Comparison (Table 1) This table is one of the most important in the paper. It is the mathematical proof that self-attention is not just architecturally elegant — it is computationally superior for the regime in which NLP models operate. | Layer Type | Complexity / Layer | Sequential Ops | Max Path Length | | --- | --- | --- | --- | | Self-Attention | O(n² · d) | O(1) | O(1) | | Recurrent | O(n · d²) | O(n) | O(n) | | Convolutional | O(k · n · d²) | O(1) | O(logk(n)) | | Self-Attn (restricted) | O(r · n · d) | O(1) | O(n/r) | When sequence length n is smaller than representation dimensionality d — the typical case in NLP with word-piece or byte-pair representations — self-attention layers are computationally faster than recurrent layers. Crucially, self-attention achieves O(1) path length between any two positions, making it categorically superior for learning long-range dependencies. No other architecture achieves this simultaneously with full parallelism. The Parallelism Advantage: O(1) Sequential Operations The minimum number of sequential operations is the formal measure of parallelizability. Self-attention — both standard and restricted — requires O(1) sequential operations, meaning the entire computation can be performed in a constant number of parallel steps regardless of sequence length. Recurrent networks require O(n) sequential operations — they must process tokens one at a time, serializing computation in a way that no amount of hardware can overcome. Convolutional layers match self-attention's O(1) sequential operations but cannot connect all pairs of positions without stacking O(n/k) layers, reintroducing depth and path-length penalties. The Transformer eliminates all of these constraints simultaneously. Interpretability as a Side Benefit Self-attention is not merely computationally efficient — it is interpretable by design. Because attention weights are explicit, normalized distributions over input positions, researchers can directly inspect what information each head is aggregating and from where. The paper demonstrates that individual attention heads learn to specialize in specific linguistic phenomena. Syntactic Structure Multiple attention heads in encoder self-attention at layer 5 exhibit behavior directly related to syntactic structure — attending along grammatical dependencies, phrase boundaries, and constituent relationships without any explicit syntactic supervision. Long-Distance Dependencies Many heads learn to track long-range dependencies. In the visualization of Figure 3, heads attending to the verb "making" correctly identify "more difficult" as its distant complement — across 8+ intervening tokens — demonstrating global context capture. Anaphora Resolution As shown in Figure 4, certain heads specialize in coreference — the word "its" sharply attends to "Law" many positions earlier. These heads appear to solve anaphora resolution as an emergent behavior, with no explicit training signal for this task. Chapter VI: Training Regime Training Details The training configuration of the Transformer is as important as its architecture. The authors made careful, principled choices at every level — from data preprocessing to optimizer scheduling — that collectively enable the model to train faster and generalize better than any prior approach. Training Data EN-DE: WMT 2014 English-German — ~4.5M sentence pairs. Encoded with byte-pair encoding; shared source-target vocabulary of ~37,000 tokens. EN-FR: WMT 2014 English-French — 36M sentences. Tokenized into a 32,000 word-piece vocabulary. Sentence pairs batched by approximate sequence length with ~25,000 source and target tokens per batch. Hardware One machine with 8 NVIDIA P100 GPUs. Base model: ~0.4 seconds per training step, trained for 100,000 steps (12 hours total). Big model: ~1.0 second per step, trained for 300,000 steps (3.5 days total). This is an extraordinary demonstration of training efficiency relative to prior state-of-the-art. Optimizer Adam optimizer with β₁ = 0.9, β₂ = 0.98, ε = 10⁻⁹. Learning rate schedule: linear warmup for warmupsteps = 4,000 steps, then decay proportional to inverse square root of step number. This schedule is critical — too high an initial learning rate destabilizes training; the warmup prevents early divergence. The Learning Rate Schedule The adaptive learning rate formula used in this paper has become a template for training large neural networks. It encodes a sophisticated training dynamic: build up slowly, then anneal smoothly. lr = d{\text{model}}^{-0.5} \cdot \min\!\left(\text{step}^{-0.5},\ \text{step} \cdot \text{warmup\steps}^{-1.5}\right) Phase 1: Linear Warmup For the first 4,000 steps, learning rate increases linearly. This prevents the optimizer from making large, destabilizing updates before the model has built a stable gradient estimate. Critical for training stability with Adam and large batch sizes. Phase 2: Inverse Square Root Decay After 4,000 steps, the learning rate decreases proportionally to 1/√step. This smoothly anneals the learning rate as training progresses, allowing the model to refine its weights without overshooting optima. The dmodel^(-0.5) prefactor scales the magnitude appropriately for models of different sizes. Regularization Strategy The Transformer employs two complementary regularization techniques during training, each targeting a different failure mode of deep neural networks. Both are standard tools applied with principled choices of hyperparameter values. Residual Dropout (Pdrop = 0.1) Dropout is applied to the output of each sub-layer before it is added to the sub-layer input and normalized. Additionally, dropout is applied to the sums of embeddings and positional encodings in both the encoder and decoder stacks. Rate Pdrop = 0.1 for the base model; 0.3 for the big English-French model. Dropout prevents co-adaptation of features and is essential for generalization in overparameterized networks. Label Smoothing (εls = 0.1) Instead of training the model to output probability 1.0 for the correct token, label smoothing distributes εls = 0.1 of the probability mass uniformly across all tokens. This hurts perplexity — the model is trained to be slightly uncertain — but improves accuracy and BLEU score on the evaluation set. It prevents the model from becoming overconfident on training examples, improving calibration and generalization. Chapter VII: Results — A New State of the Art Results 28.4 — EN-DE BLEU Transformer (big) on WMT 2014 English-to-German — over 2 BLEU above all prior ensembles 41.8 — EN-FR BLEU New single-model state-of-the-art on WMT 2014 English-to-French after just 3.5 days of training 3.5 — Days to Train Big model training time — a small fraction of the training cost of prior competitive models 8 — P100 GPUs Single machine, 8 GPUs — commodity hardware producing world-best translation quality Translation Results vs. Prior State of the Art (Table 2) The comparison below presents the decisive empirical case. The Transformer (big) achieves 28.4 BLEU on English-German — surpassing even ensembles of the best prior models — and 41.8 on English-French, establishing a new single-model record. Both achievements come at a fraction of the training cost. | Model | EN-DE BLEU | EN-FR BLEU | EN-DE FLOPs | EN-FR FLOPs | | --- | --- | --- | --- | --- | | ByteNet | 23.75 | — | — | — | | GNMT + RL | 24.6 | 39.92 | 2.3 × 10¹⁹ | 1.4 × 10²⁰ | | ConvS2S | 25.16 | 40.46 | 9.6 × 10¹⁸ | 1.5 × 10²⁰ | | MoE | 26.03 | 40.56 | 2.0 × 10¹⁹ | 1.2 × 10²⁰ | | ConvS2S Ensemble | 26.36 | 41.29 | 7.7 × 10¹⁹ | 1.2 × 10²¹ | | Transformer (base) | 27.3 | 38.1 | 3.3 × 10¹⁸ | — | | Transformer (big) | 28.4 | 41.8 | 2.3 × 10¹⁹ | — | The Transformer base model alone outperforms every prior single model and ensemble on English-German — at just 3.3 × 10¹⁸ FLOPs, the lowest training cost of any competitive model in the table. The big model then exceeds even the best ensemble by over 2 BLEU. This is not a marginal improvement. It is a complete reordering of the competitive landscape. Model Variations: Ablation Study (Table 3) To understand which architectural choices matter most, the authors systematically varied the base model — one component at a time — and measured the effect on English-German translation performance on newstest2013. The findings reveal important insights about the sensitivity of transformer performance to hyperparameter choices. Row (A): Number of Attention Heads Varying h while keeping total computation constant. Single-head attention (h=1) is 0.9 BLEU worse than the best setting. Too many heads (h=32) also degrades quality. The sweet spot is h=8 — enough heads to capture diverse relationships, few enough that each head has sufficient dimensionality (dk = 64) to be expressive. Row (B): Attention Key Size Reducing dk consistently hurts model quality. This suggests that dot-product compatibility is a nontrivial function to learn, and that reducing the space in which compatibility is computed impairs the model's ability to identify meaningful relationships. A richer compatibility function — potentially beyond dot product — may be beneficial. Rows (C) & (D): Model Size and Dropout Larger models are better — monotonically. dmodel = 1024 with dff = 4096 yields 26.0 BLEU vs. 25.8 for base. Dropout is essential: removing it (Pdrop = 0.0) collapses performance significantly. Adding it (0.2) provides marginal overfitting protection. The base dropout of 0.1 is well-calibrated. Row (E): Positional Encoding Replacing sinusoidal positional encodings with learned positional embeddings yields 25.7 BLEU vs. 25.8 for sinusoidal — virtually identical. This validates the sinusoidal choice as principled rather than merely pragmatic: it matches learned embeddings on training distribution while offering potential extrapolation benefits at longer sequence lengths. Chapter VIII: English Constituency Parsing Generalization To demonstrate that the Transformer is not a translation-specific architecture, the authors apply it — with minimal modification — to English constituency parsing. This task is structurally very different from translation: the output must satisfy strong syntactic constraints, is significantly longer than the input, and RNN sequence-to-sequence models have historically underperformed in low-data regimes. A 4-layer Transformer with dmodel = 1024 is trained on the Wall Street Journal portion of the Penn Treebank (~40K sentences) and in a semi-supervised setting using ~17M sentences from high-confidence and BerkleyParser corpora. No task-specific architectural modifications were made — only dropout, learning rate, and beam size were tuned on the development set. Parsing Results (Table 4) The Transformer's parsing results are striking precisely because they are achieved without any task-specific design choices. In both the WSJ-only and semi-supervised settings, the model outperforms a wide range of purpose-built parsers. | Parser | Training Setting | WSJ 23 F1 | | --- | --- | --- | | Vinyals & Kaiser et al. (2014) | WSJ only, discriminative | 88.3 | | Petrov et al. (2006) | WSJ only, discriminative | 90.4 | | Dyer et al. (2016) | WSJ only, discriminative | 91.7 | | Transformer (4 layers) | WSJ only, discriminative | 91.3 | | McClosky et al. (2006) | semi-supervised | 92.1 | | Vinyals & Kaiser et al. (2014) | semi-supervised | 92.1 | | Transformer (4 layers) | semi-supervised | 92.7 | | Luong et al. (2015) | multi-task | 93.0 | | Dyer et al. (2016) | generative | 93.3 | The semi-supervised Transformer (92.7 F1) surpasses all prior semi-supervised and discriminative parsers. Even the WSJ-only model (91.3 F1) outperforms the BerkeleyParser on the same data — a result that RNN sequence-to-sequence models had historically failed to achieve. This demonstrates that the Transformer architecture generalizes across modalities of structured prediction tasks. Chapter IX: Attention Visualizations Interpretability One of the most scientifically compelling sections of the paper presents direct visualizations of learned attention patterns. Unlike the weights of a recurrent network — which are distributed across time and difficult to interpret — attention weights are explicit, normalized distributions that can be visualized as a matrix of token-to-token affinities. What these visualizations reveal is extraordinary: individual heads appear to have learned distinct, linguistically meaningful behaviors entirely from the translation objective — with no explicit linguistic supervision. Figure 3: Long-Distance Dependencies This visualization from encoder self-attention at layer 5 of 6 shows attention patterns for the word "making" in the sentence: "It is in this spirit that a majority of American governments have passed new laws since 2009 making the registration or voting process more difficult." Many attention heads converge on "more" and "difficult" — correctly identifying the long-distance syntactic dependency that completes the phrase "making...more difficult" across more than 8 intervening tokens. Different colors correspond to different attention heads. The model has learned, from translation data alone, to track the complement structure of an English verb phrase at long range. Figure 4: Anaphora Resolution Also from layer 5 of 6, this figure shows two attention heads apparently specialized for coreference resolution. The sentence is: "The Law will never be perfect, but its application should be just..." When the isolated attention from the word "its" is visualized for heads 5 and 6, both heads attend with sharp, concentrated weight to the word "Law" — the correct antecedent, located 6 positions earlier. The attention is described as "very sharp" for this word — an emergent specialization for anaphora that was never explicitly trained. Figures 5a & 5b: Structural Specialization Across Heads Head A (green): Diffuse attention distributed across many positions — a "global context" head that integrates information from across the sentence broadly. Head B (red): Sparse, localized attention along syntactic boundaries and phrase breaks — a "structural" head tracking constituent structure across the sentence. Two heads from the same layer (encoder self-attention, layer 5) on the same sentence exhibit qualitatively different behaviors. Neither head was told what to learn. Both emerged from gradient descent on the translation objective. This is multi-head attention doing exactly what it was designed