ARTIFICIAL INTELLIGENCE
The 2017 Transformer: attention becomes the architecture
The original Transformer reorganized sequence learning around attention. A worked example explains its routing, positional information, and computational tradeoffs.
Research checked
Newest cited publication
An attention-head schematic illustrates the query, key, and value operations underlying transformer attention. The 2017 Transformer combined attention with the encoder–decoder architecture examined in the article.
Cosmia Nebula · Image source · CC BY-SA 4.0
The Transformer arrived as a proposal about how to organize computation, rather than a claim that machines had acquired a human faculty of attention. On June 12, 2017, researchers released Attention Is All You Need, introducing the Transformer for translation. Its historical importance becomes clearer when we separate three questions: what information can travel, what operations can run together, and what prevents a model from exploiting unavailable information. Small calculations reveal the architecture more clearly than treating its name as an explanation.
Before attention became the backbone
Sutskever, Vinyals, and Le's September 2014 sequence-to-sequence paper trained one recurrent network to encode an input and another to generate its output. Their implementation used long short-term memory, or LSTM, networks. It demonstrated a general translation framework without manually aligning each source position to an output position. The input representation had a fixed dimensionality even when sentence lengths varied. This was a successful experimental design, rather than a proof that every sentence could be compressed without practical loss.
Imagine describing a route containing twenty turns using a notebook with a fixed number of numerical entries. A learned encoding may preserve the instructions well, but it must decide what to retain before all later questions are known. Increasing numerical precision does not make the learning problem disappear. Separately, processing each turn through a state dependent on the previous state creates a chain of computation. Compression pressure and sequential dependence are different constraints, so improving one does not automatically remove the other.
Attention first changed what could be retrieved
Bahdanau, Cho, and Bengio's September 2014 preprint proposed learning to align and translate jointly. Its decoder could consult a weighted combination of source representations when choosing the next output. This kept multiple source positions available instead of forcing every decision through one final source vector. The system still used recurrent computation. Attention therefore predates the Transformer; the 2017 landmark reorganized an existing mechanism into a different computational backbone.
In the route example, retrieval resembles keeping an entry for each turn and consulting the relevant entries when asked where the bridge occurs. A weighted combination allows graded rather than all-or-nothing access. If two entries receive weights 0.8 and 0.2, both contribute, but not equally. Such weights are numerical routing choices, not a declaration that the model understands a landmark. A useful experiment would change or remove a source entry and examine the effect on the answer, rather than infer correctness from a visually concentrated attention pattern alone.
A complete calculation with two memory entries
Scaled dot-product attention compares a query with keys, rescales the resulting scores, and normalizes their exponentials into weights. It then combines the associated value vectors. Self-attention obtains these ingredients from representations of the same sequence. The distinction between a key and a value is useful: a key helps determine where to retrieve information, while a value supplies the information that is mixed into the output.
For example, consider two dimensions. Choose a query of (1, 0), a first key of (square root of two times ln 3, 0), and a second key of (0, 1). Dividing the dot products by the square root of two gives scores ln 3 and zero. Exponentiation gives three and one, so the normalized weights are 0.75 and 0.25. If the values are (2, 0) and (0, 4), the output is (1.5, 1). The result is a numerical mixture rather than a choice of one whole entry. Changing the values changes what is retrieved, even when the attention weights stay fixed.
The model needs order as well as relevance
The 2017 architecture adds position signals, multiple attention heads, and a mask blocking future output positions. The original encoder–decoder model uses encoder self-attention, masked decoder self-attention, and decoder cross-attention to the encoder's outputs. These choices address different requirements: identifying order, representing several relationships, and respecting the information available at prediction time. They should not be collapsed into a claim that one attention map explains everything a model knows.
Consider the sentences 'dogs chase cats' and 'cats chase dogs'. If a sequence operation only permutes its output vectors when its input vectors are permuted, averaging those outputs cannot distinguish the two sentences. An order signal breaks that limitation. Multiple heads offer another kind of flexibility: two separate mixtures can preserve different information before their outputs are combined. They are not automatically assigned interpretable jobs such as grammar or meaning. Meanwhile, a causal mask has an exact bookkeeping role. For four positions, allowing each position to consult itself and earlier positions leaves one plus two plus three plus four permitted connections, rather than all sixteen. No future answer token can be used as evidence.
The surrounding network matters
The 2017 system also includes position-wise feed-forward transformations, residual connections, and layer normalization. The title does not mean that an attention-weight matrix is the whole model. Layer normalization, introduced by Ba, Kiros, and Hinton in July 2016, computes normalization statistics within an individual example's hidden representation, with learned scale and offset parameters. It provides a way to control numerical behavior without requiring those statistics to come from other examples in a batch.
For a small calculation, take a three-component vector (2, 4, 6). Its mean is four and its population variance is eight thirds. Subtracting the mean and dividing by the standard deviation produces approximately (−1.225, 0, 1.225), before learned scale and offset. This example omits the small stabilizing constant used in implementations. Normalizing does not prove the representation is informative. It changes the coordinates through which learning operates. Similarly, a residual path can preserve an existing representation while a learned branch proposes a change; its presence says nothing by itself about the truth of a generated sentence.
Parallel training still has a computational price
The larger Transformer reported 28.4 BLEU on the WMT 2014 English-to-German benchmark. BLEU, short for Bilingual Evaluation Understudy, compares translations with references rather than measuring a percentage of perfect sentences. Training allowed parallel work across positions, but autoregressive generation still depended on previously produced outputs.
A useful distinction is whether the target sequence is available. During training, known earlier target tokens can be supplied while masks prevent looking ahead. Several predictions can therefore be evaluated together. At generation time, the next token is unavailable until it is produced. Attention also trades a long sequential path for potentially many pairwise comparisons. An illustrative full attention table has sixteen entries for four positions and sixty-four for eight positions. Doubling length quadruples that table's size. This arithmetic describes the attention-score component, rather than the complete model's runtime or memory bill. Faster communication between distant positions does not make arbitrarily long contexts free.
What to carry forward from the 2017 result
The historical claim is specific: an attention-centered sequence architecture worked competitively in measured translation experiments while changing the structure of computation. It did not establish that recurrence was useless, that position ceased to matter, or that an attractive attention visualization certified an explanation. Its design can be understood as a separation of concerns: routing information, transforming representations, preserving order, and controlling access to future outputs. Each part has a testable mathematical role.
For someone implementing a small version, useful checks follow directly from those roles. Permute inputs while controlling position signals, change one value while holding keys fixed, and verify that masked locations contribute no weight. Compare memory growth across sequence lengths rather than extrapolating from a short demonstration. A routing error should affect the matching tests, while an incorrect mask should fail even on an otherwise plausible output. The landmark deserves attention because a concrete architectural choice made strong results possible, and because its mechanisms remain understandable enough to question, reproduce, and improve without relying on the mythology of a name.
Sources & further reading
Primary papers, project documentation, and research releases used in this article. Publication dates below refer to the sources themselves.
- Attention Is All You NeedVaswani and colleagues / arXivPublished Checked
- Sequence to Sequence Learning with Neural NetworksSutskever, Vinyals, and Le / arXivPublished Checked
- Neural Machine Translation by Jointly Learning to Align and TranslateBahdanau, Cho, and Bengio / arXivPublished Checked
- Layer NormalizationBa, Kiros, and Hinton / arXivPublished Checked
FOLLOW THE QUESTION
Study engineering foundations