Transformer

The story, event by event

Every point below is traced to a real source — nothing on this page is invented.

1. 12 Jun 2017impact 98

   Attention Is All You Need

   The paper that introduced the transformer — an architecture built entirely on attention, with no recurrence or convolution.

   A self-attention encoder-decoder let every token attend to every other token in parallel, replacing the step-by-step recurrence of RNNs and LSTMs as the default way to model sequences.

   arXiv

   Because attention parallelises across the whole sequence, training time scaled with available hardware rather than sequence length — the property that made internet-scale pre-training economically possible.

   Google AIIllustrated

   Every model in this development descends from this paper. It is the origin event of the entire current wave of AI.

2. 11 Oct 2018impact 80

   GPT-1 & BERT prove pre-training at scale

   Two models showed a transformer pre-trained on raw text learns representations that transfer to almost any downstream task.

   GPT-1 took the autoregressive route — predict the next token — and showed the same backbone generates as well as it understands, fine-tuned with minimal task-specific change.

   OpenAI

   BERT pre-trained bidirectionally with a masked-language objective and set new state-of-the-art across a wide span of language-understanding benchmarks.

   arXiv

   This collapsed NLP from many task-specific models into one pre-trained backbone fine-tuned everywhere — the template the whole field still follows.

3. 28 May 2020impact 88

   GPT-3 — 175B parameters proves scaling

   A 175-billion-parameter model demonstrated that capability emerges from scale alone, with no task-specific training.

   With 175B parameters and a 2,048-token context, GPT-3 performed new tasks from a few examples in the prompt — few-shot learning — without any gradient updates.

   arXiv

   Proving that scale buys capability triggered the race to ever-larger models that defines the field's economics to this day.

4. 22 Oct 2020impact 70

   Vision Transformer (ViT) unifies modalities

   The Vision Transformer proved attention works on images, by treating image patches as tokens.

   Splitting an image into fixed patches and feeding them to a standard transformer matched or beat convolutional networks at scale — no vision-specific architecture required.

   arXiv

   Collapsing vision and language onto one architecture was the precondition for today's multimodal models.

5. 27 May 2022impact 66

   FlashAttention makes long context tractable

   An IO-aware reformulation of exact attention that made long contexts computationally tractable without any approximation.

   By tiling the attention computation to keep it in fast on-chip SRAM and never materialising the full attention matrix, FlashAttention cut memory use from quadratic toward linear in sequence length.

   arXiv

   The follow-up FA-2 delivered roughly an 800% training speedup over a standard attention baseline, turning long-context training from impractical into routine.

   arXiv

   Without it, the million-token context windows that arrived two years later would not have been physically affordable.

6. 1 Dec 2023impact 58

   State-space / Mamba hybrids emerge

   A selective state-space model matched Transformer quality while scaling linearly, not quadratically, with sequence length.

   Mamba replaces attention with a selective state-space mechanism that scales linearly in sequence length, delivering roughly 5x higher inference throughput than a same-size Transformer.

   arXiv

   At the 1-3B parameter scale tested in the paper, Mamba matched or beat Transformers of the same size on language modeling quality — the first credible evidence an attention-free architecture could compete, not just run cheaper.

   arXiv

   Within months, hybrid designs like Jamba interleaved Mamba layers with attention and mixture-of-experts blocks, keeping attention where it matters most while inheriting Mamba's linear-time long-context efficiency.

   arXiv

   Linear-time sequence models challenge quadratic attention on long context — the succession pressure that resolved into hybrids rather than a clean replacement.

7. 11 Dec 2023impact 64

   Sparse Mixture-of-Experts (Mixtral 8x7B)

   Sparse mixture-of-experts activates only a fraction of a model's parameters per token, breaking the link between quality and cost.

   Mixtral 8x7B routes each token to 2 of 8 expert sub-networks, so it carries the quality of a large model while paying the inference cost of a much smaller one.

   arXiv

   This is the dominant efficiency pattern in frontier models since — the economics that make serving them at scale viable.

8. 15 Feb 2024impact 62

   Million-token context (Gemini 1.5)

   Context windows expanded from a few thousand tokens toward one to two million — a roughly 500x jump in what a single forward pass can hold.

   Gemini 1.5 demonstrated near-perfect recall across a 1M-token window in long-context retrieval tests, holding whole codebases, books, or hours of video in one prompt.

   Google

   It shifts work from external retrieval back into the model itself — changing how applications are architected around LLMs.

9. 9 Feb 2026impact 60

   Transformers.js v4 Performance Enhancements

   new runtime: WebGPU Runtime in C++; model support: Support for models exceeding 8B parameters; performance improvement: 4x speedup for BERT models

10. 26 Feb 2026impact 60

    Integration of Mixture of Experts into Transformers

    details: {'industry_adoption': 'Recent industry adoption of MoEs', 'engineering_improvements': ['New weight loading pipeline', 'Introduction of an Experts Backend system']}; description: MoEs enhance efficiency and scalability of Transformers.

Lineage