Foundation Models · since 2012
Deep Learning Revolution
The 2012 moment when deep neural networks trained on GPUs decisively beat hand-engineered computer-vision systems, reigniting the field after decades of stagnation. AlexNet's ImageNet win proved that representation learning at scale — not feature engineering — was the path forward, and established the GPU + big-data + backprop recipe that every later development inherits.
6
events traced
6
source records
30 Jun 2026
first signal
30 Jun 2026
last activity
Who drove it
Key movements
26.2%→15.3%−11pt
ImageNet top-5 error (AlexNet vs prior SOTA)
2012origin
the breakthrough year
The story, event by event
Every point below is traced to a real source — nothing on this page is invented.
9 Oct 1986impact 55
Foundations: perceptron → backprop → LeNet
Decades of foundational ideas — the perceptron, backpropagation, and convolutional networks — accumulated long before hardware could make them competitive.
Rumelhart, Hinton, and Williams popularised training multi-layer networks via backpropagation in a 1986 Nature paper, giving the field a general method for assigning credit across hidden layers.
Rosenblatt's 1958 perceptron supplied the original learnable linear unit, and LeCun's 1998 LeNet-5 showed convolutional networks could read handwritten digits at production accuracy — both a decade or more ahead of the compute needed to scale them.
The primitives that would later win ImageNet already existed decades before 2012 — the missing ingredient was compute, not ideas.
20 Jun 2009impact 70
ImageNet establishes the benchmark
A large-scale, human-labeled image dataset gave the field a common benchmark large enough to make deep learning's advantage measurable.
Deng and Fei-Fei introduced ImageNet at CVPR 2009 with roughly 3.2 million labeled images at publication, organised across the WordNet hierarchy — orders of magnitude larger than prior vision benchmarks.
The dataset's annual ILSVRC challenge gave every computer-vision approach the same test, which is what let AlexNet's 2012 result register as a decisive, comparable win rather than an isolated claim.
Without a benchmark this large, there would have been no shared substrate for AlexNet's margin of victory to mean anything.
30 Sept 2012impact 98
AlexNet wins ImageNet 2012
A deep convolutional network trained on consumer GPUs beat every hand-engineered computer-vision system at ImageNet 2012 by a wide margin.
Krizhevsky, Sutskever, and Hinton's AlexNet cut ImageNet top-5 error to 15.3%, versus 26.2% for the runner-up — a gap far larger than any prior year-over-year improvement in the competition.
The network trained on two NVIDIA GTX 580 GPUs, splitting the model across both cards — proof that commodity gaming hardware, not custom supercomputers, could train competition-winning deep nets.
This is the result that restarted the field — every subsequent development in this map traces back to the recipe AlexNet proved here.
16 Jan 2013impact 62
word2vec — everything becomes a vector
Word2vec showed that words could be represented as dense vectors whose geometry captures meaning, establishing embeddings as the field's default representation.
Mikolov et al.'s 2013 word2vec learned distributed word vectors from raw text such that semantic and syntactic relationships appeared as consistent vector offsets — the canonical example being the analogy king − man + woman ≈ queen.
The method was cheap enough to train on billions of words, which made distributed word representations a practical default input layer for downstream language models rather than a research curiosity.
This established "represent meaning as a vector" as the default move underpinning every representation-learning system since.
1 Sept 2014impact 64
Seq2seq + attention precursor
Encoder-decoder recurrent networks paired with an attention mechanism gave sequence-to-sequence models a way to look back at any part of the input, not just the last hidden state.
Sutskever et al. showed a pair of LSTMs — one encoding a source sequence, one decoding a target sequence from it — could perform end-to-end sequence transduction, such as translation, without hand-built alignment rules.
Bahdanau et al. then added an additive attention mechanism letting the decoder weight and reference every encoder state directly at each output step, removing the fixed-length bottleneck of compressing a whole sentence into one vector.
This is the conceptual bridge from recurrence toward the Transformer — attention as a mechanism existed here three years before it replaced recurrence entirely.
10 Dec 2015impact 74
ResNet — residual connections unlock depth
Residual connections let networks be trained far deeper than before by giving gradients a direct path around each block.
He et al.'s ResNet added identity skip connections around stacked layers, so each block only needed to learn a residual correction rather than a full transformation — which let networks of over 100 layers train without degrading.
An ensemble of residual networks won ILSVRC 2015 with a 3.57% top-5 error, and the skip-connection primitive was carried forward into essentially every deep architecture that followed, including the Transformer's residual-plus-layernorm blocks.
Residual connections became a load-bearing primitive in every architecture since, including the Transformer.
Lineage
Descended from
This is one node. The map holds the whole field.
Watch Deep Learning Revolution — and everything it connects to — grow as real news threads onto the map every day.