Machines That Learned · Part 5

Why 2012 and Not 1992: The Breakthrough That Was Not an Idea

In one year an error rate that had been creeping down by a point at a time fell by ten. Nothing conceptual had been invented. The architecture was from 1980, the training method from 1986 — what arrived was a database and a games console component.

The question: Why did a familiar learning method become a striking winner in 2012?

Big idea: AlexNet's leap came from combining established methods with a shared labeled dataset and affordable parallel computation, then measuring a specific image task.

There is a chart that gets shown at the start of a great many talks, and it deserves to be, because the shape of it is genuinely strange.

The ImageNet Large Scale Visual Recognition Challenge asked systems to label photographs across a thousand categories. The scoring metric is top-five error: the system offers five guesses, and it fails only if the correct label is in none of them.

In 2010 the best system scored 28.2%. In 2011, 25.8% — the kind of improvement a competitive field produces annually.

In 2012 the winner scored 15.3%. The runner-up that year was at 26.2%, right on the old trend line.

That is not an improvement. It is a different method arriving.

What AlexNet was

The winning entry came from Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton at Toronto: a deep convolutional neural network, trained by backpropagation, on two consumer graphics cards.

Every component of that sentence was old.

Convolution came from Kunihiko Fukushima's neocognitron in 1980 — a network of local receptive fields repeated across the image, so that a feature detector learned in one place applies everywhere, itself inspired by Hubel and Wiesel's work on the cat visual cortex. Training convolutional networks by backpropagation came from LeCun in 1989, and by the late 1990s that lineage was reading a large share of the handwritten cheques written in the United States.

The 2012 network had genuine engineering improvements — rectified linear units instead of saturating activations, dropout as a regulariser, an efficient implementation across two GPUs. These mattered. They are not a new theory of learning.

So the question is not what was invented in 2012. It is why the same family of ideas produced a modest result in 1992 and a rout in 2012.

The data

ImageNet was not a machine learning project. It was a database project, and it was widely considered a poor use of a researcher's time when it started.

Fei-Fei Li's argument, running against the consensus of the period, was that the field was over-focused on better algorithms for small datasets and under-focused on the datasets. Children, she pointed out, see an enormous number of examples. Models were being asked to generalise from a few thousand.

The project, from 2009, assembled millions of labelled images organised around WordNet, an existing lexical hierarchy — a decision that saved inventing a category scheme and made the labels mean something linguistically. The labelling was done through crowd work at a scale that had not previously been attempted for a research dataset.

Then, crucially, it was run as a public annual competition with a fixed protocol, which made results comparable. Without that, 2012 would have been one lab's claim rather than a demonstrated fact everyone could check.

This is the unglamorous half of the breakthrough and it is usually given one sentence. It should get more. The bottleneck had been examples, and someone spent years on the boring work of removing it.

The hardware

The other half arrived from an industry with no interest in any of this.

Graphics processing units are built to apply the same operation to enormous numbers of values simultaneously, because that is what rendering a scene requires. Training a neural network is, computationally, a great deal of matrix multiplication — which is the same shape of work.

The AlexNet paper is candid about the constraint: the network was split across two GPUs because it did not fit on one. That is not a design principle, it is a hardware limit being worked around, and it is a good indication of how close to the edge the whole thing was.

What the GPUs bought was iteration speed. Training runs that would have taken months became days, which meant architectures could be tried, evaluated and abandoned. Research proceeds at the speed of its feedback loop, and the loop had just got an order of magnitude tighter.

Nobody at any graphics company designed for this. It is the second time in this history that a field was unblocked by something invented elsewhere for another purpose — the first being reverse-mode differentiation arriving from numerical analysis.

After

The following years look like normal rapid progress once the method is established. ZFNet in 2013 at 11.7%, largely from a careful analysis of what the layers were actually detecting. GoogLeNet in 2014 at 6.7%. ResNet in 2015 at 3.6%, using residual connections that let the error signal skip layers — which is, once again, a fix for the vanishing gradient problem from part 4, in a new costume.

By 2015 the error rate was below commonly cited estimates for human performance on the same task, a comparison worth treating with care: the human figure came from a small number of people doing a task with a thousand fine-grained categories, many of them dog breeds, and it measures something narrower than "seeing".

The ImageNet challenge stopped running in 2017. Its problem was solved.

The moral, which is not the usual one

The story is usually told as a triumph of an idea that was ahead of its time, persisted with by a few believers until the world caught up. There is truth in that; Hinton, LeCun and Bengio did keep working through a period when the field had a proof against them, and the 2018 Turing Award recognised exactly that.

But the more useful reading is less romantic.

Neither of those was going to be produced by thinking harder about neural networks.

Which raises the question the field lives with now, and it is a genuinely open one. If the constraint was never the ideas, then what is the constraint currently — and would we recognise it if it were, again, something being built in an adjacent industry for entirely unrelated reasons?

Part 6 is about the last of the old ideas to have its moment, and about why it brings this series back to where the memory articles started.