Machines That Learned · Part 2

The Perceptron: A Machine That Learned, and the Line It Could Not Draw

In 1958 a psychologist built a machine that adjusted its own wiring, and a newspaper reported it would soon be conscious. Eleven years later a book proved a real limitation. The story everyone tells about what happened next is mostly wrong.

The question: What could the perceptron learn, and why did its limits matter?

Big idea: The perceptron learned real linear boundaries, but XOR exposed a representational limit and hidden layers posed a separate training problem.

Frank Rosenblatt was a psychologist, not an engineer, and the distinction matters for understanding what he built. He was not trying to make a useful classifier. He was trying to build a physical model of how a brain could come to recognise things without being told how.

His 1958 paper in Psychological Review introduces the perceptron as a "probabilistic model for information storage and organization in the brain". The hardware followed: the Mark I Perceptron, built at Cornell Aeronautical Laboratory, took input from a 400-pixel camera — a twenty by twenty grid of photocells — and held its weights in motor-driven potentiometers. Learning was mechanical. When the machine was wrong, motors turned dials.

It worked. Given examples, it learned to separate simple shapes and characters, and nobody had told it how.

The press conference

Then came the part that set a template the field has followed ever since.

The 1958 coverage — a Navy press conference, reported in The New York Times — described the perceptron as the embryo of an electronic computer expected to walk, talk, see, write, reproduce itself and be conscious of its existence.

It is easy to enjoy this at Rosenblatt's expense, and unfair to. The claims were speculative extrapolation by people who had just seen something genuinely new — a machine that improved by itself — and had no calibration for how far that would go. Nobody did. That is the recurring condition.

But the announcement created a debt, and the debt came due.

What was actually proved

Underneath the theatre there was a real theorem, and it has outlived everything else from that decade.

The perceptron convergence theorem — argued by Rosenblatt and proved cleanly by Albert Novikoff in 1962 — says that if the training examples can be separated by a straight cut, the perceptron learning rule will find such a cut in a finite number of steps. Not maybe. Always.

That is a strong guarantee and it established something genuinely new.

Every gradient-trained model since works this way.

Around the same time, Widrow and Hoff at Stanford developed ADALINE and the least-mean-squares rule, a closely related idea that went on to become standard in adaptive signal processing — and which, unlike the perceptron, was quietly used in production for decades without anyone declaring a winter.

The limitation

The guarantee has an obvious hole once you look at it, and it is the hole Minsky and Papert made famous in Perceptrons (1969).

"If the examples can be separated by a straight cut." What if they cannot?

Take two binary inputs. AND fires only when both are on: one corner of the square, three others quiet, and a single line separates them. Fine.

Now take XOR — "either, but not both". The firing cases sit at diagonally opposite corners. No straight line puts those two on one side and the other two on the other. There is no such line, so the convergence theorem promises nothing, because its precondition is not met.

A single-layer perceptron cannot learn XOR. This is not an engineering shortfall. It is a proof.

Minsky and Papert's book is a careful piece of mathematics that goes considerably further, analysing which predicates single-layer perceptrons can and cannot compute, including results about how the required weights scale for problems like connectedness. The mathematics is not in dispute. It was right.

The story that is wrong

What is in dispute is the legend: that this book single-handedly killed neural network research for fifteen years.

Three things complicate it.

The authors knew about multiple layers. Stacking layers solves XOR trivially — the hidden layer builds a new representation in which the problem becomes linearly separable. Minsky and Papert discuss multilayer networks. Their position was that no learning procedure for the middle layers was known and that they were sceptical one would be found. That is a claim about the state of the art, and it was accurate in 1969. It was not a claim that layers do not help.

The timing does not fit neatly. Funding for connectionist work was already under pressure. The major cuts of the early 1970s — which part 3 is about — hit symbolic AI at least as hard, and they came from reports that had nothing to say about perceptrons.

Historians have looked. Mikel Olazaran's 1996 study of the controversy examined how the "official history" formed and found a more ordinary story: a research community with limited funding, two competing programmes, and a rhetorical victory that got retold until it became a single dramatic cause.

What is true is that the book gave the symbolic side a citable argument at a moment when resources were contested, and that neural network research became much harder to fund for roughly the next fifteen years. That is a real effect. It is not the same as one book stopping a field.

Rosenblatt died in a boating accident in 1971, at 43. He did not see any of what followed.

What the gap actually was

Here is the precise statement of where things stood in 1969, because it explains the shape of the next twenty years.

Layers help. Everyone knew layers help. The problem was that the perceptron rule updates a weight using the error at the output, and for a weight in a hidden layer there is no direct error to use — nobody told that unit what it should have produced. It has no target of its own.

This is the credit assignment problem: when the output is wrong, which of the internal weights is to blame, and by how much?

Without an answer, the middle layers cannot be trained, and a network whose middle layers cannot be trained is a single-layer network wearing extra hardware.

The answer existed. In 1969 it had already been published, in a master's thesis in Finland, in a completely different field, and nobody in this story had read it.

That is part 4. Part 3 is what happened to the field while it waited.