Machines That Learned · Part 6

Attention Is a Memory Machine, and That Is Why This Series Meets the Other One

Two fields spent decades on what they thought were different problems. Both arrived at the same answer: storage was never the hard part. Everything interesting is in how you get back to what you already have.

The question: What do human memory cues and machine attention have in common?

Big idea: Both face retrieval by a usable cue, though their mechanisms differ; design storage around the question you will later have.

By 2014, machine translation worked roughly like this. A recurrent network read the source sentence one word at a time, updating an internal state. At the end, that state — a single fixed-size vector — was supposed to contain the sentence. A second network then generated the translation from it.

The approach worked, and it had an obvious and much-discussed flaw. Everything about a sentence of any length had to survive being squeezed through one fixed-size representation. Performance degraded as sentences got longer, in the way you would expect from a system being asked to hold more in a container that does not grow.

The fix, from Bahdanau, Cho and Bengio, was to stop squeezing. Keep a representation of every source position. Then, when generating each output word, let the model look back over all of them and compute how relevant each one is right now.

They called it attention, and the name has caused some trouble, because what it describes is not really a spotlight. It is a lookup.

Lookup by content

A conventional computer memory is addressed by location. You ask for cell 4,096 and you get what is in cell 4,096. To find something you must already know where you put it.

Attention does something else. It takes a query — a representation of what is needed right now — and compares it against a key attached to every stored item. The comparison produces a match score for each. Those scores are normalised into weights, and what comes back is a weighted blend of the stored values.

That is content-addressable memory, and it is what your own memory does. You do not retrieve a fact by its storage location. You supply a fragment — a smell, half a sentence, a corner of a room — and the system returns whatever best completes it.

The 2017 paper "Attention Is All You Need" took the obvious next step: if this retrieval operation is doing the work, remove the recurrence entirely. Let every position attend to every other position directly, in parallel. That is the transformer, and it is the architecture behind current language models.

The practical consequence was less about elegance than about hardware. Recurrence forces sequential processing — step two cannot begin until step one is finished. Attention over all positions is a matrix operation, which is exactly the shape of work that the GPUs from part 5 do well. The architecture scaled with the hardware that existed, and that is a large part of why it won.

Three machines, one problem

Once you notice that the question is retrieval, a lot of this history rearranges itself.

John Hopfield, 1982. A network whose stable states are the stored patterns. Present a corrupted or partial version and the network settles into the nearest stored one. It is explicitly an associative memory, built to be content-addressable, and Hopfield's framing came from physics — energy landscapes and spin glasses. This work, with Hinton's, was recognised by the 2024 Nobel Prize in Physics.

Hochreiter and Schmidhuber, 1997. A cell that holds a value across time, with gates deciding what is written, kept and read. An explicit store with explicit access control, invented to solve what looked like a gradient problem and was equally a memory problem.

Vaswani and colleagues, 2017. No persistent store at all — instead, every position is available to every other, retrieved by content match, in parallel.

There is a fourth worth naming because it made the connection explicit rather than implicit: the Neural Turing Machine, from Graves, Wayne and Danihelka in 2014, which attached a neural controller to an external memory matrix with differentiable read and write operations addressed by both content and location. It was a direct attempt to give a network the memory architecture of a computer, and its addressing mechanism is a close relative of attention.

Where the two series meet

The memory articles on this site argued something specific: that human memory is not a filing cabinet with a bad index, but a map running on navigation hardware, and that the reason abstract material slides off is not that it is hard. It is that it is unaddressed.

The evidence for that was a set of numbers that do not fit a storage story. People recognise thousands of photographs seen once, and lose nine digits in twenty seconds. Capacity is obviously not the constraint. Addressing is.

Eighty years of this history arrived at the same place from the opposite direction, and mostly without meaning to. Every time a network hit a wall, the wall turned out to be about getting back to something rather than holding it:

• One layer of weights cannot represent enough, so learn representations — and then you must solve credit assignment to reach the ones in the middle.
• Recurrent state loses the past, so add gates that control what is written and read.
• A fixed vector cannot hold a sentence, so keep everything and retrieve by content.

At no point was the answer "store more". Every time, the answer was a better way of getting to what was already stored.

I want to be careful with this parallel, because it is the kind that becomes a bad metaphor if it is pushed. Attention is not how the hippocampus works. Transformers are not models of cognition, and the McCulloch–Pitts neuron was a poor model of a nerve cell in 1943 and has not improved. The similarity is at the level of the problem, not the mechanism: two very different systems, built under completely different constraints, both discovered that retrieval is where the difficulty lives.

That convergence is interesting on its own terms. It does not license claims about machines remembering the way people do.

The practical version

There is one thing that does transfer directly, and it is the reason both of these series are on the same site.

If addressing is the constraint, then the productive question when you are trying to learn something is not "how do I store this more firmly". It is "what will I have in my hand when I need this, and does the thing I am building connect to that?"

That is what a memory palace does mechanically. It manufactures a key — a place, in a fixed order, that you will definitely be able to reach — and attaches the material to it. The place is the query. The content is the value.

Eighty years of machine learning worked out, expensively, that this is the part that matters. It is a reasonable thing to know before your next study session.