Alex Wang · 2026-10-15 · notable
Alex Wang — you could have invented transformers from memory
Alex Wang explains linear layers, MLPs and self-attention as associative memory stores and works out how many memories each can hold. The essay reached 274 points on Hacker News.
A former Cartesia research scientist rebuilds the transformer layer by layer as a series of better key-value memory stores.
What is it?
In 'You could have invented transformers from memory', Alex Wang treats each part of a transformer as a store that saves key-value pairs and recalls a value by key similarity. Starting from a simple outer-product store, he adds one fix at a time until he arrives at MLPs and softmax attention.
How does it work?
Wang estimates the capacity of each store before interference ruins recall. With a key dimension of 64, an outer-product store holds about 64 memories, and least-squares fitting doubles that. A squared-ReLU nonlinearity raises it to about 2,700, roughly 40 times the linear store, and softmax attention at the usual sharpness reaches about 1.2 million. Scaling the sharpness with the log of the context length keeps recall working as context grows.
Why does it matter?
The essay gives a short, math-light path to why attention and MLPs look the way they do, with code published on GitHub. Wang notes the capacity numbers are one lens only, since they ignore what the training data actually encodes.
Who is it for?
ML engineers and students learning how transformers work