An analysis of a one-hour explanation where sixteen models from 1958 to 2023 answer six questions. What is correct in it, what is simplified, what numbers are behind each image, and what follows from this for us.
This is an analysis of an hour-long explanation that conveys one idea: all the big names in deep learning (CNN, LSTM, Transformer, BERT, GPT) are not sixteen separate inventions, but six questions and several recurring ideas. The material carries one example through the entire text: a photograph of a cat and the sentence “the cat sat on the mat.” Each network either looks at the photo or reads the sentence.
The six questions are: what can a single neuron learn, how to look at a picture, how to read a sentence, how to turn one sentence into another, what if every word looks at every other, and how to make the model bigger and read longer. Sixteen architectures answer them in turn, and almost every subsequent one appears because the previous one hit a specific wall.
The value of the material is that it shows the walls, not just the solutions. A single neuron cannot learn XOR. A fully connected network for a photo costs 196 million weights and breaks if the cat moves by eight pixels. A recurrent network forgets the beginning of a long sentence, and this is visible in the gradient numbers. Convolution, skip connection, gates, attention, experts, and fixed memory are six answers to six specific walls.
Progress in deep learning is not an accumulation of tricks, but six recurring ideas: nonlinearity, shared weights, skip connections, gated memory, attention, and route selection. All sixteen architectures are built from these, which is why they can be understood in an hour.
The presentation format here is more important than the content. The author does not follow a chronological order or divide the topic into “convolutions separately, recurrent networks separately.” He asks six questions and answers each with one or more architectures, with each subsequent one appearing as a response to the failure of the previous one. This order makes visible the causal connection that is usually lost: not “ResNet was invented in 2015,” but “networks stopped improving with increasing depth, and skip connections fixed this.”
One cat goes through all the material, and this is not decoration, but a technique. It carries the comparison: the same photograph turns into 196,608 numbers for a fully connected network, into an edge map for a convolutional network, into row 3797 of a table for embeddings, and into a sentence continuation for GPT-2. When the same subject is processed differently, the difference is immediately visible.
It provides a map, not a reference book. If you need to quickly explain why attention appeared or why a skip connection is needed, there is a ready-made order here: a wall, then an answer, then a number that shows that the wall really existed.
Perceptron, 1958, Rosenblatt's work. This is a single artificial neuron, the simplest possible network. Rosenblatt built it to distinguish pictures: light fell on a grid of sensors, which he called the retina, and the perceptron learned which pattern corresponded to which class. He tested it on an IBM 704 machine.
The mechanics are as follows. Input numbers are multiplied by their weights, summed, a bias is added to the sum, and the result passes through an activation function. For a perceptron, activation is a step function: if the sum is above zero, the answer is yes, otherwise no. In the example from the material, there are two inputs, weights 3.1 and 3.1, bias minus 4.9. For the pair of ones and ones, the sum is 1.3, above zero, meaning yes.
A key consequence that the explainer emphasizes and that is worth retaining: if the inputs can be any numbers, then the points where the sum is zero form a straight line. On one side, the answer is yes, on the other, no. This means one neuron is one straight line. Learning is the movement of this line: weights and bias are shifted at each step so that the answers become less incorrect. In the example, after a hundred steps, the error dropped to 0.14 and no incorrect answers remained.
This is not a metaphor. One neuron is indeed equivalent to a linear separator, and this can be verified on paper: the equation of the sum equal to zero defines a hyperplane. For two inputs, it's a line; for three, a plane; for more, a hyperplane. The limit immediately follows from this.
The material pits one neuron against the “exclusive OR” problem: is it true that exactly one of the two inputs is equal to one. Of the four possible pairs, two are true, and they are in opposite corners of the square. Try to draw a single straight line such that both true points are on one side, and both false points are on the other. This is impossible, and the author invites the reader to verify this for themselves.
What happens during training: gradient descent still tries, the line turns, but the error gets stuck at 0.69. This is a coin-flip error, meaning the model knows nothing. The weights then shrink to zero, and all answers drift to fifty-fifty. After one hundred steps, three out of four points are still incorrect.
An error of 0.69 is not "almost there." It is precisely the value given by random guessing with two equally probable classes. Getting stuck at this value means that the model has learned nothing, and no number of steps will change that. It is important to remember: a plateau at 0.69 in a real problem is a signal that the model has hit the limit of its form, not a lack of data.
And immediately the author draws a conclusion that leads to the next model: almost all real "yes or no" problems, for example, "there's a cat in the photo," require a boundary that bends, not a straight line. This means more than one line is needed.
The answer to XOR is to add neurons. A multilayer perceptron, 1986. A hidden layer of four neurons appears between the input and output, and each of them is a separate perceptron with its own weights, meaning each has its own line. Then, instead of a step function, a smooth tanh activation, an S-shaped curve, is used.
Why smooth, not a step function: a step function's derivative is zero almost everywhere, and training gets no direction. A smooth function gives a non-zero slope, and the gradient can flow. This is the first place in the material where it is clear that the choice of activation is not a matter of taste, but a condition for the operability of training.
The main idea then works: without nonlinearity, a stack of linear layers remains linear, i.e., a single straight line that cannot do XOR. Nonlinearity is what turns a set of lines into a curved boundary. The output neuron takes four numbers from the hidden layer, calculates their weighted sum, and outputs a single number: above zero means yes.
One straight line. Can only handle linearly separable problems. Does not solve XOR, error gets stuck at 0.69.
Four straight lines plus nonlinearity. The boundary bends. XOR is solved in 300 steps, error from 0.70 almost to zero.
Next, the author shows the limit of this approach on a clear task: points inside a cat silhouette with two ears versus points outside. A two-layer network with 64 neurons per layer, two thousand training steps, and on four thousand new points the network had not seen, 99.5 percent correct answers, 21 errors. The boundary first becomes a blob, then a head, then ears.
A fully connected network is suitable where the example is already a short list of numbers. An example from the material: a bank evaluates a payment by amount, time, and location and flags suspicious ones. And the same block is found within almost all other models: at the end of a convolutional network and inside each transformer block.
First, you need to understand what a photograph is for a machine. It's a grid of small squares, pixels, 256 by 256 in the example. Each pixel is three numbers: how much red, green, and blue to mix, each from zero to 255. An orange pixel on a cat's fur is 198 red, 137 green, and 71 blue.
Why exactly three: the human eye has three types of color sensors, so three lights in the correct quantities can match almost any color we see. Red and green make yellow, all three make white. This means a photograph is three grids, one for each color, stacked together. In the PyTorch library, this is a tensor of shape 3 by 256 by 256, where each number is scaled to the range from zero to one.
Next, the author asks the question for which this section exists: why not feed all these numbers to the very same fully connected network we just built. The answer is calculated directly in the material, and it also explains the emergence of convolutional networks.
The first problem is cost. A fully connected network accepts one flat list of numbers, so the photograph is cut into rows and laid out into one long string: 196,608 numbers. If each of them is connected to the first hidden layer of a thousand neurons, that's 196,608 multiplied by 1,000 weights, plus a bias for each neuron. That's over 196 million weights in the first layer alone.
The second problem is more subtle and important. The author shifts the cat eight pixels to the right, meaning the photograph remains the same, the cat just moved slightly, and marks all inputs that changed. There are 34 percent of them. Since each input has its own weight, what the network learned about the cat in one place does not transfer to the same thing in another. For the network, this is a completely new picture.
This is the real reason for the appearance of convolutions, and not just weight saving. Even if there were any number of weights, a fully connected network would not be able to recognize the same pattern in different places. Saving is a consequence, not a cause.
Convolutional network, 1998. It solves both problems with one idea: not to flatten the picture into a line, but to leave it two-dimensional and slide a small filter over it. Instead of one weight per pixel, the filter is a window of weights three by three in size, that is, nine weights. The filter stands on nine pixels, multiplies each by its weight, adds them up, and writes one number into the edge map. Then it shifts by one pixel and repeats across the entire picture, row by row.
The fact that the same nine weights look at each location is called weight sharing. This works because an edge on an ear looks like an edge on a mat, and one filter can find both. Hence the economy: sixteen filters of three by three by three colors plus bias give only 448 weights, as opposed to 196 million in the first layer of a fully connected network.
The second wall also falls. We shift the cat by eight pixels, and the edge map shifts with it, because the same filter looks everywhere. It's the same map, just shifted. This means that once it learns what an ear looks like, the network finds it anywhere in the photo.
In the example, the filter's left column is negative, and its right column is positive. This means it activates where dark transitions to light when moving from left to right. This is why it draws the cat's edges. This filter is hand-defined, and a trained network learns similar ones itself.
Convolution never works alone. It is followed by a nonlinearity, the same one as in a fully connected network: negative numbers turn into zero, so edges where light transitions to dark are discarded, and only those the filter was looking for remain. Then comes pooling: each two-by-two area is taken, and only the largest number is kept, so the map shrinks from 256 by 256 to 128 by 128, that is, to a quarter of the numbers, while the edges remain.
This is one block: convolution, nonlinearity, pooling. Then the block repeats, and each time the maps become smaller, and their number grows, until 256 maps of size 8 by 8 are obtained. At the end, the maps are laid out into a single list of 16,384 numbers, twelve times shorter than the original photograph, and a fully connected layer turns it into a label. Here, the flattening no longer interferes: after pooling, the list says what was found, not exactly where.
One filter sees only nine pixels, so the question arises how the network sees the whole cat. The answer lies in repetitions: each block looks at the maps of the previous one, and pooling reduced them, meaning each subsequent block sees a wider piece of the photograph. In a trained network, this is visible as a ladder: edges become textures, textures become parts like eyes and ears, parts become a whole cat. The trained network's answer: “tabby, confidence 0.63”.
The story continues with two numbers. By 1998, a small network like this was reading several million bank checks a day. In 2012, a much larger network called AlexNet won the ImageNet competition, where the task was to categorize photographs into a thousand categories. AlexNet made errors in 15.3 percent of cases with five attempts per photograph, while the subsequent participants made errors in 26.2 percent.
| Year | Event | Number |
|---|---|---|
| 1998 | Convolutional network reads bank checks | Several million checks per day |
| 2012 | AlexNet wins ImageNet | 15.3 percent errors versus 26.2 for subsequent entries |
| 2012 | Training on labeled photographs | 1.2 million photos |
Other participants looked for patterns that people had manually devised. AlexNet learned its filters itself from 1.2 million labeled photos and trained quickly on video cards, that is, on chips made for games. Here, three things converge for the first time: data, computation, and the absence of manual rules.
After convolutional networks, people wanted to make them deeper, because deeper meant better. AlexNet assembled eight layers, the best networks after it sixteen and thirty. Then an obvious question arises: is it enough to simply stack layers on top of each other. A 2015 work led by He Kaiming at Microsoft tested this on two networks of identical shape, one 120 layers deep, the other 156, trained on the same small photos. The deeper one performed worse.
This has a technical name: the degradation problem. Starting from a certain depth, adding layers makes the network worse. Importantly, this is not overfitting: the error is higher on both new photos and training photos. That is, a deeper network is simply harder to train.
The author explains why this is strange. Put a 20-layer network next to a 56-layer network, both ordinary, where each layer feeds only the next. The 56-layer network could copy the first 20 and make its extra 36 layers do nothing, simply passing the input through themselves. This is called an identity mapping. Then it would be exactly as good. This means training should never end up worse, but it does.
The reason is that a stack of layers finds it difficult to learn to do nothing. To simply pass the input unchanged, each layer needs to perfectly precisely select weights. Making something complex is easy, doing nothing is difficult. This is a non-obvious conclusion, and it explains the emergence of the next model.
Residual network, 2015. Idea: instead of forcing a block to redraw its input, give it a copy of the input for free and allow it to learn only the correction. The block contains two layers with weights; let's call what they compute f of x. The new thing is a skip connection that carries x past them. Then the block outputs f of x plus x, that is, the input plus the correction, which is called the residual.
Now, to do nothing, a block only needs to learn f equal to zero, which is simple, and zero plus x is just x. There is also a side benefit here, and it explains the second half of the name. Backpropagation carries the gradient backward one layer at a time, multiplying by one factor per layer, and these factors are often less than one. The more layers, the smaller the gradient becomes, until the first layers learn almost nothing. This is called vanishing gradient. The skip helps here too: on the way back, the gradient goes straight through the skip and does not weaken. Twenty blocks give twenty skips, which line up into one straight path from bottom to top.
The result reverses the order: without skips, 56 layers lose to 20; with skips, 56 layers win, 6.97 percent error versus 8.75. In 2015, networks up to 152 layers deep won on ImageNet, and for years to come, residual networks became the standard for images.
The skip is not a trick just for images. Eighteen years later, the same idea will appear in LSTM as a long memory line, and then inside each transformer block. The same move: not to redraw, but to edit.
The material shows the most common way to use it: take a network already trained on a large dataset, change its last layer to one suitable for your task, and fine-tune it a little. In PyTorch's own tutorial, they do this to distinguish an ant from a bee, having only about 120 photos of each.
Millions of labeled photos, days of computation, many video cards are needed. This was the case in 2012.
About 120 photos per class and a little time are needed. Only the last layer changes; the other weights only shift slightly.
This is the same trick that will later appear in language models under the name fine-tuning, and the author specifically emphasizes the connection: first the network learns on a large volume, then it is adapted to a specific task. Expensive once, cheap thereafter.
Transfer learning works because early layers learn general things: edges, textures, simple shapes. They are the same for an ant, a bee, and a cat. Only what is closer to the answer changes. This is why this approach saves so much.
Before this, all inputs had a fixed size. A sentence can be of any length, and the order of words in it is important, and most of what we would like to give a machine to read is text: a letter to sort, a query to search, a page to translate.
First, a word needs to be converted into numbers. A photograph is already numbers, these are pixels, but a word is just a symbol. Therefore, text is cut into tokens, a word or a piece of a word, and each is given a number. "Cat" is number 3797. But the number itself means nothing: 3797 is no closer to "kitten" than to "car."
The answer lies in vector representations of words, from 2013, a work known as word2vec. The idea: give each token its own list of numbers, a vector, learned so that similar words receive similar vectors. How they are learned: a network reads a lot of text and for each word tries to predict the words around it. Or, as in a true GPT-2 language model, the table is learned incidentally with the prediction of the next token. In both cases, words that appear in similar sentences should yield similar predictions, and therefore their vectors turn out to be similar.
| What was checked | Number from the material | Meaning |
|---|---|---|
| Word vector size in GPT-2 | 768 numbers per row | One table row per token |
| Nearest neighbors of the word "cat" | "cats", "kitten", "dog" | Similar words are close together |
| Similarity to dog | 0,55 | Highest of the four checked |
| Similarity to mat | 0,26 | Lowest |
| King minus man plus woman | The closest word is "queen", 0.71 | Directions in space carry meaning |
The last line is the most famous property of such vectors. The arrow from "man" to "woman" points approximately from male to female. Move this arrow so that it starts from "king", and it will point to the area near "queen". In numbers, this is king minus man plus woman. No one told the model about gender or royalty: it picked up these directions simply from predicting words.
The word2vec paper directly states why such vectors are needed: they improve machine translation and question answering. A search for the word "kitten" might find a page that only says "cat", because their vectors are close. And every text model after this, including GPT-2, starts by looking for its words in such a table.
Words became numbers, but the sentence still doesn't read. The problem is the same: all networks before this accepted a fixed size. A sentence comes in any length, five words or five hundred, and word order is important. You can average word vectors into a single fixed-size list, but then the order is lost, and the author gives a vivid example: "not bad, very good" and "not good, very bad" consist of the same four words and would average into the same list with opposite meanings.
The answer is a recurrent network, 1990. It reads the same way you do: one word at a time, carrying memory of everything read. This memory is a short list of numbers, a hidden state. The network reads "cat", writes memory, then reads "sat" along with this memory and writes a new one, and so on until the end.
Why this is smart: it's one cell with one set of weights, reused on each word, so it works for sentences of any length. And memory doesn't grow: everything read is compressed into 32 numbers, so by the end, the last memory holds a compressed summary of the entire sentence.
What comes out and what it's for: anything we train it for. Put a small output layer on each memory and train it to predict the next word, and you get a language model. Put it only on the last memory to label the entire sentence, for example, a positive or negative review. It learns the same way as all networks before it: error, backpropagation, gradient descent.
A recurrent network has a trap, and the material shows it not with words, but with a number. On each word, memory passes through the same weights, so what an early word left in it gets compressed a little at each step. After many words, it almost disappears. It's like a game of telephone: whisper a word down a chain of six people, and by the end, it's gone.
The author checks this in code, and the check is important because it's easy to see. Backpropagation carries the gradient from the error back through each step to all input words, and it is this gradient that allows each word to teach the network something. The test is this: take a cell, run it through a made-up sentence of fifty words, then send the gradient back from the last memory to all words and measure how much reached.
| Word | How much gradient reached | What this means |
|---|---|---|
| Word 50 | About 2 | The last word teaches the network normally |
| Word 40 | About one thousandth | The influence has almost disappeared |
| Word 20 | Several billionth | The word practically does not participate |
| Word 1 | Less than one ten-trillionth | The first word teaches the network nothing |
This is why a recurrent network forgets the beginning of a long paragraph, and this is why it translates long texts poorly. The wall is named, and the next model appears as an answer to it.
It's not that the network is “a little worse.” The first word gets almost no signal, meaning it doesn't participate in training at all. If the meaning of the sentence depends on the beginning, the recurrent network physically cannot use it. This is a limit of form, not data.
Long Short-Term Memory, 1997. It has two memories. The recurrent network's memory is overwritten with each word, so the last few words drown out everything else; this is short-term memory. LSTM keeps it and adds long-term memory: a line that runs along the entire sentence and changes only when its gates allow it.
Example from the material: at the word “cat,” a gate opens and writes “cat” into long-term memory. When the verb comes, another gate opens and reads “cat” back. One cat, therefore “was,” not “were.” That is, the network decides what to remember and what to retrieve, and does so by meaning, not by distance.
Inside one cell, long-term memory is the thick line along the top, and only multiplication and addition touch it. Below, short-term memory and the word go into four layers with weights. The forget gate multiplies the long-term memory to remove outdated information, such as an old subject when a new one appears. The new memory layer proposes what to write, and the input gate decides how much of it will enter. Then the output gate reads the long-term memory as new short-term memory.
The straight line at the top is the whole trick. It makes a short path out of the sentence, just as a skip connection in a residual network makes a short path through layers. The author emphasizes: this is the same idea that ResNet would apply to layers eighteen years later. Not to rewrite, but to edit, and to leave one path along which the signal travels without loss.
The difference is formulated precisely: a recurrent network rewrites its memory at each word, an LSTM only edits its long-term memory. The forget gate erases a little, the input gate adds a little, the rest passes straight through. Therefore, a word like “cat” can travel along this line all the way to the verb and not fade, and learning can just as easily propagate backward. The gates are learned, meaning a trained LSTM itself decides what to store and what to forget. This is why LSTMs were behind machine translation and speech recognition for years.
Count the parts: three gates, two memories, and four layers of weights on each word where a regular recurrent network had one. That's four times more weights and four times more work on each word. From this immediately follows the next question: are all these parts really necessary.
Gated Recurrent Unit, 2014. It preserves the main idea of LSTM, memory that is edited rather than rewritten, but with fewer parts: one memory instead of two and two gates instead of three, an update gate and a reset gate.
The update gate decides how much to change. It mixes the old memory with a new guess, keeping a part of each. This single regulator does the work of LSTM's forget and input gates combined. Where it stays low, the old memory passes straight through, and this is GRU's shortcut, like LSTM's long line. The reset gate decides how much of the past to use when constructing a new guess, so for a new phrase, it can set the past aside.
Result: GRU retains the LSTM shortcut with one memory instead of two and a quarter fewer weights. Its own GRU work applied it inside an English-to-French translator, and, as the author notes, the encoder that will be built below for translation is precisely a GRU.
By this point, the material has gathered three approaches, and it is useful to keep them together, because they differ not in idea, but in cost.
| Model | Year | Memory | Gates | What it provides |
|---|---|---|---|---|
| Recurrent network | 1990 | One, rewritten | No | Reads any length, but forgets the beginning |
| LSTM | 1997 | Two, one is edited | Three | The word travels through the entire paragraph |
| GRU | 2014 | One, being edited | Two | Same, a quarter cheaper |
The general idea for all three is the same: carry memory word by word and use gates to decide what to retain. The difference is how many lines and how many regulators are needed for this. And, importantly for what follows, all three read text one word at a time, which limits learning speed.
It is sequential reading that will displace these models. As long as the network cannot start the next word until it has finished the previous one, most of the computing power is idle. This is the wall from which attention will grow.
We can read a sentence and respond with a label or the next word. But translation is a whole new sentence with its own length and word order. Why one recurrent network won't work: it writes outputs in sync with what it reads, one per word. But French can be longer or shorter than English, and words can be in a different order. This means you need to read the entire sentence before you start writing.
The first answer is sequence-to-sequence, 2014. Two recurrent networks are used. The first, an encoder, reads an English sentence into its memory, and its last memory is compressed into a single vector of 128 numbers. The second, a decoder, starts with this vector and writes French word by word, feeding each word back to choose the next, until it writes an end-of-sentence token.
This is like a human translator who listens to the entire English sentence, holds its meaning in their head, and only then speaks in French. They are trained in the same way: English sentences with French translations are provided, and the decoder learns to guess each subsequent French word, like a language model.
Sutskever, Vinyals, and Le at Google trained a larger version, four layers of LSTM, on English-to-French translation. It achieved 34.8 BLEU, the standard measure of how close a translation is to human, against 33.3 for a traditional system assembled from separate parts. A single network, trained end-to-end, beat a system made of many pieces. Two years later, Google Translate was using this scheme.
Their diagram shows why this works: in a single vector, sentences with the same meaning end up close to each other, even when words are rearranged. "I gave her a card in the garden" and "in the garden I gave her a card" lie close, while "she gave me a card," where the meaning is reversed, lies far away.
This scheme has a weak point, and the material shows it with a number. Everything the decoder knows about English must pass through one vector. And a vector has a fixed size, no matter how long the input. A six-word sentence and a sixty-word paragraph are compressed into the same 128 numbers. The longer the input, the more is discarded.
In the 2014 work by Bahdanau et al., this is visible on a graph: sentence length at the bottom, translation quality at the top. After approximately twenty words, the ordinary encoder-decoder falls apart. The longer the sentence, the worse the French. This has a name: the bottleneck.
The answer from the same work is attention. The idea: don't compress at all. Store all encoder memories, one per English word. Each time the decoder writes a word, let it look at each English word and decide which ones are important.
On the correspondence map, English is across the top, French is down the left side, and each square is how much one French word looks at one English word. “Chat” looks mostly at “cat,” “est” at “sat,” “tapis” at “mat.” And the most interesting thing: no one told the model that “chat” is related to “cat.” These numbers are taken from a real translator.
First: if you store each memory instead of compressing, won't the model bloat as the input grows? No. The learned weights don't grow at all; one small scoring layer is used for each English word, whether there are six words or sixty in the sentence. Only what is stored and the amount of work grows. Second: how does the model know where to look? It isn't told. The scoring layer is trained along with everything else, so it learns to look at “cat” when it writes “chat” because that makes the translation correct.
In the real translator from the material, Marian, there are about 75 million weights, which is small for a translator. For three French words, you can see where each looks the most: “chat” gives 0.70 attention to the word “cat,” “tapis” gives 0.75 to the word “mat.” The French word that is looking comes from one sentence, and the words it looks at come from another, so this is called cross-attention.
Let's add four words to the sentence: “the cat sat on the mat because he was tired.” Who was tired? The cat, of course. To understand this, the word “he” must look back at the labels of earlier words, and the label “animal” is highlighted.
In the translator we drew, only French could look back at English. Within a single sentence, a recurrent network still has to pass “cat” to “he” through its memory, word by word, like in a game of telephone. So, the question is: can words in the same sentence see each other directly?
The answer is self-attention. The same attention we just encountered, but now words in the same sentence look at each other. How should one word score another? Comparing their lists of numbers directly doesn't work: each list best matches itself, so each word would look mostly at itself.
Instead, self-attention gives each word three things, and each answers its own question. The query asks: what am I looking for? For the word “he,” it's something that can be tired. The key answers: what do I have? This is the label the word carries, for example, “animal” for the word “cat.” The value says: if you choose me, here's what I pass on. This is what the word “cat” means.
| Role | Question | Example |
|---|---|---|
| Query | What I'm looking for | The word "he" is looking for something that can get tired |
| Key | What I have | The word "cat" has the label "animal" |
| Value | What I pass if you choose me | The meaning of the word "cat" |
Attention in the translator we drew did not have these names, but it did have roles: the state of the decoder played the query, and each English memory was both what is evaluated and what is mixed, that is, both the key and the value. Self-attention gives each role its own learned layer.
Next, the author demonstrates this on a real GPT-2, where a word only looks backward. It compares its query with the key of each word before it. The better the match, the greater the weight, the thicker the arc. These are GPT-2's own attention weights: the word "cat" gets 0.84, and together they sum to one. Then it collects a mixture of values, weighted by these numbers. After one step, it mainly carries "cat," which made one look, without a game of telephone in between.
Attention works with several copies side-by-side, they are called heads. One head makes one mix of words, meaning it can only track one type of connection. But to understand a word like "he," several connections are needed at once: who it refers to, what came right before it, where the sentence begins. Therefore, each head receives its own query, key, and value and learns its own job. No one assigns these jobs: each head finds its own during training. GPT-2 has twelve heads in each block.
The material does something to be respected: it runs attention through the same gradient test where the recurrent network failed, and places the results side-by-side. Fifty random words pass through attention, then a backward pass is launched from the output of the last word, and the gradient flows back to each input word. Its size on each word indicates how much that word can change the output, that is, how much learning it can receive.
We read the green columns against the red ones. At word forty, attention receives five hundredths. And at word one, it still has eight hundredths, whereas the recurrent network has less than one ten-trillionth there. There is no gradient vanishing here. Distance no longer matters, and the first word moves the output roughly as much as the fortieth. This is why attention can learn from the beginning of a long text.
This is not an improvement of a few percent. This is the removal of a limitation. A recurrent network physically could not use the beginning of a long text, but attention can. Hence the next step: if attention connects any two words, near and far, then a recurrent network is no longer needed.
Attention connects any two words, near and far. So the question is: is a recurrent network still needed? The answer is no. The Transformer, 2017, removes it completely, so all seven tokens pass through at once, without waiting.
One block does two things. First, words talk to each other through attention. Then each word thinks for itself, using a small fully connected network. After each part, the block adds the result back, as a residual network does, so each part only needs to learn the correction.
But attention itself has a blind spot: it ignores word order. The sentences “the cat sat on the mat” and “the mat sat on the cat” give the same set of words to mix, so each word would come out the same in both. Therefore, each word is assigned its position. This is simple and works: “cat” at position two and “cat” at position six start with different numbers, so the two sentences yield different results.
Next, the blocks are stacked into a tower, the very one that guesses the next word. Tokens go into embeddings, positions are added to them, then a block follows, and the block is repeated n times, and an output layer and softmax are placed on top. Tokens enter from below, and from above, for each token, a guess about the next word comes out.
Twelve blocks plus word and position tables give just over 124 million weights, which matches GPT-2's own count. Inside the block, approximately two-thirds of the weights are held by that small fully connected network, not attention. This will be useful when discussing mixtures of experts.
The transformer's first task was translation: the encoder reads English, the decoder writes French, looking back at the encoder via cross-attention. It beat recurrent translators and trained faster because each word passes through it once. In 2020, Google wrote that it had replaced its old recurrent translator with models that read with a transformer but still write recurrently.
In 2017, two things were missing. First: most tasks are not translation. Sorting reviews, answering questions, and ranking search results require a model that understands a sentence but does not write a new one. Second: apart from a word table, each task was trained from scratch on its own labeled examples, and labeling is expensive, whereas the internet contains infinite simple text with no labels at all. So, the question is: can we learn something from all this free text once and reuse it for each task.
This is called pre-training and fine-tuning. First, half of the transformer is trained on simple text, then it is adapted to each task, like a residual network that has learned an ant and a bee. A transformer has two halves, and the question arises as to which one to train.
BERT, 2018, leaves only the encoder. It reads and never writes. It is pre-trained on plain text by playing a fill-in-the-blank game: hide a word, and BERT reads both sides of the blank to reconstruct it. This works because its attention is bidirectional, each word looks at both the words before it and the words after it. In the example from the material, the sentence ends with "purred," and the word "cat" is hidden. Arcs extend from both sides, and the brightest one comes from the last word. BERT's best guess is "cat," and "dog" is second.
Does BERT really use words after the blank? We only change the last word from "purred" to "barked", and the second line prints "dog" with 0.55 confidence versus "cat" with 0.35 in the first case. In this sentence, one word is enough to flip the answer, and it comes after the blank.
How BERT is applied in practice: nobody forces it to fill in the blanks. A pre-trained BERT is taken, a small layer is placed on top, and it is trained on labeled examples of its specific task. This is the same trick with the ant and the bee, and it is fast because BERT already understands words. In the work, pre-training took four days on sixteen Google chips, while fine-tuning took only a few hours on a single GPU.
To sort reviews, a small layer reads one vector, the vector of a special first token, and transforms its 768 numbers into just two: how negative and how positive. Why one vector can represent an entire review: this token is not a real word; BERT places it at the beginning of every input, and in each block, its attention looks at every word, so its vector can gather the entire sentence. Fine-tuning teaches it what exactly to gather for the task. A real BERT, fine-tuned this way on movie reviews, reads two sentences about our cat: “the cat loved this mat” gives positive with 0.999 confidence, “the cat hated this mat” gives negative with 0.998.
| Number | Value |
|---|---|
| 110 million | Weights for the base BERT |
| 12 | Encoder blocks in the base version |
| 4 days on 16 chips | Pre-training in action |
| A few hours on 1 GPU | Fine-tuning for the task |
| 32 million | Monthly downloads of the base model on Hugging Face |
| 1 in 10 searches | Share of searches in the US in English that BERT helped in 2019 |
A 2019 example from Google is illustrative: the query “2019 Brazil traveler to USA need a visa.” Before BERT, the search engine didn't understand what the small word “to” was doing and showed a page about Americans traveling to Brazil. With BERT, which reads each word in context, the US embassy page about visas for Brazilian visitors was found.
BERT is good at understanding sentences, but each task requires its own fine-tuning: its own small superstructure, thousands of labeled examples, its own copy of the model with 110 million weights. The question arises: can one model do any job if its only output is words? Then one could discard special superstructures, write the task itself in words, for example, “help sort this review,” and get the answer also in words.
This turns any job into a writing job. But BERT has a blind spot: it can't write. It fills in one blank using words from both sides, so it never learned to continue a sentence whose end it hadn't seen. Writing requires the opposite game: guessing the next word using only the words before it.
GPT, 2018, short for Generative Pre-trained Transformer, leaves only the decoder. It writes. It is pre-trained on simple text, guessing the next word. Each row is one word, and the highlighted cells are the words it can see. Each word can only look backward, never forward, so everything above the diagonal is closed. That's why the GPT-2 attention we looked at only looked backward: so the model can't peek at the word it's guessing.
The first three guesses for the last word are printed with “floor” in the first place, and “mat” only in the thirty-first. Taking the top word each time, the model finishes writing: “the cat sat on the floor, and the cat was still sleeping.” That is, the model does not reproduce the original sentence, but continues its own.
Next, scale. A much larger decoder, trained on a much larger volume of text, yields the chatbots people use today. And the answer to the original question took three works: the first GPT in 2018 came out a few months before BERT and was still fine-tuned for each task with its own output layer, like BERT. GPT-2 in 2019 showed that one general system should do many different tasks, and language can record the task, input, and output in a single sequence of characters. GPT-3 in 2020 required no fine-tuning at all: the task and a few examples were simply given as text.
All three are assembled from transformer blocks, and one thing distinguishes them: what game decides where attention is allowed to look.
| Model | Half | Game | Where attention looks | What for |
|---|---|---|---|---|
| Transformer 2017 | Encoder and decoder | Translation | Decoder looks at encoder | Translate one sentence into another |
| BERT 2018 | Encoder only | Fill in the blank | Both ways | Understand: search, sort, questions |
| GPT 2018-2020 | Decoder only | Guess the next word | Backward only | Write: continue text |
One type of block, three games. The game decides where attention is allowed to look, and this decides what the model is good at.
Wording from the materialThis is perhaps the most useful table in the entire explanation, because it clears up confusion that has persisted for years. BERT and GPT are not “old and new models,” but two halves of one construct with different viewing rules.
In real systems, boundaries are blurred. In 2020, Google described a translator that reads with a transformer and writes recurrently, meaning it mixes both approaches. And most modern chatbots are a decoder plus a separate fine-tuning stage to make them follow requests. The table is accurate as a map, but not as a description of every product on the market.
A large GPT costs more per word. GPT-3 has 175 billion weights, and the longer the text, the more each new word has to look back. So the question breaks down into two halves: how to make the model bigger and how to read longer.
Let's start with “bigger.” Inside a transformer block, roughly two-thirds of the weights are held by a small fully connected network. A large model knows more, but each word pays for each weight. The question: can the model grow without forcing each word to pay for everything.
Mixture of Experts. The idea: replace this one network with eight, called experts, and add a router that sends each word to only two of them. The model gets many more weights, more room for knowledge, but each word only pays for two experts.
The router is trained just like the scoring layer in attention: no one tells it where to send words, it learns this along with everything else. In the code from the material, the total number of weights is 37.8 million, and each word actually runs 9.4 million. The idea of gated experts dates back to 1991, and a 2017 paper made it big: a green routing network selects two experts, and they are placed between LSTM layers. Today, they are placed inside the transformer block, as in Mixtral, an open model from 2023 with eight experts and two per word.
The Mixtral paper from early 2024 states directly: each word has access to 47 billion weights, but uses only 13 billion. This is more than a quarter, because each word also runs common attention layers. That is, the model became larger, and each word pays approximately as much as a 13-billion-parameter model.
The second half of the question is length, and experts don't solve it. First, you need to understand what GPT holds in memory while writing, and why this grows. GPT writes word by word, and each new word looks back at all previous ones through attention, using their keys and values. These keys and values do not change once calculated, so GPT stores them instead of recalculating: the keys and values of each word in all blocks. This storage is called the key-value cache.
Unlike weights, this is a running record of text, so it grows by one cell for each token. Double the text, and you double what's stored. This is why a long conversation or an entire book costs GPT so much memory.
The answer to length is Mamba, 2023. It returns to the idea of a recurrent network: fixed-size memory. But each word decides for itself how much of it to keep. The update is only multiplication and addition, like LSTM's long-term memory, so a word that has been kept does not fade. As text comes in, the blue cache continues to grow, but the Mamba box never does.
But wait, didn't attention teach us not to compress text into a single memory? This is a trade-off. A fixed-size box cannot store every word precisely, so Mamba allows each word to decide whether to store it. What it loses: it cannot look back at one specific early word the way attention does. This is why many new models mix both approaches: mostly Mamba layers plus a few attention layers. Jamba, a 2024 model, has one attention layer for every seven Mamba layers and processes texts up to 256,000 tokens.
Inside a Mamba block, the input is split into two parts. The main path first goes through a small convolution with a width of four tokens, which mixes each word with several previous ones, and then keeps a fixed-size memory where each word specifies how much to keep. Keep 0.5, and the word fades. Keep 1.0, and it stays. The second path is a gate, like in an LSTM, deciding how much of the result will pass through.
And here is an important difference from LSTM, which explains why Mamba appeared so late. LSTM also keeps fixed memory, but it is obliged to read text word by word even during training, so it is slow on large volumes. Mamba's update is structured so that training proceeds over the entire text at once, in parallel. And it worked: in a 2023 paper, Mamba with three billion weights matched transformers twice its size and wrote about five times faster.
The material concludes by suggesting to look at the sixteen architectures once again and see that they are assembled from a handful of ideas. This is the most valuable conclusion of the entire explanation, and it is worth writing down separately.
| Idea | What it does | Where it is found |
|---|---|---|
| Non-linearity | Allows learning curves, not just straight lines | Perceptron, MLP, each transformer block |
| Shared weights | One set of weights searches for a pattern anywhere | Convolutional networks, recurrent networks |
| Skip connection | Carries the input forward, so layers help, not hurt | ResNet, LSTM, transformer block |
| Encoder and decoder | Read one sentence and write another | seq2seq, transformer |
| Gated memory | Remember early words in a long sentence | LSTM, GRU, Mamba |
| Attention | Look at any part of the input and decide what's important | Attention, self-attention, transformer |
And then the author immediately returns to the cat to show that each model saw it in its own way. For the fully connected network, it was a list of 196,608 numbers. The convolutional network saw a map of edges and called it a tabby. For vector representations, the word “cat” was one row of a table, row 3797. And GPT-2 read “the cat sat on” and added its own ending: “the cat sat on the floor, and the cat was still sleeping.”
Sixteen names are six ideas. If you keep six ideas in mind, not sixteen names, the whole story fits into one line: each model appears because the previous one hit a wall, and each wall is named with a number. This is much more useful than a list of dates.
The author's last remark is also worth noting: none of these models can answer a question about a photo, draw a new cat, or hear a sentence aloud. This is the next material, about working with multiple types of data at once.
This material is not about our product, and it would be wrong to pretend that it is. But three things from it are directly applicable to what we are building, and one thing explains our own risk.
This is shown three times here. A single neuron on XOR gets stuck at 0.69, which is exactly the error of a coin toss. A recurrent network loses the beginning of a sentence, and this is seen as 1e-13 versus 0.08 for attention. A mixture of experts is measured not by the number of weights, but by the number of weights each word pays for. The general lesson: one should name not the average result, but the difference between two states. This is exactly the rule we have already adopted for recording weight adjustments: count failures before and after, not the average score.
A single neuron will not learn XOR, no matter how much you train it: it does not have the form for it. A fully connected network will not transfer what it learned about a cat to a shifted photograph, because each input has its own weight. A recurrent network does not use the beginning of a paragraph because the gradient does not reach there. In each case, the problem is in the architecture, not in the volume. For us, this is an argument in favor of recording not only the result, but also what form the system had when it produced it.
BERT cannot write, which is why GPT appeared. Attention does not know the order of words, which is why positions are added to it. Mamba cannot look at one specific early word, which is why it is mixed with attention. No single model does everything, and maturity is about naming the blind spot first, not last.
We are building a record and verification, not a model. This means that we depend on other people's models and their blind spots. The material shows that blind spots are the norm, not a defect. This means that our record must be able to describe not only “what came out,” but also “what shape the system was”: model version, weight footprint, configuration. Otherwise, our receipt will honestly sign off on a result about which it is impossible to say under what conditions it was obtained.
This material holds attention for an hour because it has an order: wall, answer, number. We apply the same order in our own analyses, and it also works in conversation with a client. Not “here's our technology,” but “here's the wall, here's the answer, here's the number by which the wall is measured.”
The material promises sixteen architectures. Below, all of them are listed, with the year and the section of this analysis where each is mentioned. This is a check for omission: if even one were missing, the count would not add up.
| # | Architecture | Year | Section |
|---|---|---|---|
| 1 | Perceptron | 1958 | 2 |
| 2 | Multilayer Perceptron | 1986 | 4 |
| 3 | Convolutional Network | 1998 | 7 |
| 4 | Residual Network | 2015 | 10 |
| 5 | Word Embeddings | 2013 | 12 |
| 6 | Recurrent network | 1990 | 13 |
| 7 | Long Short-Term Memory | 1997 | 15 |
| 8 | Gated Recurrent Unit | 2014 | 16 |
| 9 | Sequence to Sequence | 2014 | 18 |
| 10 | Attention | 2014 | 19 |
| 11 | Self-Attention | 2014-2017 | 20 |
| 12 | Transformer | 2017 | 22 |
| 13 | BERT | 2018 | 23 |
| 14 | GPT | 2018 | 24 |
| 15 | Mixture of Experts | 1991, 2017, 2023 | 26 |
| 16 | Mamba | 2023 | 27 |
Sixteen out of sixteen. I'll note separately: self-attention in the source material is not given a separate number; it's part of the discussion about attention and the transformer. Here, it's moved to a separate section because without it, query, key, and value cannot be explained.
Verified: 16 architectures, 6 questions, each answer found in the text. None were missed.
The transcript was generated automatically and contains recognition errors. They are listed below because otherwise, they could be mistaken for real names and then repeated in conversation. This is not a quibble: half of these words do not exist in their original form.
| In the transcript | Correct | What is this |
|---|---|---|
| Rosenlat, Rosenblat | Rosenblatt | Author of the perceptron, 1958 |
| XR, XR1 | XOR | Exclusive OR |
| Real U, realu, RLU | ReLU | Non-linearity that clips negative values |
| Tan H | tanh | Smooth S-shaped non-linearity |
| Kgevsky | Krizhevsky | First author of AlexNet |
| Kaiming Hei | He Kaiming | Lead author of the ResNet paper |
| Sudskever vinyls and Lei | Sutskever, Vinyals, Le | Authors of seq2seq at Google |
| Baktanau | Bahdanau | Author of the attention paper |
| word tovec, wordtovec | word2vec | Work on word vectors, 2013 |
| Marion | Marian | Open translator |
| Mixrol, Mixtrol | Mixtral | Open model with a mixture of experts |
| Shazir | Shazer | Author of the 2017 work on mixture of experts |
| CHO and others | Cho et al. | Authors of GRU, 2014 |
| PieTorch | PyTorch | Library on which everything is written |
| atom (optimizer) | Adam | Optimizer with a separate step per weight |
| requires decimal grad | requires_grad | Marking numbers that training changes |
| loss dobackward | loss.backward() | Calculates how each weight should move |
| opt.ep | optimizer.step() | Takes a step on the weights |
| 196.68 and 196.000 68 | 196 608 | Numbers in a 256 by 256 photo in three colors |
| BLU | BLEU | Translation quality assessment |
| tappy, Matt | tapis, mat | French "tapis" and English "mat" |
| GPD2 | GPT-2 | 2019 language model |
If you repeat "XR" or "Mixrol" in a conversation with an engineer, it immediately reads as a retelling of someone else's video, not as understanding. All names in this analysis are given in the correct form, and all numbers have been cross-referenced with the transcript text.
Honest section. Below is a list of what in this analysis is taken from the material and not independently verified by us.
This is an analysis of an explanation, not an analysis of primary sources. Everything asserted here about model behavior is asserted based on what is shown in the material. Where the material shows a number, we provide the number. Where it shows a picture, we describe the picture. We do not present this as our own measurements.
Deep learning is not sixteen inventions, but six ideas and six walls, and almost everything that seems new is an old idea applied to a new wall.