A Neural Language Model.
Give every character a vector, let a small network turn the context into a distribution, and train it: similar contexts now get similar predictions.
A neural language model does the same job as a count table — context in, next-token distribution out — but computes the answer instead of looking it up. Each token gets a learned vector, an embedding; a network combines the vectors of the context; a final layer turns the result into a score, a logit, for every token in the vocabulary, and a softmax turns the logits into probabilities. All of it is adjusted by gradient descent to lower the cross-entropy on the training text (Vol. III shows how).
The workshop does two things. On the left it trains a small model live, in your browser, for a few seconds, so you can watch the loss fall. On the right it compares recorded runs made on a GPU: counting models and neural models built from exactly the same 29 million characters of English and scored on exactly the same held-out text, placed by how many numbers each has to store.
Below, left: a small model trained in your browser for a few seconds — training loss step by step and held-out loss at intervals. Right: recorded runs on the same 29 million training characters and the same held-out text: counting models of every order from 1 to 7, and three neural models of increasing size, placed by how many numbers each stores.
- Describe the parts of a neural language model: embeddings, a network, logits, softmax.
- Train a small model and read its training and held-out loss.
- Compare neural and counting models on the same text by loss and by size.
- Explain why a neural model can generalise to unseen contexts.
Learning instead of looking up
In your browser, a model of 17,536 parameters trained for 150 steps brings its loss from 4.850 nats per character — an even guess over the alphabet — down to 3.363 on held-out text: it has learned which letters are common and which tend to follow which. That is far from good; a laptop processor gives it a few seconds. The recorded runs make the real comparison. The best counting model, a 7-gram with interpolated smoothing, reaches 1.869 bits per character on the held-out text, but its table holds 7.09 million counts. The neural model char-s already beats it, at 1.833 bits, with 824 thousand parameters — about 9 times fewer numbers — and char-m, with 4.79 million, reaches 1.682. Even nano, with 113 thousand parameters, lands at 2.153 bits, between the 4-gram and the 5-gram, whose tables are 4 and 13 times larger.
The difference is generalisation. A count table stores every context it saw separately and knows nothing about the ones it did not; a network stores a way of reading contexts — embeddings that place similar characters and words near each other, and layers that combine them — so what it learned from one context transfers to similar ones. That is also why it uses so few numbers. Nothing in the recipe changes as models grow a million times larger: embeddings, a network, logits, a softmax, a cross-entropy loss and gradient descent, which Vol. III takes apart piece by piece.
Terms
- Embedding
- A learned vector for each token; similar tokens end up with similar vectors.
- Logit
- An unnormalised score for each token; the softmax turns logits into probabilities.
- Parameters
- The numbers a network learns: embeddings and weights. Their count measures the model’s size.
- Gradient descent
- Repeatedly nudging every parameter in the direction that lowers the loss.