Every chatbot you've used is built on a handful of ideas you can learn in a weekend. Start at "what even is a word" and finish at "I know why attention works." No black boxes.
Do them in order and nothing arrives out of nowhere — by the time you hit transformers, every piece of the diagram is something you already built.
Before any model, look at the raw material. A handful of words do most of the work in any language, vocabulary grows in a shape you can predict, and "how do I even cut this string into pieces" turns out to be a real research problem.
Models can't read letters, only numbers. The trick that changed everything: a word's meaning is the company it keeps. Count what appears near what, squeeze it down, and you get vectors where directions mean things — including the ugly ones.
One neuron is a line that splits a plane. Stack them and you can carve any shape. Then the only real question left: how does a pile of numbers know which way to move? That's gradient descent, and backprop is just the chain rule with good bookkeeping.
This is the whole job. Count word pairs and you have a language model — a bad one, but a real one. Everything after this is a better way to answer the same question, with memory that survives longer than three words.
Instead of reading left to right and hoping you remember, let every word look at every other word at once and decide what matters. That's it. That's the idea the entire modern field is standing on.
The part everyone skips to — except now you'll know why any of it works. Why temperature changes the vibe, why LoRA is cheap, why RAG beats fine-tuning for facts, and why BLEU scores are argued about.
Models learn the patterns in their training text — all of them, including the ones nobody wanted. This unit is about finding that bias with the same maths you used to build the thing, and knowing what you can actually do about it.
293 pages that go where the slides can't: the derivations, the worked examples, and eight projects you actually build. Reads in the browser — no download, no app.
What Natural Language Processing Is · Corpora and Preprocessing · The Empirical Laws of Text · Term Weighting and Similarity · Subword Tokenization
Vector Space Models · Count Vectors, PPMI, and SVD · Neural Network Fundamentals · Learned Word Embeddings
n-gram Language Models and Perplexity · Neural Language Models · Recurrent Networks · Gated Recurrence: LSTM and GRU · Text Classification and Evaluation
Contextual Embeddings · Self-Attention and the Transformer · Pretrained Transformers: BERT and GPT
Steering LLMs: Decoding and Prompting · Adaptation and Alignment · Retrieval-Augmented Generation · Evaluating Generated Text · Building with Language Models · Efficiency: Smaller, Cheaper, Faster
Machine Translation · Summarization
Responsible NLP
About the projects · Project 1 — The Shape of Language · Project 2 — Build the Tokenizer · Project 3 — Embedding Forensics · Project 4 — Language Models · Project 5 — Classification Leaderboard · Project 6 — Transformer Anatomy · Project 7 — Steering LLMs · Project 8 — RAG Capstone
n-gram Language Model · Scaled Dot-Product Attention · Embedding Geometry · Decoding Mechanics · BM25 Retrieval · Bibliography
All of this runs in your browser, on your text, with no server and no model download. Drag the sliders until something breaks — that's the point.
Models don't see words — they see tokens. Byte-pair encoding starts from single characters and repeatedly glues together whichever pair shows up most often. Drag the slider to run more merges and watch letters fuse into chunks that look suspiciously like morphemes. This is really training BPE, live, on the text in the box.
Rank every word by how often it appears. The 2nd-most-common word shows up about half as much as the 1st, the 3rd about a third as much, and so on — in every language anyone has checked. On a log–log plot that curve becomes a straight line. Paste in song lyrics, an essay, your group chat: you'll get the same line.
If words are points in space, then relationships are arrows, and the
same arrow works everywhere: the step from man to woman is the step
from king to queen. Do the arithmetic, look for the nearest word,
and analogies fall out of pure vector maths.
Count which words follow which. To generate, look at the last word (or two) and sample from what came next in the training text. That's an n-gram model — the direct ancestor of every LLM, and it makes the same trade-offs. Watch the probability bars on the right update at every step.
Every one of these gets a proper treatment somewhere in the decks. This is just so you're never lost in a thread.
The chunk a model actually sees. Usually not a whole word — "unhappiness" might arrive as un + happi + ness. Everything is priced and limited in tokens.
A list of numbers standing in for a word, sentence or document, arranged so similar meanings sit near each other. Usually a few hundred numbers long.
A pile of text you're learning from. Plural: corpora. The single biggest lever on how a model behaves, and the least glamorous part of the job.
Turns any list of scores into probabilities that add to 1, with the biggest score getting the biggest share. It's how a model converts "I like this word 8.2" into "42% chance".
How surprised a language model is by real text. Lower is better. Perplexity 20 means it was about as unsure as picking uniformly between 20 options at each word.
Each token looks at all the others and computes a weighted average of them, weighted by relevance. Repeat a few dozen times, and that's a transformer.
The architecture underneath basically everything since 2017. Attention plus small feedforward layers, stacked, with no recurrence — which is why it trains fast on GPUs.
A knob on randomness when generating. Low = repetitive and safe. High = creative then incoherent. It literally divides the scores before softmax.
Taking a trained model and nudging its weights on your own data. Expensive at full size, which is why LoRA exists.
Freeze the big model, train two tiny matrices alongside it. You get most of the benefit of fine-tuning for a fraction of the memory. The "PEFT" family.
Retrieval-augmented generation: search your documents first, paste the hits into the prompt, then answer. How you give a model facts it was never trained on.
Humans rank model outputs, a reward model learns those preferences, then the model is tuned to score well. It's what makes a raw LLM feel like an assistant.
An old translation metric that counts overlapping word sequences with a reference translation. Widely used, widely complained about, still the default.
Word frequency is inversely proportional to rank. A handful of words are everywhere and most words are almost nowhere. See the lab.
Fluent, confident, wrong. Not a bug bolted on — a direct consequence of a model whose only training objective was "sound like the next word".
How many tokens the model can look at in one go. Anything outside it doesn't exist, no matter how important you thought it was.
Lists, loops, dictionaries, functions. If you can write a word counter, you're fine. You'll pick up NumPy along the way.
Vectors, dot products, a bit of probability. Derivatives help by unit 03 but the slides build them up rather than assuming them.
Everything through unit 04 runs on a laptop. Colab's free tier covers the rest if you want to train something.
Read the unit, then come back here and break the matching demo. Units 01, 02 and 04 each have one waiting.
Fifteen pages. Twenty minutes. You'll know what the field is actually asking.