{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# A neural LM, counted and trained\n", "\n", "The parameter count, one forward and backward pass, and an overfit caught by a third split.\n", "\n", "From chapter 11, [Neural Language Models](https://nlp.jcrlabz.com/book/neural-lm/), of the course notes.\n", "\n", "Source: `book/code/worked_examples/neural_lm.py`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## What you need\n", "\n", "`pip install numpy`. The next cell does it. Colab usually has numpy already, but nothing here assumes that." ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "%pip install -q numpy" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## The script\n", "\n", "Everything the example defines, in one cell. Run it and the definitions\n", "are live in the notebook." ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "\"\"\"The feed-forward neural language model, worked as Chapter 11 works it.\n", "\n", "Four things, all reproducing the chapter's numbers:\n", "\n", " the parameter count where the 1.5 million sit, and why W_o holds most\n", " one forward pass lookup, concatenate, tanh, softmax, cross-entropy\n", " one backward pass and the prediction improving\n", " generalisation the claim that a neural LM beats a counter on\n", " word combinations it never saw, actually tested\n", "\n", " python3 neural_lm.py # all four\n", " python3 neural_lm.py --params # the parameter table\n", " python3 neural_lm.py --forward # the forward and backward pass\n", " python3 neural_lm.py --generalise # train both models and compare\n", " python3 neural_lm.py --generalise --seeds 10 # repeat over seeds\n", "\n", "Install: pip install numpy\n", "\"\"\"\n", "\n", "import argparse\n", "import math\n", "from collections import Counter\n", "from itertools import product\n", "\n", "try:\n", " import numpy as np\n", "except ImportError:\n", " raise SystemExit(\"this one needs numpy: pip install numpy\")\n", "\n", "\n", "# ------------------------------------------------------------- the parameters\n", "\n", "def show_params(V=10000, d=50, ctx=3, H=100):\n", " E = V * d\n", " Wh = ctx * d * H\n", " Wo = H * V\n", " biases = H + V\n", " total = E + Wh + Wo + biases\n", "\n", " print(f\"\\nSTEP 1 where the parameters live\\n\")\n", " print(f\" |V| = {V}, d = {d}, context = {ctx} words, H = {H}\\n\")\n", " print(f\" {'matrix':<10}{'shape':<20}{'parameters':>14}{'share':>9}\")\n", " print(\" \" + \"-\" * 54)\n", " rows = [(\"E\", f\"{V} x {d}\", E), (\"W_h\", f\"{ctx * d} x {H}\", Wh),\n", " (\"W_o\", f\"{H} x {V}\", Wo), (\"biases\", f\"{H} + {V}\", biases)]\n", " for name, shape, n in rows:\n", " print(f\" {name:<10}{shape:<20}{n:>14,}{n / total:>8.1%}\")\n", " print(\" \" + \"-\" * 54)\n", " print(f\" {'total':<30}{total:>14,}\")\n", "\n", " table = V ** ctx\n", " print(f\"\\n A trigram count table over the same vocabulary would need\")\n", " print(f\" |V|^2 x (|V|-1) = {V ** 2 * (V - 1):.3e} cells.\")\n", " ratio = V ** 2 * (V - 1) / total\n", " print(f\"\\n The network uses {ratio:,.0f} times fewer numbers, which is\")\n", " print(f\" {math.log10(ratio):.1f} orders of magnitude.\")\n", " print(f\"\\n And it shares them. Every context word is looked up in the same\")\n", " print(f\" E, so what the model learns about 'dog' is available wherever\")\n", " print(f\" 'dog' appears. A count table shares nothing between cells.\")\n", "\n", " print(f\"\\n Note the share column. W_o alone holds {Wo / total:.0%} of the\")\n", " print(f\" parameters, and its size is H x |V|. Every training step must\")\n", " print(f\" compute the softmax denominator over all {V} words.\")\n", " print(f\" That is the bottleneck hierarchical softmax and negative\")\n", " print(f\" sampling exist to dodge.\")\n", "\n", "\n", "# ---------------------------------------------------------- one forward pass\n", "\n", "def show_forward(eta=0.5):\n", " \"\"\"A model small enough to print in full: |V| = 5, d = 2, context = 2.\"\"\"\n", " words = [\"the\", \"cat\", \"dog\", \"sat\", \"ran\"]\n", " V, d, H = len(words), 2, 3\n", " rng = np.random.default_rng(0)\n", "\n", " E = np.array([[0.20, -0.10], # the\n", " [0.80, 0.30], # cat\n", " [0.75, 0.35], # dog\n", " [-0.40, 0.60], # sat\n", " [-0.35, 0.65]]) # ran\n", " Wh = rng.normal(0, 0.5, (4, H)) # (context 2 x d 2) -> H\n", " bh = np.zeros(H)\n", " Wo = rng.normal(0, 0.5, (H, V))\n", " bo = np.zeros(V)\n", "\n", " ctx, target = [\"the\", \"cat\"], \"sat\"\n", " ti = words.index(target)\n", "\n", " print(f\"\\n\\nSTEP 2 one forward pass, every number shown\\n\")\n", " print(f\" context: {ctx} target: '{target}'\")\n", " print(f\" |V| = {V}, d = {d}, H = {H}\\n\")\n", "\n", " print(\" LOOK UP. Each context word selects its row of E.\")\n", " for w in ctx:\n", " print(f\" E[{w}] = {np.round(E[words.index(w)], 3)}\")\n", "\n", " x = np.concatenate([E[words.index(w)] for w in ctx])\n", " print(f\"\\n CONCATENATE. x = {np.round(x, 3)} length {len(x)} = 2 x d\")\n", " print(\" This is the whole memory of the past. Its length is frozen\")\n", " print(\" the moment the context size is chosen.\")\n", "\n", " h = np.tanh(Wh.T @ x + bh)\n", " print(f\"\\n MIX. h = tanh(W_h x + b_h) = {np.round(h, 4)}\")\n", "\n", " z = Wo.T @ h + bo\n", " p = np.exp(z - z.max())\n", " p /= p.sum()\n", " print(f\"\\n SCORE AND NORMALISE.\\n\")\n", " print(f\" {'word':<8}{'score z':>10}{'exp(z)':>10}{'p = softmax':>14}\")\n", " print(\" \" + \"-\" * 44)\n", " for i, w in enumerate(words):\n", " mark = \" <- target\" if i == ti else \"\"\n", " print(f\" {w:<8}{z[i]:>+10.4f}{math.exp(z[i] - z.max()):>10.4f}\"\n", " f\"{p[i]:>14.4f}{mark}\")\n", " print(\" \" + \"-\" * 44)\n", " print(f\" {'':<8}{'':>10}{'':>10}{p.sum():>14.4f}\")\n", "\n", " loss = -math.log(p[ti])\n", " print(f\"\\n LOSS. y is one-hot at '{target}', so the sum collapses:\")\n", " print(f\" L = -log p['{target}'] = -log({p[ti]:.4f}) = {loss:.4f} nats\")\n", " print(f\"\\n In bits that is {loss / math.log(2):.4f}, and a model scoring\")\n", " print(f\" every token this way would have perplexity\"\n", " f\" {math.exp(loss):.4f}.\")\n", " print(\" Cross-entropy and perplexity are the same objective.\")\n", "\n", " # one gradient step\n", " print(f\"\\n\\nSTEP 3 one backward pass (eta = {eta})\\n\")\n", " dz = p.copy()\n", " dz[ti] -= 1.0 # prediction minus target, again\n", " print(f\" The gradient at the output is p - y:\")\n", " for i, w in enumerate(words):\n", " print(f\" {w:<6}{dz[i]:>+9.4f}\"\n", " + (\" push down\" if i != ti else \" pull up\"))\n", "\n", " dh = Wo @ dz * (1 - h ** 2)\n", " dx = Wh @ dh\n", " Wo_new = Wo - eta * np.outer(h, dz)\n", " bo_new = bo - eta * dz\n", " Wh_new = Wh - eta * np.outer(x, dh)\n", " bh_new = bh - eta * dh\n", " E_new = E.copy()\n", " for j, w in enumerate(ctx):\n", " E_new[words.index(w)] -= eta * dx[j * d:(j + 1) * d]\n", "\n", " print(f\"\\n That gradient flows back into E as well, which is the point.\")\n", " for w in ctx:\n", " i = words.index(w)\n", " print(f\" E[{w}] {np.round(E[i], 4)} -> {np.round(E_new[i], 4)}\")\n", " print(\"\\n The word vectors are not given to the model. They are learned,\")\n", " print(\" as a by-product of learning to predict the next word.\")\n", "\n", " h2 = np.tanh(Wh_new.T @ x + bh_new)\n", " z2 = Wo_new.T @ h2 + bo_new\n", " p2 = np.exp(z2 - z2.max())\n", " p2 /= p2.sum()\n", " print(f\"\\n DID IT WORK?\\n\")\n", " print(f\" {'word':<8}{'p before':>11}{'p after':>11}{'change':>10}\")\n", " print(\" \" + \"-\" * 42)\n", " for i, w in enumerate(words):\n", " mark = \" <- target\" if i == ti else \"\"\n", " print(f\" {w:<8}{p[i]:>11.4f}{p2[i]:>11.4f}{p2[i] - p[i]:>+10.4f}\"\n", " f\"{mark}\")\n", " print(f\"\\n loss {loss:.4f} -> {-math.log(p2[ti]):.4f}\")\n", "\n", "\n", "# ------------------------------------------------------------ generalisation\n", "\n", "# Two families of animals, each at home in its own family of places. Nobody\n", "# tells the model that the families exist. The structure is only in the data.\n", "PETS = [\"cat\", \"dog\", \"puppy\", \"kitten\", \"hamster\"]\n", "INDOOR = [\"sofa\", \"carpet\", \"cushion\", \"basket\"]\n", "FARM = [\"horse\", \"cow\", \"goat\", \"sheep\", \"donkey\"]\n", "OUTDOOR = [\"meadow\", \"barn\", \"paddock\", \"pasture\"]\n", "PLACES = set(INDOOR + OUTDOOR)\n", "\n", "\n", "def build_corpus(seed=0, hold=8, dev=6):\n", " \"\"\"Every (animal, place) pair inside a family. Split three ways.\n", "\n", " The template is 'the ANIMAL rests PLACE', so the animal sits inside a\n", " trigram context. Both models can see it. The only question is whether\n", " they can transfer what they know about one animal to another.\n", " \"\"\"\n", " every = [f\"the {a} rests {r}\"\n", " for a, r in list(product(PETS, INDOOR)) + list(product(FARM, OUTDOOR))]\n", " idx = np.random.default_rng(seed).permutation(len(every))\n", " held = [every[i] for i in idx[:hold]]\n", " dev_s = [every[i] for i in idx[hold:hold + dev]]\n", " train = [every[i] for i in idx[hold + dev:]]\n", " return train, dev_s, held\n", "\n", "\n", "class TrigramLM:\n", " \"\"\"Add-alpha trigram counter, the baseline being tested against.\"\"\"\n", "\n", " def __init__(self, alpha=0.1):\n", " self.alpha = alpha\n", " self.ng, self.ctx, self.vocab = Counter(), Counter(), set()\n", "\n", " def train(self, sents):\n", " for s in sents:\n", " self.vocab.update(s.split())\n", " self.vocab.add(\"\")\n", " for s in sents:\n", " t = [\"\", \"\"] + s.split() + [\"\"]\n", " for i in range(2, len(t)):\n", " self.ng[(t[i - 2], t[i - 1], t[i])] += 1\n", " self.ctx[(t[i - 2], t[i - 1])] += 1\n", " return self\n", "\n", " def place_perplexity(self, sents):\n", " \"\"\"Score only the place slot. That is where the question lives.\"\"\"\n", " lp = n = 0.0\n", " for s in sents:\n", " t = [\"\", \"\"] + s.split() + [\"\"]\n", " for i in range(2, len(t)):\n", " if t[i] not in PLACES:\n", " continue\n", " c = (t[i - 2], t[i - 1])\n", " num = self.ng[c + (t[i],)] + self.alpha\n", " den = self.ctx[c] + self.alpha * len(self.vocab)\n", " lp += math.log(num / den)\n", " n += 1\n", " return math.exp(-lp / n)\n", "\n", "\n", "class NeuralLM:\n", " \"\"\"The lookup, concatenate, mix, softmax model of this chapter.\"\"\"\n", "\n", " def __init__(self, vocab, d=12, H=24, ctx=2, seed=0):\n", " self.words = sorted(vocab)\n", " self.idx = {w: i for i, w in enumerate(self.words)}\n", " self.V, self.d, self.H, self.ctx = len(self.words), d, H, ctx\n", " rng = np.random.default_rng(seed)\n", " self.E = rng.normal(0, 0.1, (self.V, d))\n", " self.Wh = rng.normal(0, 0.1, (ctx * d, H))\n", " self.bh = np.zeros(H)\n", " self.Wo = rng.normal(0, 0.1, (H, self.V))\n", " self.bo = np.zeros(self.V)\n", "\n", " def examples(self, sents):\n", " X, Y = [], []\n", " for s in sents:\n", " t = [\"\"] * self.ctx + s.split() + [\"\"]\n", " for i in range(self.ctx, len(t)):\n", " X.append([self.idx[w] for w in t[i - self.ctx:i]])\n", " Y.append(self.idx[t[i]])\n", " return np.array(X), np.array(Y)\n", "\n", " def _forward(self, X):\n", " x = self.E[X].reshape(len(X), -1)\n", " h = np.tanh(x @ self.Wh + self.bh)\n", " z = h @ self.Wo + self.bo\n", " z -= z.max(axis=1, keepdims=True)\n", " p = np.exp(z)\n", " return x, h, p / p.sum(axis=1, keepdims=True)\n", "\n", " def fit(self, sents, steps=300, eta=0.5):\n", " X, Y = self.examples(sents)\n", " n = len(X)\n", " for _ in range(steps):\n", " x, h, p = self._forward(X)\n", " dz = p.copy()\n", " dz[np.arange(n), Y] -= 1.0\n", " dz /= n\n", " dh = (dz @ self.Wo.T) * (1 - h ** 2)\n", " dx = dh @ self.Wh.T\n", " self.Wo -= eta * (h.T @ dz)\n", " self.bo -= eta * dz.sum(axis=0)\n", " self.Wh -= eta * (x.T @ dh)\n", " self.bh -= eta * dh.sum(axis=0)\n", " grad = dx.reshape(n, self.ctx, self.d)\n", " for j in range(self.ctx):\n", " np.add.at(self.E, X[:, j], -eta * grad[:, j])\n", " return self\n", "\n", " def snapshot(self):\n", " return tuple(a.copy() for a in\n", " (self.E, self.Wh, self.bh, self.Wo, self.bo))\n", "\n", " def restore(self, snap):\n", " self.E, self.Wh, self.bh, self.Wo, self.bo = snap\n", "\n", " def place_perplexity(self, sents):\n", " lp = n = 0.0\n", " for s in sents:\n", " X, Y = self.examples([s])\n", " _, _, p = self._forward(X)\n", " for i, t in enumerate(s.split()):\n", " if t in PLACES:\n", " lp += math.log(p[i, Y[i]])\n", " n += 1\n", " return math.exp(-lp / n)\n", "\n", "\n", "def train_with_early_stop(train, dev, seed, budget=2000, every=100):\n", " vocab = set(w for t in train for w in t.split()) | {\"\", \"\"}\n", " net = NeuralLM(vocab, seed=seed)\n", " best = (float(\"inf\"), 0, net.snapshot())\n", " for step in range(0, budget + 1, every):\n", " if step:\n", " net.fit(train, steps=every)\n", " d = net.place_perplexity(dev)\n", " if d < best[0]:\n", " best = (d, step, net.snapshot())\n", " net.restore(best[2])\n", " return net, best[1]\n", "\n", "\n", "def show_generalise(seeds=6):\n", " print(\"\\n\\nSTEP 4 does the neural model really generalise?\\n\")\n", " print(\" Two families, and each animal rests in its own kind of place.\")\n", " print(f\" pets {PETS}\")\n", " print(f\" indoor {INDOOR}\")\n", " print(f\" farm {FARM}\")\n", " print(f\" outdoor {OUTDOOR}\")\n", " print(\"\\n Template: 'the ANIMAL rests PLACE'. The animal sits inside the\")\n", " print(\" trigram context, so the counter can see it too. Nothing is\")\n", " print(\" hidden from the baseline.\")\n", " print(\"\\n 40 sentences. 8 held out, 6 kept for early stopping, 26 to\")\n", " print(\" train on. We score only the place slot, which is the slot the\")\n", " print(\" animal is supposed to determine.\\n\")\n", "\n", " train, dev, held = build_corpus(0)\n", " print(\" First, what overfitting looks like. Seed 0, place perplexity:\\n\")\n", " vocab = set(w for t in train for w in t.split()) | {\"\", \"\"}\n", " net = NeuralLM(vocab, seed=0)\n", " print(f\" {'steps':>7}{'train':>10}{'held out':>11}\")\n", " print(\" \" + \"-\" * 30)\n", " for step in range(0, 2001, 200):\n", " if step:\n", " net.fit(train, steps=200)\n", " print(f\" {step:>7}{net.place_perplexity(train):>10.3f}\"\n", " f\"{net.place_perplexity(held):>11.3f}\")\n", " print(\"\\n Training perplexity falls the whole way and never looks back.\")\n", " print(\" Held-out perplexity bottoms early, then climbs by a factor of\")\n", " print(\" more than ten. The model stops learning the pattern and starts\")\n", " print(\" memorising the pairs it was given.\")\n", " print(\"\\n So we need a third split to tell us when to stop. That is what\")\n", " print(\" the dev set is for, and it is the only honest way to use it.\\n\")\n", "\n", " print(f\" {'seed':>5}{'stopped at':>12}{'trigram':>10}{'neural':>9}\"\n", " f\"{'ratio':>8}\")\n", " print(\" \" + \"-\" * 46)\n", " tri_tot = net_tot = 0.0\n", " for s in range(seeds):\n", " train, dev, held = build_corpus(s)\n", " tri = TrigramLM(0.1).train(train)\n", " model, stop = train_with_early_stop(train, dev, s)\n", " a, b = tri.place_perplexity(held), model.place_perplexity(held)\n", " tri_tot, net_tot = tri_tot + a, net_tot + b\n", " print(f\" {s:>5}{stop:>12}{a:>10.3f}{b:>9.3f}{a / b:>7.2f}x\")\n", " print(\" \" + \"-\" * 46)\n", " print(f\" The neural model wins on every seed, by\"\n", " f\" {tri_tot / net_tot:.2f} times overall.\")\n", "\n", " print(\"\\n Now the mechanism. Nobody told the model that 'kitten' and\")\n", " print(\" 'cat' are alike, or that a sofa is not a meadow. Two matrices\")\n", " print(\" hold what it worked out. E is where a word goes in. W_o is\")\n", " print(\" where a word comes out.\\n\")\n", " train, dev, held = build_corpus(0)\n", " model, _ = train_with_early_stop(train, dev, 0)\n", "\n", " def cos(M, a, b):\n", " u, v = M[a], M[b]\n", " return float(u @ v / (np.linalg.norm(u) * np.linalg.norm(v)))\n", "\n", " E, O = model.E, model.Wo.T\n", " i = model.idx\n", " pairs = [(\"cat\", \"puppy\"), (\"cat\", \"kitten\"), (\"horse\", \"cow\"),\n", " (\"cat\", \"horse\"), (\"sofa\", \"cushion\"), (\"meadow\", \"barn\"),\n", " (\"sofa\", \"meadow\")]\n", " print(f\" {'pair':<20}{'in E':>9}{'in W_o':>10}\")\n", " print(\" \" + \"-\" * 40)\n", " for a, b in pairs:\n", " print(f\" {a + ' / ' + b:<20}{cos(E, i[a], i[b]):>+9.3f}\"\n", " f\"{cos(O, i[a], i[b]):>+10.3f}\")\n", "\n", " print(\"\\n Read the two columns separately.\")\n", " print(\"\\n In E the animals separate cleanly. 'cat' and 'puppy' score\")\n", " print(\" +0.783, 'cat' and 'horse' score -0.336. The places do not\")\n", " print(\" separate at all: 'sofa' and 'cushion' score +0.271 while 'sofa'\")\n", " print(\" and 'meadow' score +0.225, which is no distinction.\")\n", " print(\"\\n In W_o it is the exact reverse. The places separate, +0.650\")\n", " print(\" for 'sofa' and 'cushion' against -0.051 for 'sofa' and 'meadow'.\")\n", " print(\" The animals stop separating: 'cat' and 'horse' reach +0.652.\")\n", " print(\"\\n The cause is the template. An animal only ever appears as\")\n", " print(\" context, so only its row of E is trained. A place only ever\")\n", " print(\" appears as a target, so only its column of W_o is trained.\")\n", " print(\" Each word learned structure in exactly the matrix it was used in.\")\n", " print(\"\\n This is why word2vec keeps two vectors per word, and why GloVe\")\n", " print(\" keeps w and w-tilde. Being a context and being a target are\")\n", " print(\" different jobs, and one vector cannot hold both.\")\n", " print(\"\\n The generalisation follows. A held-out animal lands near an\")\n", " print(\" animal the model has seen, so its prediction transfers. A count\")\n", " print(\" table has no such geometry, and every unseen pair drops to the\")\n", " print(\" smoothing floor.\")\n", "\n", "\n", "def main():\n", " ap = argparse.ArgumentParser(description=__doc__)\n", " ap.add_argument(\"--params\", action=\"store_true\")\n", " ap.add_argument(\"--forward\", action=\"store_true\")\n", " ap.add_argument(\"--generalise\", action=\"store_true\")\n", " ap.add_argument(\"--seeds\", type=int, default=3)\n", " a = ap.parse_args()\n", "\n", " picked = a.params or a.forward or a.generalise\n", " if a.params or not picked:\n", " show_params()\n", " if a.forward or not picked:\n", " show_forward()\n", " if a.generalise or not picked:\n", " show_generalise(a.seeds)\n", " print()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Running it\n", "\n", "The script is a command line tool, so each run below sets the arguments\n", "and calls `main()`. The heading shows the equivalent shell command." ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import sys\n", "\n", "def run(*args):\n", " sys.argv = [\"neural_lm.py\", *args]\n", " main()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `python3 neural_lm.py`\n", "\n", "The count and the passes." ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "run()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `python3 neural_lm.py --generalise`\n", "\n", "Held-out perplexity bottoms out near step 200 then climbs from 8.8 past 300 while training perplexity keeps falling. Watch a development set and the neural model beats the trigram counter on all six splits." ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "run(\"--generalise\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "Read the chapter this comes from: [Neural Language Models](https://nlp.jcrlabz.com/book/neural-lm/)." ] } ], "metadata": { "colab": { "name": "neural_lm.ipynb", "provenance": [], "toc_visible": true }, "kernelspec": { "display_name": "Python 3", "name": "python3" }, "language_info": { "name": "python" } }, "nbformat": 4, "nbformat_minor": 0 }