{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# word2vec, one step at a time\n",
    "\n",
    "The 0.75 exponent, a single training step worked in full, and the analogy with its trap.\n",
    "\n",
    "From chapter 9, [Learned Word Embeddings](https://nlp.jcrlabz.com/book/embeddings/), of the course notes.\n",
    "\n",
    "Source: `book/code/worked_examples/word2vec.py`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What you need\n",
    "\n",
    "Nothing. This example uses only the Python standard library."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "# Nothing to install: this example uses only the Python standard library.\n",
    "import sys; print(sys.version.split()[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The script\n",
    "\n",
    "Everything the example defines, in one cell. Run it and the definitions\n",
    "are live in the notebook."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "\"\"\"word2vec by hand, as Chapter 9 works it: negative sampling and one update.\n",
    "\n",
    "Three things, all reproducing the book's tables:\n",
    "\n",
    "    the 0.75 exponent      why negatives are drawn from count^0.75\n",
    "    one training step      sigmoid, prediction minus target, update\n",
    "    the analogy            king - man + woman, and the exclusion trap\n",
    "\n",
    "    python3 word2vec.py                  # all three\n",
    "    python3 word2vec.py --exponent 1.0   # sample from raw counts instead\n",
    "    python3 word2vec.py --exponent 0.5   # flatten it further\n",
    "    python3 word2vec.py --no-exclude     # let query words win the analogy\n",
    "    python3 word2vec.py --apple          # apple - iphone, and corpus bias\n",
    "    python3 word2vec.py --subtract mac   # strip a different association\n",
    "\n",
    "Install:  nothing, the Python standard library is enough\n",
    "\"\"\"\n",
    "\n",
    "import argparse\n",
    "import math\n",
    "\n",
    "COUNTS = {\"the\": 1000, \"of\": 600, \"dog\": 50, \"cat\": 40, \"aardvark\": 2}\n",
    "\n",
    "# A second toy space, on axes (technology, fruit, brand), standing in for a\n",
    "# space trained on technology news. 'apple' carries a big technology component\n",
    "# and a smaller fruit one, which is a fact about the corpus, not about English.\n",
    "TECH_SPACE = {\n",
    "    \"apple\":   (0.85, 0.45, 0.80),\n",
    "    \"iphone\":  (0.95, 0.02, 0.85),\n",
    "    \"mac\":     (0.90, 0.03, 0.75),\n",
    "    \"samsung\": (0.88, 0.02, 0.80),\n",
    "    \"android\": (0.92, 0.01, 0.70),\n",
    "    \"orange\":  (0.10, 0.88, 0.25),\n",
    "    \"banana\":  (0.02, 0.95, 0.05),\n",
    "    \"fruit\":   (0.03, 0.96, 0.02),\n",
    "    \"juice\":   (0.06, 0.86, 0.10),\n",
    "}\n",
    "\n",
    "# A toy space with interpretable axes: (royal, male, female).\n",
    "SPACE = {\n",
    "    \"king\":   (1.00, 0.90, 0.10),\n",
    "    \"man\":    (0.10, 0.90, 0.10),\n",
    "    \"woman\":  (0.10, 0.10, 0.90),\n",
    "    \"queen\":  (0.95, 0.15, 0.85),\n",
    "    \"prince\": (0.90, 0.85, 0.15),\n",
    "    \"throne\": (0.80, 0.45, 0.45),\n",
    "    \"child\":  (0.20, 0.50, 0.50),\n",
    "}\n",
    "\n",
    "\n",
    "def sigmoid(z):\n",
    "    return 1 / (1 + math.exp(-z))\n",
    "\n",
    "\n",
    "def dot(a, b):\n",
    "    return sum(x * y for x, y in zip(a, b))\n",
    "\n",
    "\n",
    "def cosine(u, v):\n",
    "    n = math.sqrt(dot(u, u)) * math.sqrt(dot(v, v))\n",
    "    return dot(u, v) / n if n else 0.0\n",
    "\n",
    "\n",
    "# ------------------------------------------------- the sampling distribution\n",
    "\n",
    "def sampling_distribution(counts, exponent):\n",
    "    \"\"\"Equation 8.1. Raising counts to a power below 1 lifts the tail.\"\"\"\n",
    "    total = sum(counts.values())\n",
    "    p = {w: c / total for w, c in counts.items()}\n",
    "    raised = {w: c ** exponent for w, c in counts.items()}\n",
    "    z = sum(raised.values())\n",
    "    q = {w: v / z for w, v in raised.items()}\n",
    "    return p, raised, q\n",
    "\n",
    "\n",
    "def demo_sampling(exponent):\n",
    "    p, raised, q = sampling_distribution(COUNTS, exponent)\n",
    "    print(f\"\\nNEGATIVE SAMPLING DISTRIBUTION   exponent = {exponent}\\n\")\n",
    "    print(f\"  {'word':<10}{'count':>7}{'p(w)':>9}{'count^e':>10}\"\n",
    "          f\"{'q(w)':>9}{'q/p':>9}\")\n",
    "    print(\"  \" + \"-\" * 54)\n",
    "    for w in COUNTS:\n",
    "        print(f\"  {w:<10}{COUNTS[w]:>7}{p[w]:>9.4f}{raised[w]:>10.1f}\"\n",
    "              f\"{q[w]:>9.4f}{q[w]/p[w]:>8.2f}x\")\n",
    "\n",
    "    hi, lo = max(COUNTS, key=COUNTS.get), min(COUNTS, key=COUNTS.get)\n",
    "    print(f\"\\n  '{hi}' is sampled {q[hi]/p[hi]:.2f}x its share,\"\n",
    "          f\" '{lo}' {q[lo]/p[lo]:.2f}x its share.\")\n",
    "    if exponent == 1.0:\n",
    "        print(\"  At exponent 1.0 nothing is bent. Nearly every negative would\")\n",
    "        print(\"  be a stopword, which teaches the model almost nothing.\")\n",
    "    elif exponent < 0.6:\n",
    "        print(\"  Flattened this far, rare words appear as negatives far more\")\n",
    "        print(\"  often than they ever appear as real context words.\")\n",
    "    else:\n",
    "        print(\"  This is the setting word2vec ships with. It trims the head\")\n",
    "        print(\"  and lifts the tail, without going all the way to uniform.\")\n",
    "\n",
    "\n",
    "# ------------------------------------------------------- one training update\n",
    "\n",
    "def demo_update(eta=0.1):\n",
    "    v = (0.5, -0.2)            # centre word, 'cat'\n",
    "    pairs = [(\"sat\", (0.4, 0.1), 1), (\"the\", (-0.3, 0.6), 0)]\n",
    "\n",
    "    print(f\"\\n\\nONE TRAINING STEP   centre 'cat' v = {v},  eta = {eta}\\n\")\n",
    "    print(f\"  {'pair':<22}{'z = v.u':>10}{'sigma(z)':>11}{'label':>7}\"\n",
    "          f\"{'gradient':>11}\")\n",
    "    print(\"  \" + \"-\" * 61)\n",
    "    grads, updated = {}, {}\n",
    "    for name, u, label in pairs:\n",
    "        z = dot(v, u)\n",
    "        s = sigmoid(z)\n",
    "        g = s - label                      # prediction minus target\n",
    "        grads[name] = (u, g, s)\n",
    "        kind = \"positive\" if label else \"negative\"\n",
    "        print(f\"  {'cat, ' + name + ' (' + kind + ')':<22}{z:>+10.3f}\"\n",
    "              f\"{s:>11.4f}{label:>7}{g:>+11.4f}\")\n",
    "\n",
    "    print(\"\\n  A negative gradient pulls the pair together.\")\n",
    "    print(\"  A positive gradient pushes it apart.\")\n",
    "\n",
    "    print(f\"\\n  {'vector':<8}{'update':<40}{'new value'}\")\n",
    "    print(\"  \" + \"-\" * 72)\n",
    "    for name, (u, g, _) in grads.items():\n",
    "        new = tuple(round(x - eta * g * y, 4) for x, y in zip(u, v))\n",
    "        updated[name] = new\n",
    "        arith = f\"{u} - {eta}({g:+.4f}){v}\"\n",
    "        print(f\"  {name:<8}{arith:<40}{new}\")\n",
    "\n",
    "    # the centre word collects the gradient from every pair\n",
    "    v_new = tuple(v[i] - eta * sum(g * u[i] for u, g, _ in grads.values())\n",
    "                  for i in range(len(v)))\n",
    "\n",
    "    print(f\"\\n  DID IT WORK?   centre vector also moved to\"\n",
    "          f\" {tuple(round(x,4) for x in v_new)}\\n\")\n",
    "    print(f\"  {'score':<28}{'before':>10}{'after':>10}   want\")\n",
    "    print(\"  \" + \"-\" * 60)\n",
    "    for name, (u, _, before) in grads.items():\n",
    "        after = sigmoid(dot(v_new, updated[name]))\n",
    "        want = \"up\" if name == \"sat\" else \"down\"\n",
    "        print(f\"  {'sigma(cat . ' + name + ')':<28}{before:>10.4f}\"\n",
    "              f\"{after:>10.4f}   {want}\")\n",
    "\n",
    "\n",
    "# ------------------------------------------------------------- the analogy\n",
    "\n",
    "def demo_analogy(exclude=True):\n",
    "    a, b, c = \"man\", \"king\", \"woman\"          # analogy(a, b, c) = v_b - v_a + v_c\n",
    "    target = tuple(SPACE[b][i] - SPACE[a][i] + SPACE[c][i] for i in range(3))\n",
    "\n",
    "    print(f\"\\n\\nANALOGY   v_{b} - v_{a} + v_{c}\"\n",
    "          f\" = ({target[0]:.2f}, {target[1]:.2f}, {target[2]:.2f})\\n\")\n",
    "    print(f\"  {'word':<9}{'royal':>7}{'male':>7}{'female':>8}{'cosine':>10}\")\n",
    "    print(\"  \" + \"-\" * 46)\n",
    "    ranked = sorted(((w, cosine(target, v)) for w, v in SPACE.items()),\n",
    "                    key=lambda p: -p[1])\n",
    "    for w, s in ranked:\n",
    "        v = SPACE[w]\n",
    "        note = \"   <- query word\" if w in (a, b, c) else \"\"\n",
    "        print(f\"  {w:<9}{v[0]:>7.2f}{v[1]:>7.2f}{v[2]:>8.2f}{s:>10.4f}{note}\")\n",
    "\n",
    "    pool = [w for w, _ in ranked if not exclude or w not in (a, b, c)]\n",
    "    print(f\"\\n  answer: {pool[0]}\"\n",
    "          f\"   ({'query words excluded' if exclude else 'NO EXCLUSION'})\")\n",
    "    if exclude:\n",
    "        cheat = ranked[0][0]\n",
    "        if cheat in (a, b, c):\n",
    "            print(f\"  Without the exclusion '{cheat}' would have won, and the\")\n",
    "            print(\"  analogy would look correct while proving nothing.\")\n",
    "        else:\n",
    "            order = [w for w, _ in ranked]\n",
    "            best_q = min((w for w in (a, b, c) if w in order),\n",
    "                         key=order.index)\n",
    "            rank = order.index(best_q) + 1\n",
    "            print(f\"  Here the best query word, '{best_q}', only reaches rank\"\n",
    "                  f\" {rank}. On real\")\n",
    "            print(\"  spaces it usually places first, because the target stays\")\n",
    "            print(\"  close to v_c. Forget the exclusion and your scores inflate.\")\n",
    "\n",
    "\n",
    "def demo_apple(strip=\"iphone\", target=\"apple\"):\n",
    "    \"\"\"Subtract one word from another and see which sense survives.\"\"\"\n",
    "    def rank(vec, exclude=()):\n",
    "        return sorted(((w, cosine(vec, v)) for w, v in TECH_SPACE.items()\n",
    "                       if w not in exclude), key=lambda p: -p[1])\n",
    "\n",
    "    print(f\"\\n\\nNEIGHBOURS OF '{target}'   (space trained on technology news)\\n\")\n",
    "    print(f\"  {'neighbour':<10}{'cosine':>9}\")\n",
    "    print(\"  \" + \"-\" * 21)\n",
    "    for w, c in rank(TECH_SPACE[target], exclude=(target,))[:6]:\n",
    "        print(f\"  {w:<10}{c:>+9.4f}\")\n",
    "    print(f\"\\n  The top of that list is technology. '{target}' is a company\")\n",
    "    print(\"  in this corpus, and the other sense is buried underneath.\")\n",
    "\n",
    "    d = tuple(TECH_SPACE[target][i] - TECH_SPACE[strip][i]\n",
    "              for i in range(len(TECH_SPACE[target])))\n",
    "    print(f\"\\n  vec({target}) - vec({strip}) = \"\n",
    "          f\"({d[0]:+.2f}, {d[1]:+.2f}, {d[2]:+.2f})\\n\")\n",
    "    print(f\"  {'neighbour':<10}{'cosine':>9}\")\n",
    "    print(\"  \" + \"-\" * 21)\n",
    "    for w, c in rank(d, exclude=(target, strip))[:6]:\n",
    "        print(f\"  {w:<10}{c:>+9.4f}\")\n",
    "\n",
    "    print(f\"\\n  Subtracting cancels what the two words share and keeps what\")\n",
    "    print(f\"  only '{target}' has. The technology component nearly annihilates.\")\n",
    "    print(\"  The buried sense comes to the surface, and the words that were\")\n",
    "    print(\"  nearest before now point the other way.\")\n",
    "\n",
    "\n",
    "def main():\n",
    "    ap = argparse.ArgumentParser(description=__doc__)\n",
    "    ap.add_argument(\"--exponent\", type=float, default=0.75)\n",
    "    ap.add_argument(\"--no-exclude\", action=\"store_true\",\n",
    "                    help=\"let the query words compete in the analogy\")\n",
    "    ap.add_argument(\"--apple\", action=\"store_true\",\n",
    "                    help=\"only the apple minus iphone demonstration\")\n",
    "    ap.add_argument(\"--subtract\", default=\"iphone\",\n",
    "                    help=\"which word to strip from 'apple'\")\n",
    "    a = ap.parse_args()\n",
    "    if a.apple or a.subtract != \"iphone\":\n",
    "        demo_apple(strip=a.subtract)\n",
    "        print()\n",
    "        return\n",
    "    demo_sampling(a.exponent)\n",
    "    demo_update()\n",
    "    demo_analogy(exclude=not a.no_exclude)\n",
    "    demo_apple()\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running it\n",
    "\n",
    "The script is a command line tool, so each run below sets the arguments\n",
    "and calls `main()`. The heading shows the equivalent shell command."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "import sys\n",
    "\n",
    "def run(*args):\n",
    "    sys.argv = [\"word2vec.py\", *args]\n",
    "    main()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `python3 word2vec.py`\n",
    "\n",
    "The sampling table, one update, and the analogy."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "run()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `python3 word2vec.py --exponent 1.0`\n",
    "\n",
    "Nothing is bent, so almost every negative would be a stopword. At 0.75, aardvark is sampled 4.24 times its share while the is trimmed to 0.90."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "run(\"--exponent\", \"1.0\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `python3 word2vec.py --apple`\n",
    "\n",
    "In a space trained on technology news the neighbours of apple are mac, samsung, iphone. Subtract iphone and fruit surfaces. That is a fact about the corpus, not about English."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "run(\"--apple\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `python3 word2vec.py --no-exclude`\n",
    "\n",
    "Let the query words compete. woman reaches rank 3, which is the most common way to overstate an embedding."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "run(\"--no-exclude\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "Read the chapter this comes from: [Learned Word Embeddings](https://nlp.jcrlabz.com/book/embeddings/)."
   ]
  }
 ],
 "metadata": {
  "colab": {
   "name": "word2vec.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}