{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# One request, three answers\n",
    "\n",
    "Show me the common words in this file. Nothing in that sentence says what a word is, or what common means. Three faithful readings, three different answers.\n",
    "\n",
    "From chapter 1, [What Natural Language Processing Is](https://nlp.jcrlabz.com/book/whatisnlp/), of the course notes.\n",
    "\n",
    "Source: `book/code/worked_examples/bridging_the_gap.py`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What you need\n",
    "\n",
    "Nothing. This example uses only the Python standard library."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "# Nothing to install: this example uses only the Python standard library.\n",
    "import sys; print(sys.version.split()[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The script\n",
    "\n",
    "Everything the example defines, in one cell. Run it and the definitions\n",
    "are live in the notebook."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "\"\"\"Bridging the gap: one English sentence, three defensible programs.\n",
    "\n",
    "Backs the \"Bridging the Gap\" frames in Latex/FrameLibrary/L1L2Review.tex.\n",
    "The request is \"Show me the common words in this file\". Nothing in that\n",
    "sentence fixes what a word is, or what common means. Each reading below is a\n",
    "faithful implementation of the same sentence, and each returns a different\n",
    "answer.\n",
    "\n",
    "Run: python3 bridging_the_gap.py\n",
    "\n",
    "Install:  nothing, the Python standard library is enough\n",
    "\"\"\"\n",
    "\n",
    "import re\n",
    "from collections import Counter\n",
    "\n",
    "TEXT = \"The cat sat on the mat. The Cat did not sit on the dog.\"\n",
    "\n",
    "STOPWORDS = {\"the\", \"on\", \"did\", \"not\", \"a\", \"an\", \"and\", \"of\", \"to\", \"is\"}\n",
    "\n",
    "\n",
    "def reading_a(text):\n",
    "    \"\"\"Word means whatever whitespace separates. Common means most frequent.\"\"\"\n",
    "    return Counter(text.split())\n",
    "\n",
    "\n",
    "def reading_b(text):\n",
    "    \"\"\"Word means letters only, case folded. Common means most frequent.\"\"\"\n",
    "    return Counter(re.findall(r\"[a-z]+\", text.lower()))\n",
    "\n",
    "\n",
    "def reading_c(text):\n",
    "    \"\"\"Reading B, minus the words that carry no topic.\"\"\"\n",
    "    counts = reading_b(text)\n",
    "    return Counter({w: c for w, c in counts.items() if w not in STOPWORDS})\n",
    "\n",
    "\n",
    "def top(counts, k=3):\n",
    "    \"\"\"Rank by count, break ties alphabetically so the answer is repeatable.\"\"\"\n",
    "    return sorted(counts.items(), key=lambda kv: (-kv[1], kv[0]))[:k]\n",
    "\n",
    "\n",
    "def rank_of(counts, word):\n",
    "    order = sorted(counts.items(), key=lambda kv: (-kv[1], kv[0]))\n",
    "    for i, (w, _) in enumerate(order, start=1):\n",
    "        if w == word:\n",
    "            return i\n",
    "    return None\n",
    "\n",
    "\n",
    "def show(name, counts):\n",
    "    print(f\"\\n{name}\")\n",
    "    print(f\"  token count : {sum(counts.values())}\")\n",
    "    print(f\"  vocabulary  : {len(counts)}\")\n",
    "    print(f\"  top 3       : {top(counts)}\")\n",
    "    print(f\"  full        : {top(counts, k=len(counts))}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `python3 bridging_the_gap.py`\n",
    "\n",
    "All three readings of the same request, and the counts each one returns."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "print(f\"input: {TEXT}\")\n",
    "\n",
    "a = reading_a(TEXT)\n",
    "b = reading_b(TEXT)\n",
    "c = reading_c(TEXT)\n",
    "\n",
    "show(\"Reading A  split on spaces, keep case, keep punctuation\", a)\n",
    "show(\"Reading B  lower case, letters only\", b)\n",
    "show(\"Reading C  reading B without stop words\", c)\n",
    "\n",
    "print(\"\\nWhere does 'cat' land?\")\n",
    "for name, counts in ((\"A\", a), (\"B\", b), (\"C\", c)):\n",
    "    hits = counts.get(\"cat\", 0)\n",
    "    print(f\"  reading {name}: count {hits}, rank {rank_of(counts, 'cat')}\")\n",
    "\n",
    "print(\"\\nTokens produced by reading A:\")\n",
    "print(\" \", TEXT.split())\n",
    "print(\"Tokens produced by reading B:\")\n",
    "print(\" \", re.findall(r\"[a-z]+\", TEXT.lower()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "Read the chapter this comes from: [What Natural Language Processing Is](https://nlp.jcrlabz.com/book/whatisnlp/)."
   ]
  }
 ],
 "metadata": {
  "colab": {
   "name": "bridging_the_gap.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}