{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Accuracy, F1 and a confusion matrix\n",
    "\n",
    "The trap accuracy walks into, and how the assignment wants a confusion matrix read.\n",
    "\n",
    "From chapter 14, [Text Classification and Evaluation](https://nlp.jcrlabz.com/book/classification/), of the course notes.\n",
    "\n",
    "Source: `book/code/worked_examples/classification.py`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What you need\n",
    "\n",
    "Nothing. This example uses only the Python standard library."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "# Nothing to install: this example uses only the Python standard library.\n",
    "import sys; print(sys.version.split()[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The script\n",
    "\n",
    "Everything the example defines, in one cell. Run it and the definitions\n",
    "are live in the notebook."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "\"\"\"Evaluating a classifier, worked as Chapter 14 works it.\n",
    "\n",
    "Four things, all reproducing the chapter's tables:\n",
    "\n",
    "    the accuracy trap    a useless model with 90 per cent accuracy\n",
    "    the confusion matrix per-class precision, recall and F1, computed\n",
    "    reading the matrix   hardest class and most-confused pair, derived\n",
    "    macro against micro  why they differ, and when they cannot\n",
    "\n",
    "    python3 classification.py            # all four\n",
    "    python3 classification.py --trap     # the accuracy trap\n",
    "    python3 classification.py --matrix   # the full per-class table\n",
    "    python3 classification.py --read     # hardest class, confused pairs\n",
    "    python3 classification.py --averages # macro, micro and accuracy\n",
    "\n",
    "Install:  nothing, the Python standard library is enough\n",
    "\"\"\"\n",
    "\n",
    "import argparse\n",
    "\n",
    "# The genre confusion matrix from the lectures. Rows are the true class,\n",
    "# columns the predicted class. Twelve documents of each genre.\n",
    "GENRES = [\"news\", \"fiction\", \"government\", \"learned\"]\n",
    "MATRIX = [\n",
    "    [9, 0, 2, 1],     # news\n",
    "    [0, 11, 0, 1],    # fiction\n",
    "    [3, 0, 7, 2],     # government\n",
    "    [1, 1, 1, 9],     # learned\n",
    "]\n",
    "\n",
    "\n",
    "def counts(M, i):\n",
    "    \"\"\"TP, FP, FN for class i, treating the task as one against the rest.\"\"\"\n",
    "    tp = M[i][i]\n",
    "    fp = sum(M[r][i] for r in range(len(M))) - tp\n",
    "    fn = sum(M[i]) - tp\n",
    "    return tp, fp, fn\n",
    "\n",
    "\n",
    "def prf(tp, fp, fn):\n",
    "    p = tp / (tp + fp) if tp + fp else 0.0\n",
    "    r = tp / (tp + fn) if tp + fn else 0.0\n",
    "    f = 2 * p * r / (p + r) if p + r else 0.0\n",
    "    return p, r, f\n",
    "\n",
    "\n",
    "def macro_f1(M):\n",
    "    return sum(prf(*counts(M, i))[2] for i in range(len(M))) / len(M)\n",
    "\n",
    "\n",
    "def accuracy(M):\n",
    "    return sum(M[i][i] for i in range(len(M))) / sum(sum(r) for r in M)\n",
    "\n",
    "\n",
    "# ------------------------------------------------------------ the trap\n",
    "\n",
    "def show_trap():\n",
    "    \"\"\"Imbalanced classes, and a model that has learned nothing.\"\"\"\n",
    "    support = [90, 5, 3, 2]\n",
    "    labels = [\"news\", \"fiction\", \"government\", \"learned\"]\n",
    "\n",
    "    # Everything predicted as the majority class.\n",
    "    M = [[0] * 4 for _ in range(4)]\n",
    "    for i, n in enumerate(support):\n",
    "        M[i][0] = n\n",
    "\n",
    "    print(\"\\nSTEP 1  why accuracy is not enough\\n\")\n",
    "    print(\"  A hundred test documents, unevenly distributed:\")\n",
    "    print(\"    \" + \",  \".join(f\"{l} {n}\" for l, n in zip(labels, support)))\n",
    "    print(\"\\n  The model is a single line of code: always answer 'news'.\\n\")\n",
    "    print(f\"  {'true class':<12}\" + \"\".join(f\"{l[:7]:>10}\" for l in labels))\n",
    "    print(\"  \" + \"-\" * 52)\n",
    "    for i, l in enumerate(labels):\n",
    "        print(f\"  {l:<12}\" + \"\".join(f\"{v:>10}\" for v in M[i]))\n",
    "\n",
    "    print(f\"\\n  {'class':<12}{'P':>9}{'R':>9}{'F1':>9}\")\n",
    "    print(\"  \" + \"-\" * 40)\n",
    "    for i, l in enumerate(labels):\n",
    "        p, r, f = prf(*counts(M, i))\n",
    "        print(f\"  {l:<12}{p:>9.4f}{r:>9.4f}{f:>9.4f}\")\n",
    "\n",
    "    print(f\"\\n  accuracy  {accuracy(M):.4f}\")\n",
    "    print(f\"  macro-F1  {macro_f1(M):.4f}\")\n",
    "    print(\"\\n  Ninety per cent accuracy, and the model cannot tell any two\")\n",
    "    print(\"  documents apart. Three of the four classes score exactly zero.\")\n",
    "    print(\"\\n  Macro-F1 sees it immediately, because it averages the per-class\")\n",
    "    print(\"  scores and a class the model ignores contributes 0 to that mean.\")\n",
    "    print(\"\\n  This is why Assignment 5's leaderboard uses macro-F1. You cannot\")\n",
    "    print(\"  win it by chasing the majority class.\")\n",
    "\n",
    "\n",
    "# -------------------------------------------------------- the real matrix\n",
    "\n",
    "def show_matrix():\n",
    "    print(\"\\n\\nSTEP 2  a real confusion matrix, scored\\n\")\n",
    "    print(\"  Rows are the true class, columns the prediction.\")\n",
    "    print(\"  The diagonal is correct. Everything else is an error.\\n\")\n",
    "    print(f\"  {'true / pred':<13}\" + \"\".join(f\"{g[:7]:>10}\" for g in GENRES)\n",
    "          + f\"{'total':>9}\")\n",
    "    print(\"  \" + \"-\" * 62)\n",
    "    for i, g in enumerate(GENRES):\n",
    "        cells = \"\".join(f\"{v:>10}\" for v in MATRIX[i])\n",
    "        print(f\"  {g:<13}{cells}{sum(MATRIX[i]):>9}\")\n",
    "\n",
    "    print(f\"\\n  Now one row per class, treating each as one against the rest.\\n\")\n",
    "    print(f\"  {'class':<12}{'TP':>5}{'FP':>5}{'FN':>5}{'P':>9}{'R':>9}\"\n",
    "          f\"{'F1':>9}\")\n",
    "    print(\"  \" + \"-\" * 56)\n",
    "    for i, g in enumerate(GENRES):\n",
    "        tp, fp, fn = counts(MATRIX, i)\n",
    "        p, r, f = prf(tp, fp, fn)\n",
    "        print(f\"  {g:<12}{tp:>5}{fp:>5}{fn:>5}{p:>9.4f}{r:>9.4f}{f:>9.4f}\")\n",
    "\n",
    "    print(f\"\\n  accuracy  {accuracy(MATRIX):.4f}\")\n",
    "    print(f\"  macro-F1  {macro_f1(MATRIX):.4f}\")\n",
    "    print(\"\\n  Here the classes are balanced at twelve each, so accuracy is\")\n",
    "    print(\"  not misleading. It is still less informative: it says 0.75 and\")\n",
    "    print(\"  stops, where the per-class table says which genre is failing.\")\n",
    "\n",
    "    print(\"\\n  Look at 'government'. Its recall is 0.5833, the lowest in the\")\n",
    "    print(\"  table, because five of its twelve documents went elsewhere.\")\n",
    "    print(\"  Precision is 0.7000, because three documents from other genres\")\n",
    "    print(\"  were labelled government. Both directions are wrong at once.\")\n",
    "\n",
    "\n",
    "# ------------------------------------------------------- reading the matrix\n",
    "\n",
    "def show_read():\n",
    "    print(\"\\n\\nSTEP 3  two diagnoses the matrix hands you\\n\")\n",
    "    f1 = [(prf(*counts(MATRIX, i))[2], GENRES[i]) for i in range(len(GENRES))]\n",
    "    f1.sort()\n",
    "    print(\"  HARDEST CLASS is the lowest per-class F1.\\n\")\n",
    "    for f, g in f1:\n",
    "        print(f\"    {g:<12}{f:>9.4f}\")\n",
    "    print(f\"\\n    hardest: {f1[0][1]}\")\n",
    "\n",
    "    print(\"\\n  MOST-CONFUSED PAIR is the unordered pair with the most cross\")\n",
    "    print(\"  errors. Add the two off-diagonal cells that join them.\\n\")\n",
    "    pairs = []\n",
    "    for i in range(len(GENRES)):\n",
    "        for j in range(i + 1, len(GENRES)):\n",
    "            total = MATRIX[i][j] + MATRIX[j][i]\n",
    "            pairs.append((total, GENRES[i], GENRES[j],\n",
    "                          MATRIX[i][j], MATRIX[j][i]))\n",
    "    pairs.sort(reverse=True)\n",
    "    print(f\"  {'pair':<26}{'i->j':>6}{'j->i':>6}{'total':>8}\")\n",
    "    print(\"  \" + \"-\" * 48)\n",
    "    for t, a, b, ab, ba in pairs:\n",
    "        print(f\"  {a + ' / ' + b:<26}{ab:>6}{ba:>6}{t:>8}\")\n",
    "    print(f\"\\n    most confused: {pairs[0][1]} and {pairs[0][2]}, {pairs[0][0]}\"\n",
    "          f\" errors\")\n",
    "\n",
    "    print(\"\\n  Both answers are properties of the matrix, not opinions. Two\")\n",
    "    print(\"  people reading the same matrix must reach the same pair.\")\n",
    "    print(\"\\n  Note that the errors are directional. Government loses 3\")\n",
    "    print(\"  documents to news and takes 2 back, which are different\")\n",
    "    print(\"  mistakes with different causes.\")\n",
    "\n",
    "\n",
    "# --------------------------------------------------------- macro and micro\n",
    "\n",
    "def show_averages():\n",
    "    print(\"\\n\\nSTEP 4  macro, micro, and why one of them is redundant\\n\")\n",
    "    tp = sum(counts(MATRIX, i)[0] for i in range(len(GENRES)))\n",
    "    fp = sum(counts(MATRIX, i)[1] for i in range(len(GENRES)))\n",
    "    fn = sum(counts(MATRIX, i)[2] for i in range(len(GENRES)))\n",
    "    micro = prf(tp, fp, fn)\n",
    "\n",
    "    print(\"  MACRO averages the per-class F1 scores. Every class counts once,\")\n",
    "    print(\"  however rare it is.\\n\")\n",
    "    print(\"  MICRO pools every decision first, then computes one F1. Frequent\")\n",
    "    print(\"  classes therefore dominate it.\\n\")\n",
    "    print(f\"  pooled TP = {tp}, FP = {fp}, FN = {fn}\")\n",
    "    print(f\"\\n  {'measure':<14}{'value':>9}\")\n",
    "    print(\"  \" + \"-\" * 26)\n",
    "    print(f\"  {'accuracy':<14}{accuracy(MATRIX):>9.4f}\")\n",
    "    print(f\"  {'micro-F1':<14}{micro[2]:>9.4f}\")\n",
    "    print(f\"  {'macro-F1':<14}{macro_f1(MATRIX):>9.4f}\")\n",
    "\n",
    "    print(\"\\n  Micro-F1 and accuracy are the same number, and that is not a\")\n",
    "    print(\"  coincidence. When every document gets exactly one label, every\")\n",
    "    print(\"  error is one FP and one FN at the same time, so the pooled\")\n",
    "    print(\"  precision and recall are both the fraction correct.\")\n",
    "    print(\"\\n  So on single-label tasks micro-F1 tells you nothing accuracy\")\n",
    "    print(\"  did not. Report macro-F1, or report the per-class table.\")\n",
    "\n",
    "    print(\"\\n\\n  ABLATION DISCIPLINE\\n\")\n",
    "    runs = [(\"unigrams, raw counts\", 0.612),\n",
    "            (\"unigrams, tf-idf\", 0.681),\n",
    "            (\"unigrams + bigrams, tf-idf\", 0.724),\n",
    "            (\"+ min document frequency 2\", 0.748),\n",
    "            (\"+ lowercasing\", 0.741)]\n",
    "    print(f\"  {'configuration':<32}{'validation macro-F1':>20}\")\n",
    "    print(\"  \" + \"-\" * 54)\n",
    "    best = max(runs, key=lambda r: r[1])\n",
    "    for name, score in runs:\n",
    "        mark = \"   <- best\" if (name, score) == best else \"\"\n",
    "        print(f\"  {name:<32}{score:>20.3f}{mark}\")\n",
    "\n",
    "    print(\"\\n  One factor changes per row. That is what makes it an ablation\")\n",
    "    print(\"  rather than a list of guesses.\")\n",
    "    print(\"\\n  The last row went down, and reporting it is the point. An\")\n",
    "    print(\"  ablation is a record of runs you did, not of runs that worked.\")\n",
    "    print(\"\\n  The best row must be the model you actually submit. If it is\")\n",
    "    print(\"  not, your experiment log and your system have diverged.\")\n",
    "\n",
    "\n",
    "def main():\n",
    "    ap = argparse.ArgumentParser(description=__doc__)\n",
    "    for f in (\"trap\", \"matrix\", \"read\", \"averages\"):\n",
    "        ap.add_argument(f\"--{f}\", action=\"store_true\")\n",
    "    a = ap.parse_args()\n",
    "    picked = a.trap or a.matrix or a.read or a.averages\n",
    "    if a.trap or not picked:\n",
    "        show_trap()\n",
    "    if a.matrix or not picked:\n",
    "        show_matrix()\n",
    "    if a.read or not picked:\n",
    "        show_read()\n",
    "    if a.averages or not picked:\n",
    "        show_averages()\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running it\n",
    "\n",
    "The script is a command line tool, so each run below sets the arguments\n",
    "and calls `main()`. The heading shows the equivalent shell command."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "import sys\n",
    "\n",
    "def run(*args):\n",
    "    sys.argv = [\"classification.py\", *args]\n",
    "    main()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `python3 classification.py`\n",
    "\n",
    "The whole evaluation, from counts to macro-F1."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "run()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `python3 classification.py --trap`\n",
    "\n",
    "A four-class model scoring 0.90 accuracy and 0.24 macro-F1 by always answering the majority class. Three classes score exactly zero and accuracy does not notice."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "run(\"--trap\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `python3 classification.py --read`\n",
    "\n",
    "The hardest class and the most confused pair, computed rather than judged. government has the lowest F1 at 0.6364, and news and government exchange 5 documents."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "run(\"--read\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "Read the chapter this comes from: [Text Classification and Evaluation](https://nlp.jcrlabz.com/book/classification/)."
   ]
  }
 ],
 "metadata": {
  "colab": {
   "name": "classification.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}