Chapter 14Text Classification and Evaluation

The oldest useful task

Part III has spent four chapters making a model better at predicting the next word. Counting, then a network, then a recurrence, then a gate.

At no point did we ask how you would know whether any of it worked. Perplexity was the only answer offered, and Section that section was careful to say that perplexity measures surprise rather than usefulness.

This chapter fixes that, and it does so on the simplest task in the field.

Language modelling asks what comes next. Classification asks a smaller question and answers it far more reliably.

Given a document, which of a fixed set of labels applies?

Spam or not spam. Positive, negative or neutral. News, fiction, government or learned. The label set is decided in advance, and the model picks one.

This is the task that pays for itself. It runs in every mail system, every support desk and every content moderation pipeline in production today.

It is also where the empirical method of this field is easiest to see, which is why the chapter spends more time on evaluation than on models.

A baseline you already have

Everything needed for a strong classifier is in Part II.

Represent each document as a vector of term weights, as Chapter 4 did. Feed those vectors to a linear classifier such as logistic regression or a linear support vector machine. Train on labelled documents.

That pipeline is thirty years old and it is still the right first move. On many tasks it is within a few points of a fine-tuned transformer, at a thousandth of the cost.

Two details decide most of its quality.

The features. Unigrams are the default. Adding bigrams captures short phrases such as not good, which unigrams cannot express at all.

The weighting. Raw counts let long documents shout. The tf-idf of Chapter 4 fixes both length and the dominance of common words.

A neural classifier replaces the hand-built vector with a learned one. The recurrent networks of Chapter 12 can read a document and hand their final state to a softmax. The evaluation, however, is identical, and the evaluation is what this chapter is about.

Why accuracy lies

The obvious score is accuracy, the fraction of test documents labelled correctly. It is the wrong default, and one example shows why.

Take a hundred test documents distributed unevenly.

Now build a model that is one line long. Always answer news.

Ninety per cent accuracy, from a model that cannot tell any two documents apart.

Accuracy is blind to which classes are wrong. What we want is a score where every class counts equally, however rare it is.

Precision, recall and $F_1$

Treat each class $c$ as a question on its own, that class against all the others.

Three counts follow. $TP_c$ is documents predicted $c$ and truly $c$. $FP_c$ is predicted $c$ but truly something else. $FN_c$ is truly $c$ but predicted something else.

$$\begin{equation} P_c = \frac{TP_c}{TP_c + FP_c}, \qquad R_c = \frac{TP_c}{TP_c + FN_c}, \qquad F1_c = \frac{2\,P_c R_c}{P_c + R_c}. \label{eq:prf} \quad\text{(14.1)} \end{equation}$$

Precision asks how much of what you claimed was right. Recall asks how much of what existed you found.

They pull against each other. Predict $c$ for everything and recall hits $1$ while precision collapses. Predict $c$ only when certain and precision rises while recall falls.

$F1_c$ is their harmonic mean, and the harmonic mean is the point. It is high only when both are high. An arithmetic mean would let a precision of $1.0$ hide a recall of $0.1$, and the harmonic mean will not.

Macro and micro

To get one number for the whole task, average the per-class scores.

$$\begin{equation} \text{macro-}F_1 \;=\; \frac{1}{|C|} \sum_{c \in C} F1_c . \label{eq:macrof1} \quad\text{(14.2)} \end{equation}$$

Every class contributes one term, whatever its size. A classifier that ignores one class entirely scores $F1_c = 0$ for it, and macro-$F_1$ drops by a full $1/|C|$.

Neglecting the rare class therefore costs you. That is exactly what Equation (14.2) is for, and why Project 5’s leaderboard uses it.

Micro-averaging pools every decision first and computes one $F_1$ from the totals. Frequent classes then dominate.

There is a fact about micro-averaging worth stating, because it saves confusion later. On a single-label task micro-$F_1$ equals accuracy exactly.

The reason is simple. When each document gets exactly one label, every error is simultaneously one false positive for the predicted class and one false negative for the true one. So the pooled precision and the pooled recall are both the fraction correct, and their harmonic mean is that same number.

So micro-$F_1$ tells you nothing accuracy did not. Report macro-$F_1$, or report the per-class table.

Reading a confusion matrix

One number cannot tell you what to fix. A confusion matrix can.

Rows are the true class, columns the prediction. The diagonal is correct and every off-diagonal cell is an error.

Score it with Equation (14.1), one row per class.

The classes are balanced here at twelve each, so accuracy is not misleading. It is still less useful. It says $0.75$ and stops.

The per-class table says which genre is failing and how. Look at government. Recall is $0.5833$, because five of its twelve documents went elsewhere. Precision is $0.7000$, because three documents from other genres were labelled government.

Both directions are wrong at once, which is a different problem from being wrong in one direction only.

Two diagnoses the matrix hands you

Project 5 asks for both of these, and both are computed rather than guessed.

The hardest class is the one with the lowest per-class $F_1$. Here that is government at $0.6364$, against $0.7200$, $0.7200$ and $0.9167$.

The most confused pair is the unordered pair with the most cross errors. Add the two off-diagonal cells that join them.

So the answer is news and government, with five errors between them.

Both diagnoses are properties of the matrix, not opinions. Two people reading the same matrix must reach the same pair, which is what makes them gradeable.

One more thing the matrix shows that a score cannot. The errors are directional. Government loses three documents to news and takes two back, and those are different mistakes with different causes.

The loop, not the number

A score is the end of an experiment, not the work. The work is a loop, and the lectures name its discipline.

Fix the split before you look.

Train on train. Tune on a held-out slice of train. Touch the test set once, at the end. Every decision made while watching test scores is a decision that has quietly memorised the test set.

Ablate one factor at a time.

An ablation changes exactly one thing and records the validation score. A log might read like this.

The last row went down, and reporting it is the point. An ablation is a record of the runs you did, not of the runs that worked.

Submit the best row.

The winning configuration in your ablation must be the model you actually submit. If it is not, your experiment log and your system have diverged, and one of them is lying.

Read the errors themselves.

Which class pairs blur? What vocabulary do the misclassified documents share? A confusion matrix tells you where to look, and only the documents tell you why.

That loop, baseline to iteration to ablation to confusion matrix, is the empirical method of this whole field.

It is the same loop for a linear classifier over tf-idf features and for a fine-tuned transformer. Only the middle step changes.

Try it yourself. code/worked_examples/classification.py produces every table in this chapter. --trap builds the ninety-per-cent-accuracy model that has learned nothing, --matrix scores the genre matrix class by class, --read derives the hardest class and the most confused pair, and --averages shows micro-$F_1$ landing exactly on accuracy.

Run it in ColabNotebookSource

What Part III leaves behind

Part III began with counting words to predict the next one. It ends with a classifier and a confusion matrix.

Three things carry forward unchanged.

The objective is still cross-entropy against a one-hot target, whether the target is the next word or a genre label.

The evaluation discipline is now fixed. Held-out data, a stated metric, one factor at a time, and a matrix rather than a number.

The representation is still a vector, and it is still built from context.

What Part III could not fix is the vector itself. Every word has had exactly one, and Chapter 9 already showed why that fails for bank.

Part IV takes that last constraint away.

Further reading.

cover naive Bayes, logistic regression and evaluation with more worked cases. treat classification and evaluation from the information retrieval side, where precision and recall were first made standard. survey the averaging choices and what each of them hides.