Chapter 14Text Classification and Evaluation

The oldest useful task

Part III has spent four chapters making a model better at predicting the next word, moving from counting to a network, then a recurrence, then a gate. At no point did we ask how you would know whether any of it worked. Perplexity was the only answer offered, and Section that section was careful to say that perplexity measures surprise rather than usefulness.

This chapter fixes that, and it does so on the simplest task in the field. Language modelling asks what comes next, whereas classification asks a smaller question and answers it far more reliably: given a document, which of a fixed set of labels applies?

Spam or not spam, positive or negative or neutral, news or fiction or government or learned. The label set is decided in advance and the model picks one.

This is the task that pays for itself, since it runs in every mail system, every support desk and every content moderation pipeline in production today. It is also where the empirical method of this field is easiest to see, which is why the chapter spends more time on evaluation than on models.

A baseline you already have

Everything needed for a strong classifier is already in Part II. Represent each document as a vector of term weights as Chapter 4 did, feed those vectors to a linear classifier such as logistic regression or a linear support vector machine, then train on labelled documents.

That pipeline is thirty years old and it is still the right first move, since on many tasks it comes within a few points of a fine-tuned transformer at a thousandth of the cost.

Two details decide most of its quality. The features come first, where unigrams are the default and adding bigrams captures short phrases such as not good, which unigrams cannot express at all. The weighting comes second, because raw counts let long documents shout while the tf-idf of Chapter 4 fixes both length and the dominance of common words.

A neural classifier replaces the hand-built vector with a learned one, so that the recurrent networks of Chapter 12 can read a document and hand their final state to a softmax. The evaluation, however, is identical, and the evaluation is what this chapter is about.

Why accuracy lies

The obvious score is accuracy, the fraction of test documents labelled correctly, and it is the wrong default, as one example shows. Take a hundred test documents distributed unevenly.

Now build a model that is one line long, which always answers news.

Ninety per cent accuracy, from a model that cannot tell any two documents apart. Accuracy is blind to which classes are wrong, and what we want is a score where every class counts equally however rare it is.

Precision, recall and $F_1$

Treat each class $c$ as a question on its own, that class against all the others, and three counts follow. $TP_c$ is documents predicted $c$ and truly $c$, $FP_c$ is predicted $c$ but truly something else, and $FN_c$ is truly $c$ but predicted something else.

$$\begin{equation} P_c = \frac{TP_c}{TP_c + FP_c}, \qquad R_c = \frac{TP_c}{TP_c + FN_c}, \qquad F1_c = \frac{2\,P_c R_c}{P_c + R_c}. \label{eq:prf} \quad\text{(14.1)} \end{equation}$$

Precision asks how much of what you claimed was right, while recall asks how much of what existed you found, and they pull against each other. Predict $c$ for everything and recall hits $1$ while precision collapses, or predict $c$ only when certain and precision rises while recall falls.

$F1_c$ is their harmonic mean, and the harmonic mean is the point, because it is high only when both are high. An arithmetic mean would let a precision of $1.0$ hide a recall of $0.1$, and the harmonic mean will not.

Macro and micro

To get one number for the whole task, average the per-class scores across the label set.

$$\begin{equation} \text{macro-}F_1 \;=\; \frac{1}{|C|} \sum_{c \in C} F1_c . \label{eq:macrof1} \quad\text{(14.2)} \end{equation}$$

Every class contributes one term whatever its size, so a classifier that ignores one class entirely scores $F1_c = 0$ for it and macro-$F_1$ drops by a full $1/|C|$. Neglecting the rare class therefore costs you, which is exactly what Equation (14.2) is for and why Project 5’s leaderboard uses it.

Micro-averaging pools every decision first and computes one $F_1$ from the totals, so frequent classes dominate. There is a fact about it worth stating because it saves confusion later, which is that on a single-label task micro-$F_1$ equals accuracy exactly.

The reason is simple. When each document gets exactly one label, every error is simultaneously one false positive for the predicted class and one false negative for the true one, so the pooled precision and the pooled recall are both the fraction correct and their harmonic mean is that same number. Micro-$F_1$ therefore tells you nothing accuracy did not, so report macro-$F_1$ or report the per-class table.

Reading a confusion matrix

One number cannot tell you what to fix, whereas a confusion matrix can. Rows are the true class and columns the prediction, so the diagonal is correct and every off-diagonal cell is an error.

Score it with Equation (14.1), taking one row per class.

The classes are balanced here at twelve each, so accuracy is not misleading, and it is still less useful because it says $0.75$ and stops.

The per-class table says which genre is failing and how. Look at government, whose recall is $0.5833$ because five of its twelve documents went elsewhere, and whose precision is $0.7000$ because three documents from other genres were labelled government. Both directions are wrong at once, which is a different problem from being wrong in one direction only.

Two diagnoses the matrix hands you

Project 5 asks for both of these, and both are computed rather than guessed. The hardest class is the one with the lowest per-class $F_1$, which here is government at $0.6364$ against $0.7200$, $0.7200$ and $0.9167$. The most confused pair is the unordered pair with the most cross errors, found by adding the two off-diagonal cells that join them.

So the answer is news and government, with five errors between them. Both diagnoses are properties of the matrix rather than opinions, and two people reading the same matrix must reach the same pair, which is what makes them gradeable.

One more thing the matrix shows that a score cannot is that the errors are directional. Government loses three documents to news and takes two back, and those are different mistakes with different causes.

When the threshold is yours to choose

Everything so far scores a single decision. Precision, recall and $F_1$ all describe one cut through a ranking, and moving the cut moves all three. A model that outputs scores has not made that decision for you, so there is a second question underneath the first: how good is the ordering, before anyone picks a threshold?

Take ten documents, scored and ranked, with three belonging to the class of interest:

Each row is a threshold. Summarising the whole column instead of one row gives two numbers. ROC-AUC is the probability that a randomly chosen positive outranks a randomly chosen negative, and PR-AUC averages precision over the thresholds at which recall actually moves. Here they are $0.9524$ and $0.9167$, and on a balanced problem they usually agree like this.

Why the two disagree, and which to believe

Now make the class rare, which is the ordinary situation in text. Two positive documents among a hundred, ranked third and tenth by the same model:

The ranking is the same ranking. ROC-AUC counts how many negatives sit below each positive, and ninety-eight negatives is a great deal of credit for very little work. PR-AUC asks what fraction of what you flagged was right, and at rank ten that is $2/10 = 0.20$. The first number flatters the model and the second describes what a user would experience.

Under class imbalance, report PR-AUC. This is the accuracy trap from earlier in this chapter returning in a more sophisticated costume, because any measure dominated by the majority class will always look generous.

Scores that are not probabilities

There is a last question that none of these metrics asks. A score of $0.9$ ought to mean the model is right about nine times in ten. Nothing in cross-entropy training guarantees that, and nothing in $F_1$, ROC-AUC or PR-AUC would notice if it were false, because every one of them depends only on the order.

Group the predictions by confidence and compare what each bin promised against what it delivered:

Weighting those gaps by bin size gives an expected calibration error of $0.247$. The least confident bin is honest and the most confident is the worst, promising $0.95$ and delivering $0.40$, which is the usual shape.

The ordering here may still be perfect, so every metric in this chapter so far would report a healthy model. Whenever a decision has a cost attached, and a threshold is chosen to balance that cost, the probability has to mean something and this is the table that says whether it does.

The loop, not the number

A score is the end of an experiment rather than the work. The work is a loop, and the lectures name its discipline in four rules.

Fix the split before you look. Train on train, tune on a held-out slice of train, and touch the test set once at the end, because every decision made while watching test scores is a decision that has quietly memorised the test set.

Ablate one factor at a time. An ablation changes exactly one thing and records the validation score, and a log might read like this.

The last row went down, and reporting it is the point, because an ablation is a record of the runs you did rather than of the runs that worked.

Submit the best row. The winning configuration in your ablation must be the model you actually submit, and if it is not then your experiment log and your system have diverged, with one of them lying.

Read the errors themselves. Which class pairs blur, and what vocabulary do the misclassified documents share? A confusion matrix tells you where to look, and only the documents tell you why.

That loop, running from baseline to iteration to ablation to confusion matrix, is the empirical method of this whole field. It is the same loop for a linear classifier over tf-idf features and for a fine-tuned transformer, and only the middle step changes.

Try it yourself. code/worked_examples/classification.py produces every table in this chapter. --trap builds the ninety-per-cent-accuracy model that has learned nothing, --matrix scores the genre matrix class by class, --read derives the hardest class and the most confused pair, --averages shows micro-$F_1$ landing exactly on accuracy, --threshold sweeps the ranking for ROC-AUC and PR-AUC, and --calibration checks whether the scores mean anything.

Run it in ColabNotebookSource

What Part III leaves behind

Part III began with counting words to predict the next one and it ends with a classifier and a confusion matrix. Three things carry forward unchanged.

The objective is still cross-entropy against a one-hot target, whether the target is the next word or a genre label. The evaluation discipline is now fixed, meaning held-out data, a stated metric, one factor at a time, and a matrix rather than a number. And the representation is still a vector, still built from context.

What Part III could not fix is the vector itself, since every word has had exactly one and Chapter 9 already showed why that fails for bank. Part IV takes that last constraint away.

Further reading.

Jurafsky and Martin (2025) cover naive Bayes, logistic regression and evaluation with more worked cases. Manning et al. (2009) treat classification and evaluation from the information retrieval side, where precision and recall were first made standard. Sokolova and Lapalme (2009) survey the averaging choices and what each of them hides. Everything in this chapter is one import away in sklearn.metrics, whose classification_report prints the per-class table and three different averages, only one of which would have caught the model below.

Code for this chapter

Each one reproduces the tables above. Open it in Colab, change an input and watch which numbers move.

• classification.pyAccuracy, F1 and a confusion matrix

  The accuracy trap, the confusion matrix, and the threshold-free metrics underneath them.

  standard library only Run it in Colab Notebook Source

From Natural Language Processing: From Corpus Statistics to Grounded LLMs, by Ramaseshan Ramachandran.