Course notesWorked examples

Every table and trace the notes print by hand is produced by one of these. They belong to the chapters, and each chapter lists its own at the foot of the page. This index exists so no link is ever dead.

What Natural Language Processing Is

• bridging_the_gap.pyOne request, three answers

Corpora and Preprocessing

• edit_distance.pyEdit distance, every cell

The Empirical Laws of Text

• empirical_laws.pyZipf and Heaps, fitted

Term Weighting and Similarity

• weighting.pytf-idf and PMI, by hand

Subword Tokenisation

• bpe.pyBPE, five merges

Count Vectors, PPMI, and SVD

• weighting.pytf-idf and PMI, by hand
• hal.pyHAL, ramped and asymmetric
• coals.pyCOALS, all three steps
• svd_rank.pyChoosing K from the spectrum
• svd.pySVD, and what truncation costs

Learned Word Embeddings

• word2vec.pyword2vec, one step at a time
• factorisation.pyThree methods, one matrix

n-gram Language Models and Perplexity

• ngram_lm.pySmoothing, perplexity and the U-curve

Neural Language Models

• neural_lm.pyA neural LM, counted and trained

Recurrent Networks

• rnn.pyUnrolling, and the vanishing gradient

Gated Recurrence: LSTM and GRU

• gated.pyThe gradient highway

Text Classification and Evaluation

• classification.pyAccuracy, F1 and a confusion matrix

Contextual Embeddings

• contextual.pyWhat one vector per word costs

Self-Attention and the Transformer

• attention.pySelf-attention, one query at a time

Pretrained Transformers: BERT and GPT

• contextual.pyWhat one vector per word costs

Steering LLMs: Decoding and Prompting

• decoding.pyGreedy, beam, temperature, top-p

Evaluating Generated Text

• bleu.pyBLEU, ROUGE and their blind spots

Machine Translation

• ibm_model1.pyEM learning an alignment