All tidbits Language models Code Course home

TidbitOne unseen word can make a language model say a sentence is impossible.

Multiply probabilities together and a single zero takes the whole product to zero. The fix is old and blunt: pretend you saw everything a fraction of a time more than you did. Every modern model still needs some version of it.

one zero kills the product

One unseen word can make a language model say a sentence is impossible.