TidbitLanguage models cannot see words.

They see tokens. Byte-pair encoding starts from single characters and repeatedly glues the most frequent pair together, so 'unhappiness' can arrive as three pieces while 'the' is one. Vocabulary size is a budget: too small and words shatter, too large and the embedding table eats your parameters.

50k to 100k tokens, typically