All labs Animated labs The chapter Course home

Lab 01Bag of words

Two sentences. One says the dog did the biting, the other says the man did. Watch what survives when you drop the words into a bag and keep only the counts.

Word 0 of 10

Sentence 1

The dog bites the man

Sentence 2

The man bites the dog

The bags are still filling.

What just happened

The words arrive in a shuffled order, because the bag does not care when they arrived. Once both sentences are in, the two bags hold exactly the same five counts: the 2, dog 1, bites 1, man 1. Nothing in the representation can tell you who bit whom.

That is the trade. You get a fixed-length vector you can compare, cluster and feed to a classifier, and you pay for it with word order. Every model in the rest of the course is, in some sense, an attempt to buy that order back: n-grams keep a short window of it, recurrent networks carry it forward, attention learns which of it mattered.

Read the chapter · Run the code