AICurious Logo

What is: Continuous Bag-of-Words Word2Vec?

SourceEfficient Estimation of Word Representations in Vector Space
Year2000
Data SourceCC BY-SA - https://paperswithcode.com

Continuous Bag-of-Words Word2Vec is an architecture for creating word embeddings that uses nn future words as well as nn past words to create a word embedding. The objective function for CBOW is:

J_θ=1TT_t=1logp(w_tw_tn,,w_t1,w_t+1,,w_t+n)J\_\theta = \frac{1}{T}\sum^{T}\_{t=1}\log{p}\left(w\_{t}\mid{w}\_{t-n},\ldots,w\_{t-1}, w\_{t+1},\ldots,w\_{t+n}\right)

In the CBOW model, the distributed representations of context are used to predict the word in the middle of the window. This contrasts with Skip-gram Word2Vec where the distributed representation of the input word is used to predict the context.