Commonsense Reasoning for Natural Language Processing
Probably Approximately a Scientific Blog
JANUARY 12, 2021
Traditionally, language models are trained to predict the next word in a sentence (top part of Figure 2, in blue), but they can also predict hidden (masked) words in the middle of the sentence, as in Google's BERT model (top part of Figure 2, in orange). So knowledge in language models is not the most accurate and reliable.
52 
Let's personalize your content