Sebastian Ruder

NOVEMBER 14, 2022

Research models such as BERT and T5 have become much more accessible while the latest generation of language and multi-modal models are demonstrating increasingly powerful capabilities. In Findings of the Association for Computational Linguistics: ACL 2022 (pp. RoBERTa: A Robustly Optimized BERT Pretraining Approach.

Natural Language Processing 52

Natural Language Processing NLP Computational Linguistics AI 52

Sebastian Ruder

AUGUST 18, 2019

In contrast, current models like BERT-Large and GPT-2 consist of 24 Transformer blocks and recent models are even deeper. The latter in particular finds that simply training BERT for longer and on more data improves results, while GPT-2 8B reduces perplexity on a language modelling dataset (though only by a comparatively small factor).

NLP 75

NLP BERT Natural Language Processing Computational Linguistics 75

Topbots

JUNE 15, 2023

70% of research papers published in a computational linguistics conference only evaluated English.[ In Findings of the Association for Computational Linguistics: ACL 2022 , pages 2340–2354, Dublin, Ireland. Association for Computational Linguistics. Are All Languages Created Equal in Multilingual BERT?

Natural Language Processing 59

Natural Language Processing Computational Linguistics NLP ChatGPT 59

The Stanford AI Lab Blog

JANUARY 21, 2022

Conference of the North American Chapter of the Association for Computational Linguistics. ↩ Devlin, J., BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. RoBERTa: A Robustly Optimized BERT Pretraining Approach. . ↩ Peters, M., Neumann, M., Gardner, M., Zettlemoyer, L.

Robotics 40

Robotics Computational Linguistics Computer Vision BERT 40

Explosion

DECEMBER 28, 2019

The update fixed outstanding bugs on the tracker, gave the docs a huge makeover, improved both speed and accuracy, made installation significantly easier and faster, and added some exciting new features, like ULMFit/BERT/ELMo-style language model pretraining. ✨ Mar 20: A few days later, we upgraded Prodigy to v1.8 to support spaCy v2.1.

NLP 52

NLP BERT Machine Learning Python 52

Sebastian Ruder

JANUARY 24, 2022

6] such as W2v-BERT [7] as well as more powerful multilingual models such as XLS-R [8]. For each input chunk, nearest neighbor chunks are retrieved using approximate nearest neighbor search based on BERT embedding similarity. W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training.

NLP 52

NLP ML BERT Computational Linguistics 52

Sebastian Ruder

DECEMBER 6, 2021

Reading Comprehension assumes a gold paragraph is provided Standard approaches for reading comprehension build on pre-trained models such as BERT. Using BERT for reading comprehension involves fine-tuning it to predict a) whether a question is answerable and b) whether each token is the start and end of an answer span.

BERT 52

BERT NLP Natural Language Processing Computational Linguistics 52

NLP People

JULY 24, 2019

The 57th Annual Meeting of the Association for Computational Linguistics (ACL 2019) is starting this week in Florence, Italy. Especially pre-trained word embeddings such as Word2Vec, FastText and BERT allow NLP developers to jump to the next level. Transfer learning is another approach to reusing models across different tasks.

NLP 52

NLP Neural Network Deep Learning Natural Language Processing 52