**Variational Dropout** is a regularization technique based on [dropout](https://paperswithcode.com/method/dropout), but uses a variational inference grounded approach. In Variational Dropout, we repeat the same dropout mask at each time step for both inputs, outputs, and recurrent layers (drop the same network units at each time step). This is in contrast to ordinary Dropout where different dropout masks are sampled at each time step for the inputs and outputs alone.

mBERT

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Variational Dropout

A Theoretically Grounded Application of Dropout in Recurrent Neural Networks

**PAUSE**, or **Positive and Annealed Unlabeled Sentence Embedding**, is an approach for learning sentence embeddings from a partially labeled dataset. It is based on a dual encoder schema that is widely adopted in supervised sentence embedding training. Each individual sample $\mathbf{x}$ contains a pair of hypothesis and premise sentences $(x\_{i},x^{\prime}_{i})$, each of which is fed into a pretrained encoder (e.g. [BERT](https://paperswithcode.com/method/bert)). As shown in Figure, the two encoders are identical during the training by sharing their weights.

Source	A Theoretically Grounded Application of Dropout in Recurrent Neural Networks
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com