**BigBird** is a [Transformer](https://paperswithcode.com/method/transformer) with a sparse attention mechanism that reduces the quadratic dependency of self-attention to linear in the number of tokens. BigBird is a universal approximator of sequence functions and is Turing complete, thereby preserving these properties of the quadratic, full attention model.  In particular, BigBird consists of three main parts:

- A set of $g$ global tokens attending on all parts of the sequence.
- All tokens attending to a set of $w$ local neighboring tokens.
- All tokens attending to a set of $r$ random tokens.

This leads to a high performing attention mechanism scaling to much longer sequence lengths (8x).

**MagFace** is a category of losses for face recognition that learn a universal feature embedding whose magnitude can measure the quality of a given face. Under the new loss, it can be proven that the magnitude of the feature embedding monotonically increases if the subject is more likely to be recognized. In addition, MagFace introduces an adaptive mechanism to learn a well-structured within-class feature distributions by pulling easy samples to class centers while pushing hard samples away. For face recognition, MagFace helps prevent model overfitting on noisy and low-quality samples by an adaptive mechanism to learn well-structured within-class feature distributions -- by pulling easy samples to class centers while pushing hard samples away.

MagFace

MagFace: A Universal Representation for Face Recognition and Quality Assessment

BigBird

Big Bird: Transformers for Longer Sequences

**Mask Scoring R-CNN** is a Mask RCNN with MaskIoU Head, which takes the instance feature and the predicted mask together as input, and predicts the IoU between input mask and ground truth mask.

Source	Big Bird: Transformers for Longer Sequences
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com