**SwiGLU** is an activation function which is a variant of [GLU](https://paperswithcode.com/method/glu). The definition is as follows:

$$ \text{SwiGLU}\left(x, W, V, b, c, \beta\right) = \text{Swish}\_{\beta}\left(xW + b\right) \otimes \left(xV + c\right) $$

**ClusterFit** is a self-supervision approach for learning image representations.  Given a dataset, we (a) cluster its features extracted from a pre-trained network using k-means and (b) re-train a new network from scratch on this dataset using cluster assignments as pseudo-labels.

ClusterFit

ClusterFit: Improving Generalization of Visual Representations

SwiGLU

GLU Variants Improve Transformer

**Ape-X DPG** combines [DDPG](https://paperswithcode.com/method/ddpg) with distributed [prioritized experience replay](https://paperswithcode.com/method/prioritized-experience-replay) through the [Ape-X](https://paperswithcode.com/method/ape-x) architecture.

Source	GLU Variants Improve Transformer
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com