**Randomized Leaky Rectified Linear Units**, or **RReLU**, are an activation function that randomly samples the negative slope for activation values. It was first proposed and used in the Kaggle NDSB Competition. During training, $a\_{ji}$ is a random number sampled from a uniform distribution $U\left(l, u\right)$. Formally:

$$ y\_{ji} = x\_{ji} \text{   if } x\_{ji} \geq{0} $$
$$ y\_{ji} = a\_{ji}x\_{ji} \text{   if } x\_{ji} < 0 $$

where

$$\alpha\_{ji} \sim U\left(l, u\right), l < u \text{ and } l, u \in \left[0,1\right)$$

In the test phase, we take average of all the $a\_{ji}$ in training similar to [dropout](https://paperswithcode.com/method/dropout), and thus set $a\_{ji}$ to $\frac{l+u}{2}$ to get a deterministic result. As suggested by the NDSB competition winner, $a\_{ji}$ is sampled from $U\left(3, 8\right)$. 

At test time, we use:

$$ y\_{ji} = \frac{x\_{ji}}{\frac{l+u}{2}} $$

Multi-layer neural networks traditionally use  dot products between the output vector of previous layer and the incoming weight vector as the input to activation function. The result of dot product is unbounded. To bound dot product and decrease the variance, **Cosine Normalization** uses cosine similarity or centered cosine similarity (Pearson Correlation Coefficient) instead of dot products in neural networks. 

Using cosine normalization, the output of a hidden unit is computed by:

$$o = f(net_{norm})= f(\cos \theta) = f(\frac{\vec{w} \cdot \vec{x}} {\left|\vec{w}\right|  \left|\vec{x}\right|})$$

where $net_{norm}$ is the normalized pre-activation,  $\vec{w}$ is the incoming weight vector and $\vec{x}$ is the input vector, ($\cdot$) indicates dot product, $f$ is nonlinear activation function. Cosine normalization bounds the pre-activation between -1 and 1.

Cosine Normalization

Cosine Normalization: Using Cosine Similarity Instead of Dot Product in Neural Networks

RReLU

Empirical Evaluation of Rectified Activations in Convolutional Network

**TABBIE** is a pretraining objective (*corrupt cell detection*) that learns exclusively from tabular data. Unlike other approaches, TABBIE provides embeddings of all table substructures (cells, rows, and columns). TABBIE can be seen as a table embedding model trained to detect corrupted cells, inspired by the [ELECTRA](https://www.paperswithcode.com/method/electra) objective function.

Source	Empirical Evaluation of Rectified Activations in Convolutional Network
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com