A **Deactivable Skip Connection** is a type of skip connection which, instead of concatenating the encoder features
(red) and decoder features (blue), as with [standard skip connections](https://paperswithcode.com/methods/category/skip-connections), it instead fuses the encoder features with part of the decoder features (light blue), to be able to deactivate this operation when needed.

**Conditional Instance Normalization** is a normalization technique where all convolutional weights of a style transfer network are shared across many styles.  The goal of the procedure is transform
a layer’s activations $x$ into a normalized activation $z$ specific to painting style $s$. Building off
[instance normalization](https://paperswithcode.com/method/instance-normalization), we augment the $\gamma$ and $\beta$ parameters so that they’re $N \times C$ matrices, where $N$ is the number of styles being modeled and $C$ is the number of output feature maps. Conditioning on a style is achieved as follows:

$$ z = \gamma\_{s}\left(\frac{x - \mu}{\sigma}\right) + \beta\_{s}$$

where $\mu$ and $\sigma$ are $x$’s mean and standard deviation taken across spatial axes and $\gamma\_{s}$ and $\beta\_{s}$ are obtained by selecting the row corresponding to $s$ in the $\gamma$ and $\beta$ matrices. One added benefit of this approach is that one can stylize a single image into $N$ painting styles with a single feed forward pass of the network with a batch size of $N$.

Conditional Instance Normalization

A Learned Representation For Artistic Style

Deactivable Skip Connection

Cross-modal Deep Face Normals with Deactivable Skip Connections

**Randomized Leaky Rectified Linear Units**, or **RReLU**, are an activation function that randomly samples the negative slope for activation values. It was first proposed and used in the Kaggle NDSB Competition. During training, $a\_{ji}$ is a random number sampled from a uniform distribution $U\left(l, u\right)$. Formally:

$$ y\_{ji} = x\_{ji} \text{   if } x\_{ji} \geq{0} $$
$$ y\_{ji} = a\_{ji}x\_{ji} \text{   if } x\_{ji} < 0 $$

where

$$\alpha\_{ji} \sim U\left(l, u\right), l < u \text{ and } l, u \in \left[0,1\right)$$

In the test phase, we take average of all the $a\_{ji}$ in training similar to [dropout](https://paperswithcode.com/method/dropout), and thus set $a\_{ji}$ to $\frac{l+u}{2}$ to get a deterministic result. As suggested by the NDSB competition winner, $a\_{ji}$ is sampled from $U\left(3, 8\right)$. 

At test time, we use:

$$ y\_{ji} = \frac{x\_{ji}}{\frac{l+u}{2}} $$

Source	Cross-modal Deep Face Normals with Deactivable Skip Connections
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com