**Sparsemax** is a type of activation/output function similar to the traditional [softmax](https://paperswithcode.com/method/softmax), but able to output sparse probabilities. 

$$ \text{sparsemax}\left(z\right) = \arg\_{p∈\Delta^{K−1}}\min||\mathbf{p} - \mathbf{z}||^{2} $$

**Residual Multi-Layer Perceptrons**, or **ResMLP**, is an architecture built entirely upon [multi-layer perceptrons](https://paperswithcode.com/methods/category/feedforward-networks) for image classification. It is a simple [residual network](https://paperswithcode.com/method/residual-connection) that alternates (i) a [linear layer](https://paperswithcode.com/method/linear-layer) in which image patches interact, independently and identically across channels, and (ii) a two-layer [feed-forward network](https://paperswithcode.com/method/feedforward-network) in which channels interact independently per patch. At the end of the network, the patch representations are average pooled, and fed to a linear classifier.

[Layer normalization](https://paperswithcode.com/method/layer-normalization) is replaced with a simpler [affine transformation](https://paperswithcode.com/method/affine-operator), thanks to the absence of self-attention layers which makes training more stable. The affine operator is applied at the beginning ("pre-normalization") and end ("post-normalization") of each residual block. As a pre-normalization, Aff replaces LayerNorm without using channel-wise statistics. Initialization is achieved as $\mathbf{\alpha}=\mathbf{1}$, and $\mathbf{\beta}=\mathbf{0}$. As a post-normalization, Aff is similar to [LayerScale](https://paperswithcode.com/method/layerscale) and $\mathbf{\alpha}$ is initialized with the same small value.

ResMLP

ResMLP: Feedforward networks for image classification with data-efficient training

Sparsemax

From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification

Unlike [batch normalization](https://paperswithcode.com/method/batch-normalization), **Layer Normalization** directly estimates the normalization statistics from the summed inputs to the neurons within a hidden layer so the normalization does not introduce any new dependencies between training cases. It works well for [RNNs](https://paperswithcode.com/methods/category/recurrent-neural-networks) and improves both the training time and the generalization performance of several existing RNN models. More recently, it has been used with [Transformer](https://paperswithcode.com/methods/category/transformers) models.

We compute the layer normalization statistics over all the hidden units in the same layer as follows:

$$ \mu^{l} = \frac{1}{H}\sum^{H}\_{i=1}a\_{i}^{l} $$

$$ \sigma^{l} = \sqrt{\frac{1}{H}\sum^{H}\_{i=1}\left(a\_{i}^{l}-\mu^{l}\right)^{2}}  $$

where $H$ denotes the number of hidden units in a layer. Under layer normalization, all the hidden units in a layer share the same normalization terms $\mu$ and $\sigma$, but different training cases have different normalization terms. Unlike batch normalization, layer normalization does not impose any constraint on the size of the mini-batch and it can be used in the pure online regime with batch size 1.

Source	From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com