**Cutout** is an image augmentation and regularization technique that randomly masks out square regions of input during training. and can be used to improve the robustness and overall performance of convolutional neural networks. The main motivation for cutout comes from the problem of object occlusion, which is commonly encountered in many computer vision tasks, such as object recognition,
tracking, or human pose estimation. By generating new images which simulate occluded examples, we not only better prepare the model for encounters with occlusions in the real world, but the model also learns to take more of the image context into consideration when making decisions

$s \cdot (\mathrm{ReLU}(x))^2 + b$

where $s \in \mathbb{R}$ and $b \in \mathbb{R}$ are shared for all channels and can be set as constants (s=0.8944, b=-0.4472) or learnable parameters.

StarReLU

MetaFormer Baselines for Vision

Cutout

Improved Regularization of Convolutional Neural Networks with Cutout

The **Exponential Linear Unit** (ELU) is an activation function for neural networks. In contrast to [ReLUs](https://paperswithcode.com/method/relu), ELUs have negative values which allows them to push mean unit activations closer to zero like [batch normalization](https://paperswithcode.com/method/batch-normalization) but with lower computational complexity. Mean shifts toward zero speed up learning by bringing the normal gradient closer to the unit natural gradient because of a reduced bias shift effect. While [LReLUs](https://paperswithcode.com/method/leaky-relu) and [PReLUs](https://paperswithcode.com/method/prelu) have negative values, too, they do not ensure a noise-robust deactivation state. ELUs saturate to a negative value with smaller inputs and thereby decrease the forward propagated variation and information.

The exponential linear unit (ELU) with $0 < \alpha$ is:

$$f\left(x\right) = x \text{ if } x > 0$$
$$\alpha\left(\exp\left(x\right) − 1\right) \text{ if } x \leq 0$$

Source	Improved Regularization of Convolutional Neural Networks with Cutout
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com