**Differentiable Augmentation (DiffAugment)** is a set of differentiable image transformations used to augment data during [GAN](https://paperswithcode.com/method/gan) training. The transformations are applied to the real and generated images. It enables the gradients to be propagated through the augmentation back to the generator, regularizes
the discriminator without manipulating the target distribution, and maintains the balance of training
dynamics. Three choices of transformation are preferred by the authors in their experiments: Translation, [CutOut](https://paperswithcode.com/method/cutout), and Color.

**AlexNet** is a classic convolutional neural network architecture. It consists of convolutions, [max pooling](https://paperswithcode.com/method/max-pooling) and dense layers as the basic building blocks. Grouped convolutions are used in order to fit the model across two GPUs.

AlexNet

ImageNet Classification with Deep Convolutional Neural Networks

DiffAugment

Differentiable Augmentation for Data-Efficient GAN Training

Hou et al. proposed coordinate attention,
a novel attention mechanism which
embeds positional information into channel attention,
so that the network can focus on large important regions 
at little computational cost.

The coordinate attention mechanism has two consecutive steps, coordinate information embedding and coordinate attention generation. First, two spatial extents of pooling kernels encode each channel horizontally  and  vertically. In the second step, a shared $1\times 1$ convolutional transformation function is applied to the concatenated outputs of the two pooling layers. Then coordinate attention splits the resulting tensor into two separate tensors to yield attention vectors with the same number of channels for horizontal and vertical coordinates of the  input $X$ along. This can be written as 
\begin{align}
    z^h &= \text{GAP}^h(X) 
\end{align}
\begin{align}
    z^w &= \text{GAP}^w(X)
\end{align}
\begin{align}
    f &= \delta(\text{BN}(\text{Conv}_1^{1\times 1}([z^h;z^w])))
\end{align}
\begin{align}
    f^h, f^w &= \text{Split}(f)
\end{align}
\begin{align}
    s^h &= \sigma(\text{Conv}_h^{1\times 1}(f^h))
\end{align}
\begin{align}
    s^w &= \sigma(\text{Conv}_w^{1\times 1}(f^w))
\end{align}
\begin{align}
    Y &= X s^h  s^w
\end{align}
where $\text{GAP}^h$ and $\text{GAP}^w$ denote pooling functions for vertical and horizontal coordinates, and $s^h \in \mathbb{R}^{C\times 1\times W}$ and $s^w \in \mathbb{R}^{C\times H\times 1}$ represent corresponding attention weights. 

Using coordinate attention, the network can accurately obtain the position of a targeted object.
This approach has a larger receptive field than BAM and CBAM.
Like an SE block, it also models cross-channel relationships, effectively enhancing the expressive power of the learned features.
Due to its lightweight design and flexibility, 
it can be easily used in classical building blocks of mobile networks.

Source	Differentiable Augmentation for Data-Efficient GAN Training
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com