GENet combines part gathering and excitation operations. In the first step, it aggregates input features over large neighborhoods and models the relationship between different spatial locations. In the second step, it first generates an attention map of the same size as the input feature map, using interpolation. Then each position in the input feature map is scaled by multiplying by the corresponding element in the attention map.

SCARF is a simple, widely-applicable technique for contrastive learning, where views are formed by corrupting a random subset of features. When applied to pre-train deep neural networks on the 69 real-world, tabular classification datasets from the OpenML-CC18 benchmark, SCARF not only improves classification accuracy in the fully-supervised setting but does so also in the presence of label noise and in the semi-supervised setting where only a fraction of the available training data is labeled.

SCARF

SCARF: Self-Supervised Contrastive Learning using Random Feature Corruption

Gather-Excite Networks

Gather-Excite: Exploiting Feature Context in Convolutional Neural Networks

**Deformable Attention Module** is an attention module used in the [Deformable DETR](https://paperswithcode.com/method/deformable-detr) architecture, which seeks to overcome one issue base [Transformer attention](https://paperswithcode.com/method/scaled) in that it looks over all possible spatial locations. Inspired by [deformable convolution](https://paperswithcode.com/method/deformable-convolution), the deformable attention module only attends to a small set of key sampling points around a reference point, regardless of the spatial size of the feature maps. By assigning only a small fixed number of keys for each query, the issues of convergence and feature spatial resolution can be mitigated.

Given an input feature map $x \in \mathbb{R}^{C \times H \times W}$, let $q$ index a query element with content feature $\mathbf{z}\_{q}$ and a 2-d reference point $\mathbf{p}\_{q}$, the deformable attention feature is calculated by:

$$ \text{DeformAttn}\left(\mathbf{z}\_{q}, \mathbf{p}\_{q}, \mathbf{x}\right)=\sum\_{m=1}^{M} \mathbf{W}\_{m}\left[\sum\_{k=1}^{K} A\_{m q k} \cdot \mathbf{W}\_{m}^{\prime} \mathbf{x}\left(\mathbf{p}\_{q}+\Delta \mathbf{p}\_{m q k}\right)\right]
$$

where $m$ indexes the attention head, $k$ indexes the sampled keys, and $K$ is the total sampled key number $(K \ll H W) . \Delta p_{m q k}$ and $A_{m q k}$ denote the sampling offset and attention weight of the $k^{\text {th }}$ sampling point in the $m^{\text {th }}$ attention head, respectively. The scalar attention weight $A_{m q k}$ lies in the range $[0,1]$, normalized by $\sum_{k=1}^{K} A_{m q k}=1 . \Delta \mathbf{p}_{m q k} \in \mathbb{R}^{2}$ are of 2-d real numbers with unconstrained range. As $p\_{q}+\Delta p\_{m q k}$ is fractional, bilinear interpolation is applied as in Dai et al. (2017) in computing $\mathbf{x}\left(\mathbf{p}\_{q}+\Delta \mathbf{p}\_{m q k}\right)$. Both $\Delta \mathbf{p}\_{m q k}$ and $A\_{m q k}$ are obtained via linear projection over the query feature $z\_{q} .$ In implementation, the query feature $z\_{q}$ is fed to a linear projection operator of $3 M K$ channels, where the first $2 M K$ channels encode the sampling offsets $\Delta p\_{m q k}$, and the remaining $M K$ channels are fed to a softmax operator to obtain the attention weights $A\_{m q k}$.

Source	Gather-Excite: Exploiting Feature Context in Convolutional Neural Networks
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com