A **Strided EESP** unit is based on the [EESP Unit](https://paperswithcode.com/method/eesp) but is modified to learn representations more efficiently at multiple scales. Depth-wise dilated convolutions are given strides, an [average pooling](https://paperswithcode.com/method/average-pooling) operation is added instead of an identity connection, and the element-wise addition operation is replaced with a concatenation operation, which helps in expanding the dimensions of feature maps efficiently.

**ReLIC**, or **Representation Learning via Invariant Causal Mechanisms**, is a self-supervised learning objective that enforces invariant prediction of proxy targets across augmentations through an invariance regularizer which yields improved generalization guarantees. 

We can write the objective as:

$$
\underset{X}{\mathbb{E}} \underset{\sim\_{l k}, a\_{q \mathcal{A}}}{\mathbb{E}} \sum_{b \in\left$a\_{l k}, a\_{q t}\right$} \mathcal{L}\_{b}\left(Y^{R}, f(X)\right) \text { s.t. } K L\left(p^{d o\left(a\_{l k}\right)}\left(Y^{R} \mid f(X)\right), p^{d o\left(a\_{q t}\right)}\left(Y^{R} \mid f(X)\right)\right) \leq \rho
$$

where $\mathcal{L}$ is the proxy task loss and $K L$ is the Kullback-Leibler (KL) divergence. Note that any distance measure on distributions can be used in place of the KL divergence.

Concretely, as proxy task we associate to every datapoint $x\_{i}$ the label $y\_{i}^{R}=i$. This corresponds to the instance discrimination task, commonly used in contrastive learning. We take pairs of points $\left(x\_{i}, x\_{j}\right)$ to compute similarity scores and use pairs of augmentations $a\_{l k}=\left(a\_{l}, a\_{k}\right) \in$ $\mathcal{A} \times \mathcal{A}$ to perform a style intervention. Given a batch of samples $\left$x\_{i}\right$\_{i=1}^{N} \sim \mathcal{D}$, we use

$$
p^{d o\left(a\_{l k}\right)}\left(Y^{R}=j \mid f\left(x\_{i}\right)\right) \propto \exp \left(\phi\left(f\left(x\_{i}^{a\_{l}}\right), h\left(x\_{j}^{a\_{k}}\right)\right) / \tau\right)
$$

with $x^{a}$ data augmented with $a$ and $\tau$ a softmax temperature parameter. We encode $f$ using a neural network and choose $h$ to be related to $f$, e.g. $h=f$ or as a network with an exponential moving average of the weights of $f$ (e.g. target networks). To compare representations we use the function $\phi\left(f\left(x\_{i}\right), h\left(x\_{j}\right)\right)=\left\langle g\left(f\left(x\_{i}\right)\right), g\left(h\left(x\_{j}\right)\right)\right\rangle$ where $g$ is a fully-connected neural network often called the critic.

Combining these pieces, we learn representations by minimizing the following objective over the full set of data $x\_{i} \in \mathcal{D}$ and augmentations $a_{l k} \in \mathcal{A} \times \mathcal{A}$

$$
-\sum_{i=1}^{N} \sum\_{a\_{l k}} \log \frac{\exp \left(\phi\left(f\left(x\_{i}^{a_{l}}\right), h\left(x\_{i}^{a\_{k}}\right)\right) / \tau\right)}{\sum\_{m=1}^{M} \exp \left(\phi\left(f\left(x\_{i}^{a\_{l}}\right), h\left(x\_{m}^{a\_{k}}\right)\right) / \tau\right)}+\alpha \sum\_{a\_{l k}, a\_{q t}} K L\left(p^{d o\left(a\_{l k}\right)}, p^{d o\left(a\_{q t}\right)}\right)
$$

with $M$ the number of points we use to construct the contrast set and $\alpha$ the weighting of the invariance penalty. The shorthand $p^{d o(a)}$ is used for $p^{d o(a)}\left(Y^{R}=j \mid f\left(x\_{i}\right)\right)$. The Figure shows a schematic of the RELIC objective.

ReLIC

Representation Learning via Invariant Causal Mechanisms

Strided EESP

ESPNetv2: A Light-weight, Power Efficient, and General Purpose Convolutional Neural Network

In a given dataset for semantic image segmentation, the number of samples per class should be the same, so that no classifier would be biased towards the majority class (here included the background). It is very difficult, if not impossible, to achieve a perfect balance between the several classes of objects of a dataset. Considering that the segmentation of the objects  is accomplished at the pixel level, the number of pixels for each class must be taken into account. As a matter of fact, in image semantic segmentation, 
different classes and the background may have quite different
sizes. Therefore, the image segmentation problem is naturally unbalanced. The IPBI is based on the concept of entropy, a common measure used in many fields of science. In a general sense, it measures the amount of disorder of a system. For the sake of semantic image segmentation, the ideal dataset should have the same number of instances per class, as well as the same number of pixels in all classes. Similar reasoning can be done considering the number of pixels of all samples in a class, so that we can obtain the
pixels balance measure for the dataset. Overall, IPBI evaluates the balance of pixels and number of instances of an image semantic segmentation dataset and, so, it is usefull to compare different datasets.

Source	ESPNetv2: A Light-weight, Power Efficient, and General Purpose Convolutional Neural Network
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com