**Mode Normalization** extends normalization to more than a single mean and variance, allowing for detection of modes of data on-the-fly, jointly normalizing samples that share common features. It first assigns samples in a mini-batch to different modes via a gating network, and then normalizes each sample with estimators for its corresponding mode.

The **Feature-wise linear modulation** (**FiLM**) module combines information from both noisy waveform and input mel-spectrogram. It is used in the [WaveGrad](https://paperswithcode.com/method/wavegrad) model. The authors also added iteration index $n$ which indicates the noise level of the input waveform by using the [Transformer](https://paperswithcode.com/method/transformer) sinusoidal positional embedding. To condition on the noise level directly, $n$ is replaced by $\sqrt{\bar{\alpha}}$ and a linear scale $C = 5000$ is applied. The FiLM module produces both scale and bias vectors given inputs, which are used in a UBlock for feature-wise affine transformation as:

$$ \gamma\left(D, \sqrt{\bar{\alpha}}\right) \odot U + \zeta\left(D, \sqrt{\bar{\alpha}}\right) $$

where $\gamma$ and $\zeta$ correspond to the scaling and shift vectors from the FiLM module, $D$ is the output from corresponding [DBlock](https://paperswithcode.com/method/dblock), $U$ is an intermediate output in the UBlock.

FiLM Module

WaveGrad: Estimating Gradients for Waveform Generation

Mode Normalization

A **Res2Net Block** is an image model block that constructs hierarchical residual-like connections
within one single [residual block](https://paperswithcode.com/method/residual-block). It was proposed as part of the [Res2Net](https://paperswithcode.com/method/res2net) CNN architecture.

The block represents multi-scale features at a granular level and increases the range of receptive fields for each network layer. The $3 \times 3$ filters of $n$ channels is replaced with a set of smaller filter groups, each with $w$ channels. These smaller filter groups are connected in a hierarchical residual-like style to increase the number of scales that the output features can represent. Specifically, we divide input feature maps into several groups. A group of filters first extracts features from a group of input feature maps. Output features of the previous group are then sent to the next group of filters along with another group of input feature maps. 

This process repeats several times until all input feature maps are processed. Finally, feature maps from all groups are concatenated and sent to another group of $1 \times 1$ filters to fuse information altogether. Along with any possible path in which input features are transformed to output features, the equivalent receptive field increases whenever it passes a $3 \times 3$ filter, resulting in many equivalent feature scales due to combination effects.

One way of thinking of these blocks is that they expose a new dimension, **scale**,  alongside the existing dimensions of depth, width, and cardinality.

Source	Mode Normalization
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com