**R_INLINE_MATH_1 Regularization** is a regularization technique and gradient penalty for training [generative adversarial networks](https://paperswithcode.com/methods/category/generative-adversarial-networks). It penalizes the discriminator from deviating from the Nash Equilibrium via penalizing the gradient on real data alone: when the generator distribution produces the true data distribution and the discriminator is equal to 0 on the data manifold, the gradient penalty ensures that the discriminator cannot create a non-zero gradient orthogonal to the data manifold without suffering a loss in the [GAN](https://paperswithcode.com/method/gan) game.

This leads to the following regularization term:

$$ R\_{1}\left(\psi\right) = \frac{\gamma}{2}E\_{p\_{D}\left(x\right)}\left[||\nabla{D\_{\psi}\left(x\right)}||^{2}\right] $$

**Residual Normal Distributions** are used to help the optimization of VAEs, preventing optimization from entering an unstable region. This can happen due to sharp gradients caused in situations where the encoder and decoder produce distributions far away from each other. The residual distribution parameterizes $q\left(\mathbf{z}|\mathbf{x}\right)$ relative to $p\left(\mathbf{z}\right)$. Let $p\left(z^{i}\_{l}|\mathbf{z}\_{<l}\right) := N \left(\mu\_{i}\left(\mathbf{z}\_{<l}\right), \sigma\_{i}\left(\mathbf{z}\_{<l}\right)\right)$ be a Normal distribution for the $i$th variable in $\mathbf{z}\_{l}$ in prior. Define $q\left(z^{i}\_{l}|\mathbf{z}\_{<l}, x\right) := N\left(\mu\_{i}\left(\mathbf{z}\_{<l}\right) + \Delta\mu\_{i}\left(\mathbf{z}\_{<l}, x\right), \sigma\_{i}\left(\mathbf{z}\_{<l}\right) \cdot \Delta\sigma\_{i}\left(\mathbf{z}\_{<l}, x\right) \right)$, where $\Delta\mu\_{i}\left(\mathbf{z}\_{<l}, \mathbf{x}\right)$ and $\Delta\sigma\_{i}\left(\mathbf{z}\_{<l}, \mathbf{x}\right)$ are the relative location and scale of the approximate posterior with respect to the prior. With this parameterization, when the prior moves, the approximate posterior moves accordingly, if not changed.

Residual Normal Distribution

NVAE: A Deep Hierarchical Variational Autoencoder

R1 Regularization

Which Training Methods for GANs do actually Converge?

This method works as a two-levels optimization algorithm.
The outmost layer uses Grammatical evolution to evolve a grammar to build the agent.
Then, [Q-learning](https://paperswithcode.com/method/q-learning) is used the fitness evaluation phase to allow the agent to learn to perform online learning.

Source	Which Training Methods for GANs do actually Converge?
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com