**WaveGrad DBlocks** are used to downsample the temporal dimension of noisy waveform in [WaveGrad](https://paperswithcode.com/method/wavegrad). They are similar to UBlocks except that only one [residual block](https://paperswithcode.com/method/residual-block) is included. The dilation factors are 1, 2, 4 in the main branch. Orthogonal initialization is used.

**ALBERT** is a [Transformer](https://paperswithcode.com/method/transformer) architecture based on [BERT](https://paperswithcode.com/method/bert) but with much fewer parameters. It achieves this through two parameter reduction techniques. The first is a factorized embeddings parameterization. By decomposing the large vocabulary embedding matrix into two small matrices, the size of the hidden layers is separated from the size of vocabulary embedding. This makes it easier to grow the hidden size without significantly increasing the parameter size of the vocabulary embeddings. The second technique is cross-layer parameter sharing. This technique prevents the parameter from growing with the depth of the network. 

Additionally, ALBERT utilises a self-supervised loss for sentence-order prediction (SOP). SOP primary focuses on inter-sentence coherence and is designed to address the ineffectiveness of the next sentence prediction (NSP) loss proposed in the original BERT.

ALBERT

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

WaveGrad DBlock

WaveGrad: Estimating Gradients for Waveform Generation

A **Stochastic Dueling Network**, or **SDN**, is an architecture for learning a value function $V$. The SDN learns both $V$ and $Q$ off-policy while maintaining consistency between the two estimates. At each time step it outputs a stochastic estimate of $Q$ and a deterministic estimate of $V$.

Source	WaveGrad: Estimating Gradients for Waveform Generation
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com