**SongNet** is an auto-regressive [Transformer](https://paperswithcode.com/method/transformer)-based language model for rigid format text detection. Sets of symbols are tailor-designed to improve the modeling performance especially on format, rhyme, and sentence integrity. The attention mechanism is improved to impel the model to capture some future information on the format. A pre-training and fine-tuning framework is designed to further improve the generation quality.

The **Universal Transformer** is a generalization of the [Transformer](https://paperswithcode.com/method/transformer) architecture. Universal Transformers combine the parallelizability and global receptive field of feed-forward sequence models like the Transformer with the recurrent inductive bias of [RNNs](https://paperswithcode.com/methods/category/recurrent-neural-networks). They also utilise a dynamic per-position halting mechanism.

Universal Transformer

Universal Transformers

SongNet

SongNet: Rigid Formats Controlled Text Generation

A **Sandwich Transformer** is a variant of a [Transformer](https://paperswithcode.com/method/transformer) that reorders sublayers in the architecture to achieve better performance. The reordering is based on the authors' analysis that models with more self-attention toward the bottom and more
feedforward sublayers toward the top tend to perform better in general.

Source	SongNet: Rigid Formats Controlled Text Generation
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com