A **Deep Belief Network (DBN)** is a multi-layer generative graphical model. DBNs have bi-directional connections ([RBM](https://paperswithcode.com/method/restricted-boltzmann-machine)-type connections) on the top layer while the bottom layers only have top-down connections. They are trained using layerwise pre-training. Pre-training occurs by training the network component by component bottom up: treating the first two layers as an RBM and training, then treating the second layer and third layer as another RBM and training for those parameters.

Source: [Origins of Deep Learning](https://arxiv.org/pdf/1702.07800.pdf)

Image Source: [Wikipedia](https://en.wikipedia.org/wiki/Deep_belief_network)

**Adaptive Input Embeddings** extend the [adaptive softmax](https://paperswithcode.com/method/adaptive-softmax) to input word representations. The factorization assigns more capacity to frequent words and reduces the capacity for less frequent words with the benefit of reducing overfitting to rare words.

Adaptive Input Representations

Adaptive Input Representations for Neural Language Modeling

Deep Belief Network

**HS-ResNet** is a [convolutional neural network](https://paperswithcode.com/methods/category/convolutional-neural-networks) that employs [Hierarchical-Split Block](https://paperswithcode.com/method/hierarchical-split-block) as its central building block within a [ResNet](https://paperswithcode.com/method/resnet)-like architecture.

Year	2009
Data Source	CC BY-SA - https://paperswithcode.com