A capsule is an activation vector that basically executes on its inputs some complex internal
computations. Length of these activation vectors signifies the
probability of availability of a feature. Furthermore, the condition
of the recognized element is encoded as the direction in which
the vector is pointing. In traditional, CNN uses Max pooling for
invariance activities of neurons, which is nothing except a minor
change in input and the neurons of output signal will remains
same.

**Skim and Intensive Reading Model**, or **SIRM**, is a deep neural network for figuring out implied textual meaning. It consists of two main components, namely the skim reading component and intensive reading component. N-gram features are quickly extracted from the skim reading component, which is a combination of several convolutional neural networks, as skim (entire) information. An intensive reading component enables a hierarchical investigation for both local (sentence) and global (paragraph) representation, which encapsulates the current embedding and the contextual information with a dense connection.

SIRM

Read Beyond the Lines: Understanding the Implied Textual Meaning via a Skim and Intensive Reading Model

Capsule Network

Dynamic Routing Between Capsules

**Leaky Rectified Linear Unit**, or **Leaky ReLU**, is a type of activation function based on a [ReLU](https://paperswithcode.com/method/relu), but it has a small slope for negative values instead of a flat slope. The slope coefficient is determined before training, i.e. it is not learnt during training. This type of activation function is popular in tasks where we may suffer from sparse gradients, for example training generative adversarial networks.

Source	Dynamic Routing Between Capsules
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com