AICurious Logo

What is: G-GLN Neuron?

SourceGaussian Gated Linear Networks
Year2000
Data SourceCC BY-SA - https://paperswithcode.com

A G-GLN Neuron is a type of neuron used in the G-GLN architecture. G-GLN. The key idea is that further representational power can be added to a weighted product of Gaussians via a contextual gating procedure. This is achieved by extending a weighted product of Gaussians model with an additional type of input called side information. The side information will be used by a neuron to select a weight vector to apply for a given example from a table of weight vectors. In typical applications to regression, the side information is defined as the (normalized) input features for an input example: i.e. z=(xxˉ)/σ_xz=(x-\bar{x}) / \sigma\_{x}.

More formally, associated with each neuron is a context function c:ZCc: \mathcal{Z} \rightarrow \mathcal{C}, where Z\mathcal{Z} is the set of possible side information and C={0,,k1}\mathcal{C}=\{0, \ldots, k-1\} for some kNk \in \mathbb{N} is the context space. Each neuron ii is now parameterized by a weight matrix W_i=[w_i,0w_i,k1]W\_{i}=\left[w\_{i, 0} \ldots w\_{i, k-1}\right]^{\top} with each row vector w_ijWw\_{i j} \in \mathcal{W} for 0j<k0 \leq j<k. The context function cc is responsible for mapping side information zZz \in \mathcal{Z} to a particular row w_i,c(z)w\_{i, c(z)} of WiW_{i}, which we then use to weight the Product of Gaussians. In other words, a G-GLN neuron can be defined by:

PoG_Wc(y;f1(),,f_m(),z):=PoG_wc(z)(y;f_1(),,f_m())\operatorname{PoG}\_{W}^{c}\left(y ; f_{1}(\cdot), \ldots, f\_{m}(\cdot), z\right):=\operatorname{PoG}\_{w^{c(z)}}\left(y ; f\_{1}(\cdot), \ldots, f\_{m}(\cdot)\right)

with the associated loss function log(PoG_Wc(y;f_1(y),,f_m(y),z))-\log \left(\operatorname{PoG}\_{W}^{c}\left(y ; f\_{1}(y), \ldots, f\_{m}(y), z\right)\right) inheriting all the properties needed to apply Online Convex Programming.