**Grouped-query attention** an interpolation of multi-query and multi-head attention that achieves quality close to multi-head at comparable speed to multi-query attention.

A **ResNeXt Block** is a type of [residual block](https://paperswithcode.com/method/residual-block) used as part of the [ResNeXt](https://paperswithcode.com/method/resnext) CNN architecture. It uses a "split-transform-merge" strategy (branched paths within a single module) similar to an [Inception module](https://paperswithcode.com/method/inception-module), i.e. it aggregates a set of transformations. Compared to a Residual Block, it exposes a new dimension,  *cardinality* (size of set of transformations) $C$, as an essential factor in addition to depth and width. 

Formally, a set of aggregated transformations can be represented as: $\mathcal{F}(x)=\sum_{i=1}^{C}\mathcal{T}_i(x)$, where $\mathcal{T}_i(x)$ can be an arbitrary function. Analogous to a simple neuron, $\mathcal{T}_i$ should project $x$ into an (optionally low-dimensional) embedding and then transform it.

ResNeXt Block

Aggregated Residual Transformations for Deep Neural Networks

Grouped-query attention

GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints

Please enter a description about the method here

Source	GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com