**ConvBERT** is a modification on the [BERT](https://paperswithcode.com/method/bert) architecture which uses a [span-based dynamic convolution](https://paperswithcode.com/method/span-based-dynamic-convolution) to replace self-attention heads to directly model local dependencies. Specifically a new [mixed attention module](https://paperswithcode.com/method/mixed-attention-block) replaces the [self-attention modules](https://paperswithcode.com/method/scaled) in BERT, which leverages the advantages of [convolution](https://paperswithcode.com/method/convolution) to better capture local dependency. Additionally, a new span-based dynamic convolution operation is used to utilize multiple input tokens to dynamically generate the convolution kernel. Lastly, ConvBERT also incorporates some new model designs including the bottleneck attention and grouped linear operator for the feed-forward module (reducing the number of parameters).

**Channel-wise Cross Fusion Transformer** is a module used in the [UCTransNet](https://paperswithcode.com/method/uctransnet) architecture for semantic segmentation. It fuses the multi-scale encoder features with the advantage of the long dependency modeling in the [Transformer](https://paperswithcode.com/method/transformer). The [CCT](https://paperswithcode.com/method/cct) module consists of three steps: multi-scale feature embedding, multi-head [channel-wise cross attention](https://paperswithcode.com/method/channel-wise-cross-attention) and Multi-Layer Perceptron (MLP).

Channel-wise Cross Fusion Transformer

UCTransNet: Rethinking the Skip Connections in U-Net from a Channel-wise Perspective with Transformer

ConvBERT

ConvBERT: Improving BERT with Span-based Dynamic Convolution

**Switch Transformer** is a sparsely-activated expert [Transformer](https://paperswithcode.com/methods/category/transformers) model that aims to simplify and improve over Mixture of Experts. Through distillation of sparse pre-trained and specialized fine-tuned models into small dense models, it reduces the model size by up to 99% while preserving 30% of the quality gains of the large sparse teacher. It also uses selective precision training that enables training with lower bfloat16 precision, as well as an initialization scheme that allows for scaling to a larger number of experts, and also increased regularization that improves sparse model fine-tuning and multi-task training.

Source	ConvBERT: Improving BERT with Span-based Dynamic Convolution
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com