**ResNet-D** is a modification on the [ResNet](https://paperswithcode.com/method/resnet) architecture that utilises an [average pooling](https://paperswithcode.com/method/average-pooling) tweak for downsampling. The motivation is that in the unmodified ResNet, the 1 × 1 [convolution](https://paperswithcode.com/method/convolution) for the downsampling block ignores 3/4 of input feature maps, so this is modified so no information will be ignored

**CANINE** is a pre-trained encoder for language understanding that operates directly on character sequences—without explicit tokenization or vocabulary—and a pre-training strategy with soft inductive biases in place of hard token boundaries. To use its finer-grained input effectively and efficiently, Canine combines downsampling, which reduces the input sequence length, with a deep [transformer](https://paperswithcode.com/method/transformer) stack, which encodes context.

CANINE

CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation

ResNet-D

Bag of Tricks for Image Classification with Convolutional Neural Networks

**AutoTinyBERT** is a an efficient [BERT](https://paperswithcode.com/method/bert) variant found through neural architecture search. Specifically, one-shot learning is used to obtain a big Super Pretrained Language Model (SuperPLM), where the objectives of pre-training or task-agnostic BERT distillation are used.  Then, given a specific latency constraint, an evolutionary algorithm is run on the SuperPLM to search optimal architectures. Finally, we extract the corresponding sub-models based on the optimal architectures and further train these models.

Source	Bag of Tricks for Image Classification with Convolutional Neural Networks
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com