**Inverse Square Root** is a learning rate schedule 1 / $\sqrt{\max\left(n, k\right)}$ where
$n$ is the current training iteration and $k$ is the number of warm-up steps. This sets a constant learning rate for the first $k$ steps, then exponentially decays the learning rate until pre-training is over.

**AmoebaNet** is a convolutional neural network found through regularized evolution architecture search. The search space is NASNet, which specifies a space of image classifiers with a fixed outer structure: a feed-forward stack of [Inception-like modules](https://paperswithcode.com/method/inception-module) called cells. The discovered architecture is shown to the right.

AmoebaNet

Regularized Evolution for Image Classifier Architecture Search

Inverse Square Root Schedule

PoolFormer is instantiated from MetaFormer by specifying the token mixer as extremely simple operator, pooling. PoolFormer is utilized as a tool to verify MetaFormer hypothesis "MetaFormer is actually what you need" (vs "Attention is all you need").

Year	2000
Data Source	CC BY-SA - https://paperswithcode.com