**Pythia** is a suite of decoder-only autoregressive language models all trained on public data seen in the exact same order and ranging in size from 70M to 12B parameters. The model architecture and hyperparameters largely follow GPT-3, with a few notable deviations based on recent advances in best practices for large scale language modeling.

**Stochastic Weight Averaging** is an optimization procedure that averages multiple points along the trajectory of [SGD](https://paperswithcode.com/method/sgd), with a cyclical or constant learning rate. On the one hand it averages weights, but it also has the property that, with a cyclical or constant learning rate, SGD proposals are approximately sampling from the loss surface of the network, leading to stochastic weights and helping to discover broader optima.

Stochastic Weight Averaging

Averaging Weights Leads to Wider Optima and Better Generalization

Pythia

Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling

An oscillatory function defined as $x \cdot cos(x)$ that reports better performance than Sigmoid, Mish, Swish, and ReLU on several benchmarks.

Source	Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com