**Zero Redundancy Optimizer (ZeRO)** is a sharded data parallel method for distributed training. ZeRODP removes the memory state redundancies across data-parallel processes by partitioning the model states instead of replicating them, and it retains the compute/communication efficiency by retaining the computational granularity and communication volume of DP using a dynamic communication schedule during training.

**Gaussian Processes** are non-parametric models for approximating functions. They rely upon a measure of similarity between points (the kernel function) to predict the value for an unseen point from training data. The models are fully probabilistic so uncertainty bounds are baked in with the model.

Image Source: Gaussian Processes for Machine Learning, C. E. Rasmussen & C. K. I. Williams

Gaussian Process

ZeRO

ZeRO: Memory Optimizations Toward Training Trillion Parameter Models

TAPEX is a conceptually simple and empirically powerful pre-training approach to empower existing models with table reasoning skills. TAPEX realizes table pre-training by learning a neural SQL executor over a synthetic corpus, which is obtained by automatically synthesising executable SQL queries.

Source	ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com