**Switchable Atrous Convolution (SAC)** softly switches the convolutional computation between different atrous rates and gathers the results using switch functions. The switch functions are spatially dependent, i.e., each location of the feature map might have different switches to control the outputs of SAC. To use SAC in a detector, we convert all the standard 3x3 convolutional layers in the bottom-up backbone to SAC.

**LeCun's tanh** is an activation function of the form $f\left(x\right) = 1.7159\tanh\left(\frac{2}{3}x\right)$. The constants were chosen to keep the variance of the output close to 1.

Lecun's Tanh

DetectoRS: Detecting Objects with Recursive Feature Pyramid and Switchable Atrous Convolution

**1-bit Adam** is a [stochastic optimization](https://paperswithcode.com/methods/category/stochastic-optimization) technique that is a variant of [ADAM](https://paperswithcode.com/method/adam) with error-compensated 1-bit compression, based on finding that Adam's variance term becomes stable at an early stage. First vanilla Adam is used for a few epochs as a warm-up. After the warm-up stage, the compression stage starts and we stop updating the variance term $\mathbf{v}$ and use it as a fixed precondition. At the compression stage, we communicate based on the momentum applied with error-compensated 1-bit compression. The momentums are quantized into 1-bit representation (the sign of each element). Accompanying the vector, a scaling factor is computed as $\frac{\text { magnitude of compensated gradient }}{\text { magnitude of quantized gradient }}$. This scaling factor ensures that the compressed momentum has the same magnitude as the uncompressed momentum. This 1-bit compression could reduce the communication cost by $97 \%$ and $94 \%$ compared to the original float 32 and float 16 training, respectively.

Source	DetectoRS: Detecting Objects with Recursive Feature Pyramid and Switchable Atrous Convolution
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com