Gradient Sparsification for Distributed Optimization

Two Sigma is developing capabilities in distributed optimization, focusing on reducing communication overhead. This post introduces gradient sparsification, a technique that randomly drops coordinates of stochastic gradients and amplifies the remaining ones to ensure unbiasedness, thereby minimizing communication costs in large-scale machine learning applications. Experiments on logistic regression, SVMs, and CNNs validate the approach.