Gradient Sparsification for Communication-Efficient Distributed Optimization

Introduced gradient sparsification as a method to reduce communication overhead in distributed optimization by randomly dropping coordinates of stochastic gradients and amplifying the remaining ones to maintain unbiasedness. Proposed several fast algorithms for approximate solutions with theoretical guarantees for sparseness. Validated the approach on logistic regression, SVMs, and CNNs.