TechPrufer
Get started
Two Sigma
Two Sigma logoTwo Sigma · Feature trail

GPU Non-Determinism Workaround

1 post · May 2017

What this trail covers

Two Sigma is developing capabilities to ensure repeatability in machine learning experiments, particularly when using GPUs. This involves identifying and mitigating sources of non-determinism in frameworks like TensorFlow, enabling more reliable debugging and experimentation.

All Two Sigma trails
  1. A Workaround for Non-Determinism in TensorFlowThis post details a workaround for non-determinism in TensorFlow's GPU operations, specifically addressing the `reduce_sum` operation. The author demonstrates how to replace non-deterministic `reduce_sum` with a deterministic matrix multiplication approach, and how to adapt neural network architectures (e.g., by augmenting biases into the weight matrix) to leverage this deterministic operation for repeatable training results.