Computing Euclidean distance on 144 dimensions

This post details the engineering challenges and solutions for computing Euclidean distance on high-dimensional data (144 dimensions) for an image matching algorithm used in Cloudflare's CSAM scanning tool. It explores naive brute-force approaches, the limitations of SIMD (AVX2) due to memory bandwidth, and the failure of space partitioning algorithms like VP-trees due to the curse of dimensionality. The post highlights a breakthrough with a 'short distance' variation that computes only a subset of dimensions initially, and further optimization by leveraging the application's characteristic of expecting few matches (proving non-existence of neighbors within a threshold) to achieve significant performance gains.