TechPrufer
Sign inGet started

Blogs›Cloudflare›Image Matching Algorithm Performance

Image Matching Algorithm Performance

Image Matching Algorithm Performance

2
posts
2019–2020

Cloudflare's image matching capabilities have evolved to address performance bottlenecks in high-dimensional data. Initial approaches using naive quadratic algorithms and SIMD optimizations (AVX2) showed limited gains due to memory bandwidth constraints. Advanced algorithms like VP-trees struggled with the 'curse of dimensionality' in 144-dimensional spaces. Subsequent efforts focused on optimizing brute-force methods by introducing a 'short distance' variation that computes a subset of dimensions. This has now been extended to include a CSAM Scanning Tool for all customers, leveraging fuzzy hashing (PhotoDNA) to identify visually similar images even when altered, providing automated flagging, blocking, and reporting of illegal content.

2020

Computing Euclidean distance on 144 dimensions

12/18/2020

This post details the engineering challenges and solutions for optimizing the performance of an image matching algorithm used in Cloudflare's CSAM image scanning tool. It analyzes the naive quadratic algorithm, its Euclidean distance calculation in 144 dimensions, and its performance limitations (18 qps). It then explores SIMD (AVX2) optimizations, achieving a 3x improvement but still bottlenecked by memory bandwidth (59 qps). The post discusses the failure of VP-trees due to the curse of dimensionality and introduces a 'short distance' variation that computes a subset of dimensions first, achieving 307 qps by leveraging the application's need to prove non-existence of matches within a threshold.

2019

Announcing the CSAM Scanning Tool, Free for All Cloudflare Customers

12/18/2019

Introduced a CSAM Scanning Tool for all Cloudflare customers, leveraging fuzzy hashing (PhotoDNA) to identify visually similar images. The tool automatically flags CSAM, blocks access with a 451 status code, and supports proper reporting. This builds upon existing image matching algorithm performance work by applying fuzzy hashing techniques to a critical security and content moderation use case.