Using machine learning to detect bot attacks that leverage residential proxies

What this post added

Introduced Bot Management machine learning model v8, which specifically targets the detection of residential proxy abuse. This model analyzes over 46 million HTTP requests per second in real-time, leveraging features based on request fingerprints, behavioral signals, and global statistics. The training process involves fetching and preparing datasets from Clickhouse, using Catboost models, and rigorous validation including specializations for different traffic sources. The model's effectiveness is evaluated through reproducibility checks, feature distribution analysis, and deployment in shadow mode before full activation. The detection of residential proxies is achieved by comparing direct vs. proxied requests, analyzing network latency discrepancies, and identifying unique fingerprints from browsing agents to track bots regardless of IP source, while avoiding broad IP blocking.