Using machine learning to detect bot attacks that leverage residential proxies

This post introduces Bot Management ML model v8, which specifically targets the detection of residential proxy abuse. It details the challenges posed by residential proxies, such as IP address rotation and masquerading as legitimate users. The post outlines the ML model training process, including data fetching from Clickhouse, orchestration with Airflow, and Catboost model training with hyperparameter tuning. It also describes the validation process, including the use of specializations and SHAP explainers. The core contribution is the detection methodology for residential proxies, which involves comparing network latency between direct and proxied requests and analyzing behavioral signals, while acknowledging the need to go beyond solely network characteristics due to internet diversity.