Automatic rule backtesting with large quantities of data

This post details the development of a cloud-agnostic, Spark-based data pipeline and web portal for rule backtesting. The system allows analysts to replay existing or proposed rules against historical data, define custom metrics and dimensions, and manage jobs from a rule engine UI. It leverages SQS, Lambda, and EMR for job orchestration and execution, with historical events stored in S3 in Parquet format. The architecture aims to improve the accuracy and efficiency of rule performance analysis, reducing reliance on manual processes and shadow mode deployments.