URL Normalization for Content Deduplication

Pinterest developed the Minimal Important Query Param Set (MIQPS) algorithm to dynamically learn and apply URL normalization rules. This system automatically identifies and strips non-essential URL parameters, significantly reducing redundant content ingestion and processing. MIQPS complements static rules by adapting to the long tail of domains with varying URL structures, using empirical testing based on page content identity to distinguish important parameters from noise. An anomaly detection layer guards against regressions by comparing new MIQPS computations against previous versions, rejecting updates if too many patterns become anomalous.