Have it both ways: stay discoverable in search while disallowing AI training

Introduced the 'Disallow AI Training' setting within Bot Management, enabling granular control over AI training while preserving search discoverability. This setting leverages robots.txt and crawler classification (Search, Training, Agent) to differentiate crawler intent. The post details the 'Accountable' crawler designation, requiring mechanisms for opting out of AI training and summaries, URL-level visibility, and assurance that opting out does not impact search ranking. It outlines changes to existing settings ('Block', 'Block on pages with ads') and migration paths for existing customers, along with recommended configurations for new domains. Specific details on how Applebot, Googlebot, and Microsoftbot honor these new controls are provided.