The three types of AI bot traffic and how to handle them

This post categorizes AI bot traffic into three types: AI training crawlers, AI engine grounding bots, and AI referrals. It explains the value of allowing AI crawlers for discoverability and potential referrals, likening it to early SEO. The post advises a selective blocking strategy, recommending blocking sensitive routes like `/login` and `/admin` while allowing access to discovery content such as documentation and product pages. It highlights Vercel Firewall, Bot Protection, and BotID as tools to manage this traffic.