Make Your Website Conversational for People and Agents with NLWeb and AutoRAG
Aug 28, 2025Catarina Pires Mota, Gabriel Massadas, Nelson Duarte, Daniel Leal, Anni Wang
Summary
This post introduces the integration of AutoRAG with the NLWeb standard to enable conversational search on websites. It details the architectural changes within AutoRAG to support website crawling and indexing, including the introduction of JobManager and FileManager Durable Objects to handle large, dynamic data sources and improve scalability. The post also describes the deployment of a Cloudflare Worker that implements the NLWeb protocol, providing both a conversational UI and an MCP server endpoint for AI agents. This represents a significant expansion of AutoRAG's capabilities to treat websites as first-class data sources.
More from the Artificial Intelligence Integration & Impact trail
- Introducing Clef-omni with full multimodality, plus a faster Clef and a cheaper Clef-flash
- Introducing Web Search API via AI Gateway
- Introducing Clef: our open-source decision models, and new RL fine-tuning platform
- One year later: Sovereign AI and the fight for choice
- AI Search is now generally available