TechPrufer
Get started
Glean
Glean logoGlean · Feature trail

Enterprise Data Crawler

13 posts · Jul 2026 – Aug 2026

What this trail covers

Glean's enterprise search solution relies on a robust and scalable crawler as a foundational component. This crawler is designed to be quick, resilient, and flexible, handling various data sources and their unique API nuances. It incorporates sophisticated retry mechanisms for transient errors, tracks partial completions to avoid redundant work, and manages different work types (incremental, full, webhooks) to prevent data staleness. The system also prioritizes crawl tasks against API quotas and user identity schema for permissions-aware indexing. This post details optimizations to the identity crawl process, specifically for complex permission models like Salesforce, by concurrently executing non-API tasks and filtering stale users, leading to significant crawl time improvements.

All Glean trails
  1. How Glean uses Apache Beam Streaming to efficiently read a large corpus from MySQLThis post details the implementation of a streaming pipeline using Apache Beam with the Google Cloud Dataflow runner to efficiently read and process a large corpus from MySQL. It describes the creation of a custom UnboundedSource and UnboundedReader that queries a cache of documents maintained by a scanner object. The scanner issues parallel database reads, fetching documents in batches ordered by document ID to ensure each document is processed at most once per full scan. This approach replaces a batch job, enabling daily reprocessing of documents and improving scalability and cost-effectiveness.
  2. Make every business decision a better-informed one with Glean's Deep ResearchIntroduces the Deep Research agent, which enhances the enterprise data crawler's capabilities by orchestrating specialist sub-agents to perform parallel fact-finding across internal and external data sources. The agent plans research objectives, delegates tasks, analyzes patterns, identifies themes, and synthesizes insights into structured, citation-rich reports, effectively transforming raw data into actionable intelligence for decision-making.
  3. So, your AI vendor is building enterprise connectors — here are 9 questions to differentiate real enterprise AI systemsThis post outlines critical questions for evaluating enterprise AI data connectors, focusing on the technical differentiators of indexing vs. federated fetch, permission enforcement (ACLs, identity graphs, real-time updates), sensitive data protection, data update frequency (minutes, API rate limits, crawl strategies), data modeling complexity (e.g., Slack's structure), breadth and depth of indexing (content, activity, people data, dynamic algorithms), knowledge graph organization, index storage location (cloud-prem, zero data retention with LLMs), and scalable connector infrastructure (horizontally distributed, handling API quotas and data complexity). It emphasizes that robust connectors are the foundation for intelligent AI agents.
  4. Tripling streaming pipeline throughput by monitoring and debugging MySQL inefficienciesThis post details the debugging and optimization of MySQL read inefficiencies within Glean's streaming pipeline. It describes how monitoring revealed that large offsets in parallel MySQL queries led to significantly increased read latencies. The core technical contribution is the identification of MySQL's behavior with large offsets and the implementation of a query modification to force late row lookups, resulting in a tripling of pipeline throughput.
  5. Unlock rich insights across your entire business and BigQuery environments with Glean’s new structured query agentsIntroduces structured query agents that enable natural language querying of BigQuery data via Google's Conversational Analytics API, alongside direct SQL execution. These agents can be integrated with other Glean actions like notifications and workflow triggers.
  7. Using our identity schema to deliver personalized, permissions-aware resultsThis post details optimizations to the identity crawl process for connector applications, specifically addressing inefficiencies in complex permission models like Salesforce. The author implemented concurrent execution of non-API related tasks and filtering based on user domains to improve crawl speed and accuracy. This resulted in up to a 98% identity crawl time improvement for one customer and at least a 22% improvement for others.
  8. What is an intranet search engine, and what should modern enterprise search actually do?This post describes the fundamental architecture and operational principles of an enterprise search engine's crawler. It details the four-step process: crawling internal sources, indexing and tagging content, matching queries, and ranking results. Key technical aspects highlighted include the importance of cross-app connectivity and permission enforcement, the use of a self-learning language model for semantic understanding, and the integration of an Enterprise Graph for relationship-based ranking. It contrasts intranet search with internal/site search and enterprise search, emphasizing the need for comprehensive, permission-aware indexing. The post also outlines what modern intranet search should achieve: unifying knowledge, personalizing relevance, supporting AI answers, and enabling action from insights. It addresses common challenges like information overload, inconsistent tagging, incompatible systems, outdated content, and irrelevant results, proposing solutions rooted in advanced ranking and graph-based approaches.
  9. Why a unified document model is essential for a comprehensive workplace ChatGPTThis post elaborates on the necessity of a unified document model for enterprise generative AI, emphasizing how it structures diverse data sources (documents, messages, tickets, code, etc.) into a coherent format. It details the complexities involved in capturing user identities, permissions, relationships, and metadata across these sources. The core argument is that a well-designed unified model is crucial for building scalable and effective search and AI systems, enabling them to provide comprehensive and trustworthy answers by treating all data sources equally.
  10. Why building a workplace ChatGPT requires a great crawlerThis post details the engineering challenges and solutions for building a quick, resilient, and scalable enterprise data crawler. Key technical contributions include: defining uniform transient error handling, implementing retry mechanisms with partial-completion tracking for idempotent execution, managing diverse work types (incremental, full, webhooks) to prevent staleness, prioritizing crawl tasks against API quotas with weighted workloads and dynamic rate limit policy accommodation, and employing regular full crawls with validation to purge stale data safely. The system is designed for horizontal scalability to handle varied datasources, complex permission schemas, and API gaps.
  11. Enterprise context starts with indexing, but it doesn’t end thereThis post elaborates on Glean's indexing strategy, emphasizing that indexing is a foundational component but not the entirety of an enterprise AI system. It highlights the importance of breadth (reaching all relevant work locations) and depth (preserving metadata, structured entities, identities, permissions, and activity signals) in an enterprise index. The post introduces the concept of specialized indexes for different data types (company data, code, experts, profiles, calendars) and retrieval methods (semantic, lexical, structured) to address diverse retrieval problems. It also reinforces that a strong index improves token efficiency and yield by preparing context before it reaches the model. The post reiterates that Glean's system of context, comprising indexes, graphs, memory, data connectors, and tools, is crucial for AI to understand the enterprise and complete real work, going beyond mere indexing.