TechPrufer
Get started
Harvey
Harvey logoHarvey · Feature trail

Cloud Agent Infrastructure

10 posts · Mar 2025 – Jun 2026

What this trail covers

Harvey's Cloud Agent Infrastructure, embodied by Spectre, has evolved into a collaborative cloud agent platform. Spectre provides a durable, isolated, and collaborative environment for executing agent runs, moving beyond local coding agents. It integrates with existing systems like GitHub, Datadoc, and Linear, offering reviewable artifacts and enabling cross-functional collaboration. Key technical primitives include durable run records, ephemeral workers, sandboxed execution, a robust harness for orchestrating agent runs, and a centralized model inference client library for managing AI model performance reliably across millions of daily requests. This library encapsulates multiple features and relies on a distributed key-value store, an in-house proxy service, and a custom-built model health tracker to enable scalable, fault-tolerant interaction with AI model inference servers and maximize their availability. The system implements active load balancing and monitoring of LLM resources across all environments to optimize utilization and maintain consistent performance and a reliable user experience. A centralized quota & rate limiting system using a Redis-backed approximate sliding-window token bucket algorithm prevents server degradation and handles bursty traffic. A thin proxy forwards model requests made outside of the Kubernetes cluster back through the cluster to the model servers, compatible with the OpenAI API spec, adding security and instrumentation. Granular observability and strict burn rate alerts around SLAs ensure rapid response to model-related incidents. Detailed accounting of every prompt and output token consumed is collected via an in-house telemetry events pipeline and exported to Snowflake.

All Harvey trails
  1. Why we Built our own Cloud Agent Infrastructure | HarveyThis post details the rationale and technical considerations behind Harvey's decision to build and maintain its own cloud agent infrastructure. Key contributions include: 1. **Multi-Model Flexibility:** Architectural design to abstract away differences in agent harnesses, tool-call formats, and execution sandboxes across various AI models, enabling dynamic routing for conflict resolution, quality optimization, and cost control. 2. **Zero Data Retention (ZDR):** Architectural commitment to ZDR by ensuring customer data is not written to durable application storage by default, with agent sandboxes designed for lifecycle-bound transient working disks that are automatically cleaned up. This addresses the critical requirement for privileged and confidential legal data. 3. **Cost Optimization:** Implementation of fine-grained control over model routing and execution sandboxes to enable routing tasks to the most efficient model (including self-hosted open-source models) that meets quality thresholds, leading to significant cost reductions. The post also highlights the importance of owning the runtime for legal-specific layers, multi-cloud resilience, data residency, and sovereign deployments.
  2. How Harvey Secures Embeddings at Scale | HarveyThis post details Harvey's architectural controls to prevent embedding reversal, a security threat to sensitive data. It introduces the concept of embedding reversal, explains its implications for legal data privacy, and outlines Harvey's defense philosophy. Key technical contributions include: dedicated, isolated vector database storage per workspace with separate collections and storage; treating embeddings as extensions of source data with customer-defined retention policies; governing access via infrastructure-as-code with version-controlled permissions and reviewed network rules; minimizing the working set for each request to limit blast radius; and integrating embedding access into the same monitoring fabric as the rest of the data plane for anomaly detection and continuous verification of controls. The post emphasizes that these controls are designed to be robust, future-proof, secure-by-default, and enterprise-grade.
  3. Building Spectre: Internal Collaborative Cloud Agent Platform | HarveyThis post details the architecture and design decisions behind Spectre, Harvey's internal collaborative cloud agent platform. It introduces core technical primitives: durable runs (the primary object of collaboration) and ephemeral workers (short-lived execution units), sandboxes as execution boundaries for security and correctness, a harness component that orchestrates models and tools into a durable and observable system, and the integration of collaboration surfaces (Slack, web run page) as part of the runtime. It also highlights the importance of scheduled runs using the same runtime as interactive work and emphasizes security through explicit boundaries.
  4. How Autonomous Agents Will Transform Legal | HarveyThis post details the evolution of Harvey's internal agent system, Spectre, into a collaborative cloud agent platform. It highlights Spectre's transition from a local coding agent to a system capable of autonomous decision-making based on incidents, bug reports, customer feedback, and Slack messages. The system now functions as a company world model, monitoring internal operations and triggering actions. The post emphasizes the shift in leverage from individual speed to organizational coordination, impacting review, prioritization, and design.
  5. Agents and Ethical WallsThis post details the critical challenge of enforcing ethical walls within autonomous AI agents in the legal domain. It highlights how agents, by accessing data directly, maintaining context across time, and operating at scale, fundamentally break traditional information governance models. The proposed solution involves a 'fail closed' approach, matter-centric product isolation, deep integration with existing conflicts and DMS infrastructure (like Intapp and iManage), and auditable logging to prove compliance. The core technical contribution is the architectural requirement for AI platforms to mirror firm-specific matter structures as hard security boundaries, ensuring agents only access data within their assigned matter's scope and that all actions are logged for auditability. This directly impacts the design and operation of the Cloud Agent Infrastructure by mandating these security primitives.
  7. 3 Principles That Helped us Scale Agent DevelopmentThis post details Harvey's adoption of an agent framework for Assistant development, focusing on three core principles: 'No Custom Orchestration' (using external SDKs like OpenAI Agent SDK instead of custom orchestration), 'Capabilities are Tool Bundles' (packaging multiple tools and sub-agents into reusable Tool Bundles that can inject instructions into the main system prompt), and 'Eval Gates on Capabilities' (requiring datasets and evaluators for Tool Bundles and system prompt upgrades to prevent conflicts and regressions). It highlights the shift from bespoke orchestration to a standardized agent framework to scale feature development and accelerate delivery.
  8. Resilient AI Infrastructure | HarveyThis post details the technical implementation of Harvey's centralized model inference client library, designed to manage AI model performance reliably across millions of daily requests. It introduces a distributed key-value store, an in-house proxy service, and a custom-built model health tracker for scalable, fault-tolerant interaction with AI model inference servers. Key technical contributions include active load balancing and monitoring of LLM resources, a weighted selection process for healthy model deployments based on SLIs (latency, success rate), and a centralized quota & rate limiting system using a Redis-backed approximate sliding-window token bucket algorithm for handling bursty traffic. A thin proxy compatible with the OpenAI API spec is described for forwarding external requests and adding instrumentation. Granular observability with strict burn rate alerts and detailed token accounting via an in-house telemetry events pipeline are also highlighted.
  9. Introducing Agents in Harvey | HarveyThis post introduces the concept of 'agents' and 'workflows' as the next evolution of AI systems at Harvey, building upon 'models' and 'model systems'. It defines agents by their ability to plan, adapt, and interact. The primary contribution is the launch of 'Assistant Workflows', which are client-facing instantiations of agents designed to guide users through specific legal tasks. The post details the benefits of these workflows (personalized, guided, transparent, expert quality) and presents initial benchmark data comparing workflow outcomes against human lawyers. It also mentions the ongoing integration of agents into other parts of the platform like Assistant, Vault, and the broader platform.