TechPrufer
Get started
AI Engineer
AI Engineer logoAI Engineer · Feature trail

Agent Memory and Harness Engineering

36 videos · Sep 2026 – Oct 2026

What this trail covers

This post contributes to the understanding of agent memory and harness engineering by detailing how to give AI agents memory that learns and changes their behavior. It introduces a framework for agent memory that includes semantic, episodic, and procedural types, separating working memory from long-term memory. The read-write cycle for memory updates (read context, run, filter evidence for signal, write back) is explained, along with practical lessons on what to keep, caching strategies, and the necessity of human review for procedural changes. LangSmith and Context Hub are presented as tools for capturing, analyzing, and updating this context, enabling agents to improve over time.

All AI Engineer trails
  1. From Context to Memory: Your Agents Need a Real Memory Layer — Anders Swanson, OracleThis talk contributes to the field of agent memory and harness engineering by arguing that current approaches like context windows and scratchpads are insufficient for true agent memory. It proposes a systems-level solution centered around a dedicated memory layer, emphasizing typed memories (episodic, semantic, procedural, working state) and a multi-model database schema that supports hybrid search (vectors, text indexes, graph links, time, audit history). The talk details the memory lifecycle: formation (with pre-write redaction), recall (using hybrid search), evolution, and evaluation, all governed by specific policies. It also identifies key failure modes such as stale, poisoned, and conflicting memories, and stresses the importance of feedback loops for continuous improvement and reliability.
  2. Designing CLIs for Agents, Not Humans — Pedro Lopez, AirbyteThis post contributes to agent harness engineering by detailing the design and implementation of two distinct interfaces for AI agents to interact with company data via Airbyte's Context Store: a Multi-Cloud Platform (MCP) server and a Command Line Interface (CLI). For the MCP server, it highlights best practices such as limiting tool count with progressive discovery, using OAuth for authentication, supporting long-lived sessions, and the current limitations of URL elicitation. For the CLI, it emphasizes designing for agents rather than humans, advocating for JSON in/out, consistent noun-verb commands, shipping skills with the tool, and avoiding interactive prompts. The talk concludes by providing guidance on when to choose an MCP versus a CLI for agent integration.
  3. Building on the Codex Harness — Dominik Kundel, OpenAIThis post introduces the Codex App Server, an open-source JSON-RPC protocol that acts as a harness for the Codex AI model. It details how this server powers various clients (Codex app, IDE extensions) and allows for deep customization. Key contributions include: exposing over 120 client messages for control, enabling 'Sign in with ChatGPT' for user subscriptions, layering developer instructions on top of system prompts, implementing dynamic and deferred tools, and configuring per-thread model and sandbox settings. The talk also covers practical aspects like shipping the binary, generating bindings, and using the Python SDK, with a demonstration of Codex running within the game Doom, equipped with custom tools.
  4. Stop Prompting — Greg Pstrucha, SentryThis post contributes to agent harness engineering by arguing against re-prompting AI coding agents and advocating for codifying rules through linters, strict types, and policies. It highlights that agents lack persistent memory, leading to recurring errors. The talk introduces the concept of mining transcripts and PR reviews to derive these rules, providing examples from Sentry where custom lint rules keep APIs in sync with OpenAPI schemas and ensure agent skills remain accurate. It also discusses the use of a "Grumpy Engineer" review loop for qualitative judgment and warns against agents gaming metrics like test coverage.
  5. Brains vs Hands: How to Run AI Agents Safely in Production — Viren BaraiyaThis post introduces the concept of running AI agents in production by framing their harnesses as applications, akin to microservices, rather than simple components. It advocates for a strict separation between the LLM's planning ('brain') and the deterministic, durable execution engine ('hands'). The Orkes Conductor is presented as a solution for this, enabling agents to be scheduled, event-driven, and long-running. Key features highlighted include human approval gates, idempotent steps, recorded side effects, and the compilation of agent plans into durable workflows, effectively treating agent harnesses as late-bound sagas.
  7. We Mapped 115 Microservices for Our Coding Agents — Kamalakannan Nandagopal, PostmanThis post contributes to the field of agentic systems by detailing Postman's approach to enabling coding agents to operate effectively within a large-scale microservice architecture. The core contribution is the development of an API context graph, built from code and production telemetry, that maps services, endpoints, implementations, and dependencies. This graph serves as the grounding context for agents, overcoming limitations of stale documentation and limited agent memory. The talk highlights the success of this approach in API discovery, redesign, and impact assessment, with a notable failure due to graph staleness underscoring the importance of continuous updates. The work demonstrates that a significant portion of PRs (75%) involve API touchpoints, validating the need for such a context-aware agent system.
  8. Turning Agent Memory Into Skills That Work — Will Lyon, Neo4jThis post introduces the Neo4j Agent Memory Service (NAMS) as a solution to the 'amnesiac loop' of AI agents. It posits that retrieval alone is insufficient for actionable knowledge and proposes a context graph model for agent memory, encompassing short-term, long-term, and reasoning memory. Key components include entity resolution, shared ontologies, and decision traces for inter-agent learning. The core contribution is the distillation of 'skills' from this memory, defining skills as typed execution graphs with rigorous checks for grounding, coverage, and coherence, and demonstrating this process with a healthcare patient-intake skill.
  9. Your AI Agent Has No Nervous System — Matt Gibiec, DynatraceThis talk introduces the concept of an AI agent's 'nervous system' – real-time environmental context – as a critical missing piece for effective and safe AI agent operation. It highlights the problems arising from agents working 'blind' (loops, operational/security issues, unmeasurable value) and proposes Bluebox by Dynatrace as a solution. Bluebox provides observability for coding agents, enabling them to integrate production dependencies into local testing, detect issues early, suggest context-aware fixes, and be informed by real usage data before development, with human approval for final changes.
  10. Giving AI Agents Memory That Learns — Jake Broekhuizen, LangChainThis post details how to imbue AI agents with learning memory by moving beyond simple observability and logs. It categorizes memory into semantic, episodic, and procedural types, and distinguishes between working and long-term memory. A core contribution is the 'read-write cycle' for memory updates: agents read context, execute tasks, filter evidence for actionable signals, and then write updated context back. Practical lessons are shared on trace retention, caching strategies for the hot path, and the importance of human review for critical procedural memory changes. The talk uses a financial services agent example to illustrate how memory improvements can fix behavioral drift.
  11. Why Bigger Context Windows Won't Save Your Agent — Elizabeth Fuentes Leone, AWSThis post contributes to the understanding of agent memory and harness engineering by explaining why larger context windows are insufficient for agents dealing with large tool outputs. It details how models forget information in the middle of long contexts and proposes four strategies: externalize, select, compress, and isolate. The talk demonstrates these strategies using the Strands Agents framework, covering conversation managers (sliding-window and summarizing), various memory types (short-term, long-term, graph), memory pointers for large tool outputs, sharing pointers across agent swarms, limiting tool calls to prevent endless loops, and using async handles for slow MCP tools. It also highlights anti-patterns to avoid.
  12. From 36% to 100%: How Self-Improving Agents Write Their Own Skills — Rafal Wilinski, RunlayerThis post contributes to the field of AI agent development by proposing a system where agents can autonomously learn and generate new skills by distilling knowledge from their successful and failed execution traces. It highlights the limitations of current skill development (developer-centric, time-consuming, client-dependent) and introduces 'Skills Over MCP' (Multi-Cloud Platform) as a distribution layer. By analyzing agent traces, particularly failures, the system creates reusable skills that significantly improve task success rates (e.g., from 36% to 100%), fostering an organizational knowledge flywheel that retains expertise even as underlying models evolve.
  13. The 6 Pillars of an Agentic Harness for Production — Varun Krovvidi, Resolve AIThis post contributes to the understanding of agent memory and harness engineering by detailing a practical workflow for evaluating and shipping AI agents that work reliably in production. It emphasizes moving beyond 'vibes' to rigorous evaluation, instrumenting agents with observability tools (OpenTelemetry, OpenInference), and establishing clear success criteria. The workflow includes building agents with SDKs (Claude Agent SDK), implementing deterministic checks, creating custom actionability, and addressing production challenges like anchoring bias, context management, causal reasoning, guardrails, and learning loops. The Resolve AI harness provides model orchestration, context engineering, causal reasoning, governed actions, learning systems, and evals.
  14. The Self-Improving OSS Agent Stack — Marc Klingen, LangfuseThis post details how to automate the AI agent improvement loop, moving from manual trace analysis to AI-driven proposal of fixes, dataset maintenance, and evaluation criteria updates. It introduces the concept of an 'online + offline agent loop' that connects tracing and monitoring with datasets, experiments, and evals. The talk emphasizes the importance of owning trace data for agents to effectively read and learn from, and demonstrates a coding agent using Langfuse's skills to identify and fix internal jargon in a changelog writer, including back-testing improvements before shipping.
  15. How AI Agents Pay: Checkout in ChatGPT and Google AI Mode — Sam Parsons, PayPalThis talk details three distinct protocols and integration patterns for enabling AI agents to perform financial transactions: the Agentic Commerce Protocol (ACP) used with ChatGPT, the Universal Commerce Protocol (UCP) used with Google AI Mode, and a custom MCP app with an external checkout. It emphasizes the security benefits of tokenization, where sensitive card details are abstracted away from the agent and merchant's PCI scope. The talk also covers the trade-offs between in-agent checkout (offering a seamless user experience) and external checkout handoffs (providing merchants with greater control over the UI and user journey). Key integration points and the constraints applied to tokens (merchant, amount, currency, expiry) are explained.
  16. How Many Credentials Should Your AI Agent Have? Zero. — Jim Clark, DockerThis post contributes to agent harness engineering by detailing a strategy for AI safety through sandboxing and credential management. It proposes using MCP gateways as a central control point for agent sandboxes, defining the sandbox by the tools and context it has access to, rather than the harness itself. The core principle is that the right number of credentials for an agent sandbox is zero, with tools and permissions granted progressively via Cross App Access (XAA) with Okta, aligned with the specific intent of the task. Examples include a newsroom split into researcher, fact-checker, and publisher sandboxes, and a coding agent that only receives signing keys when committing code.
  17. Your LLM Judge Is a Confident Liar: Building Better Verifiers — BrowserbaseThis post contributes to agent evaluation by highlighting the limitations of LLM-as-judge verifiers on benchmarks like WebVoyager, which can overstate agent success. It introduces the 'Universal Verifier' which uses task-specific rubrics, ranks relevant screenshots as evidence, isolates errors, catches hallucinations, and distinguishes controllable from uncontrollable failures. This approach significantly reduced false positives and improved agreement with human judgments. The research also investigates the use of autoresearch to build verifiers, demonstrating its potential while acknowledging the continued importance of human oversight.
  18. From Vibes to Production: Evaluating and Shipping AI Agents That Work 201 — Laurie Voss, Arize AIThis post details a practical workflow for evaluating and shipping AI agents that work reliably in production, moving beyond manual trace review to automated problem detection and remediation. It introduces Arize Signal as a layer that groups recurring problems and suggests fixes, which can be translated into GitHub issues, evaluation datasets, new evaluators, or proposed pull requests. The demonstration shows how to add observability to an agent, inspect traces to diagnose issues like missing search filters, and use a coding agent to implement fixes. Regression evaluations are highlighted as a mechanism to prevent introduced bugs, and Signal itself is presented with its own tracing and evaluation suite, enabling a continuous improvement loop.
  19. From Raw Documents to AI-Ready Data — Leo Platzer & Jeff KossThis post contributes to agent harness engineering by demonstrating the generation of context files for coding agents using an SDK. It also significantly advances the understanding of data quality's impact on RAG performance by quantifying the negative effects of duplicates, conflicts, and staleness in large document corpora (80,000 SharePoint files). The talk quantifies that stale data can fill up to 80% of an agent's retrieved context and that cleaning this data roughly doubled recall in multi-hop RAG evaluations. It showcases AI-assisted taxonomies, metadata tagging with evidence and confidence scores, sensitive data detection, and quality checks for duplicates, conflicts, and freshness, all crucial for preparing unstructured data for AI applications.
  20. From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AIThis post contributes to agent memory and harness engineering by presenting a practical, step-by-step workflow for evaluating and shipping AI agents. It details the use of the Claude Agent SDK, OpenTelemetry, and OpenInference for instrumentation and tracing. Key contributions include: implementing a deterministic ticker check for initial evaluation, developing a custom actionability rubric with explicit criteria and focused evaluators, using human annotations to compare and refine correctness judges, and transforming failing traces into datasets for prompt revision and controlled experiments. The workflow emphasizes reading actual traces to define success and using production monitoring to guide coding agent fixes.
  21. Your AI Agent Is Confidently Wrong About Production — Willem Pienaar, ClericThis post contributes to agent memory and harness engineering by detailing the challenges of using AI agents for production debugging, where feedback loops are absent and environments are unpredictable. It identifies four common failure modes: mistaking symptoms for causes, lacking baseline knowledge, abstraction loss in sub-agent summaries, and anchoring on past incidents. The talk proposes grounding techniques such as generating parallel theories, computing baselines, using dependency graphs, incorporating timing, and propagating evidence. Crucially, it emphasizes verifying fixes against real outcomes to calibrate agent confidence, thereby enabling more reliable production debugging.
  22. Lifestyles of the AI-Native — Nick Nisi & Zack Proser, WorkOSThis post contributes to agent memory and harness engineering by detailing a practical workflow for evaluating and shipping AI agents that work reliably in production. It emphasizes defining completion with checks the system must satisfy, moving beyond simple task completion to verifiable outcomes. The workshop demonstrates voice coding with Handy and Claude Code, session tracking with Fleet, and the use of Git worktrees for parallel changes. It introduces concepts like goals with measurable stopping conditions, loops for repeating tasks within boundaries, and verification gates with hooks (linting, type checks, tests). A key contribution is the adversarial review process using a second model before a pull request reaches a human. The post also covers turning working sessions into scheduled tasks for recurring reports, dependency updates, and meeting preparation, and shows how to generate weekly summaries of engineering work and token usage to preserve a record of automated tasks.
  23. Agentic Search vs Vector Search for Coding Agents: We Ran the Eval — BraintrustThis post contributes to the field of AI agent evaluation by presenting a comparative study of agentic search (using tools like grep and find) versus vector search for coding agents. It details the methodology for building an evaluation dataset from real-world code fixes (Microsoft's TypeScript Go repo) and scoring agent performance using the repository's test suite. The key finding is that while both approaches achieved comparable accuracy in fixing bugs, vector search was approximately four times more expensive due to its lack of contextual understanding, leading to repeated searches and higher computational costs. The post also touches upon the importance of observability and the 'eval flywheel' for iterative improvement of AI agents.
  24. Agent Memory Is Solved. Agent Learning Isn't. — Karthik Ranganathan, YugabyteThis post contributes to the field of agent memory and harness engineering by highlighting the limitations of current agent architectures where context and reasoning are lost when passing work between agents. It introduces the concept of 'agent learning' as distinct from 'agent memory,' arguing that true learning requires shared, governed, and traceable context. The talk details how tuning a RAG pipeline improved faithfulness from 14% to 82% while reducing context size by 7x. It introduces Meko, an agent-native persistence layer, which enables resuming sessions across different agents (Claude, Codex) and promotes private memories into shared 'data packs.' This system aims to capture tribal knowledge, reduce token waste, and facilitate collective learning among AI agents.
  25. Agents That Write Their Own Tools at Runtime — Sandhya Subramani, AWSThis post introduces the Strands Agents SDK, a framework for building agents that can dynamically generate and utilize their own tools at runtime. It demonstrates meta-tooling capabilities using an editor, shell, and load_tool, along with a system prompt to guide tool creation. The talk showcases agentic patterns like swarm, graph, handoff, and workflow, including a demo of a self-assembling travel planner. Crucially, it outlines essential evals for self-modifying agents (goal success, tool choice, parameters, inter-agent flow) and guardrails (sandboxed execution, constrained permissions, observability) necessary for deploying such agents in production.
  26. We Built an AI Support Agent That Resolves 80% of Tickets — AssemblyAIThis post details the creation and implementation of an AI support agent, Joey, built using the Claude Agent SDK. Joey achieves an 80% end-to-end ticket resolution rate, a significant improvement over previous solutions. The technical contributions include the agent's architecture, which utilizes local markdown documentation, embeddings for retrieval, and agentic file search. The deployment strategy on Railway enables rapid iteration with 30-second fix shipping. Furthermore, the integration of AssemblyAI's Voice Agent API to provide Joey with real-time voice interaction (speech-to-text, LLM processing, text-to-speech) via a single WebSocket is a key technical advancement. The talk also highlights the FDE model's scalability challenges and the strategy of using escalations as a roadmap for agent improvement.
  27. I Built a Personal AI Agent on a Raspberry Pi — Jeremy Adams, Neo4jThis post details the creation of a wearable personal AI agent on a Raspberry Pi 4B, leveraging NanoClaw and Neo4j Agent Memory. It demonstrates building graph memory using the POLE schema (person, object, location, event) for capturing contextual information, integrating voice-to-text via a USB microphone and hardware button, and implementing an offline-first capture mode with a local Neo4j instance that syncs and enriches data in the cloud. The architecture involves WhatsApp for messaging, Claude for LLM processing, and Neo4j for persistent, queryable memory.
  28. Stop Rationing Tokens: Let the Harness Pick the Model — Kimchi by Cast AIThis post introduces Kimchi, an open-source coding harness designed to address the challenges of escalating AI agent costs and inefficient token rationing. The core contribution is the shift from a cost-per-token model to a cost-per-task model, where the harness intelligently selects the optimal model (proprietary or open-source) based on task outcomes. It presents internal results showing a 2.5x cost saving over three months with 300 employees. Key features demonstrated include Ferment for milestone-based, self-scoring autonomous coding runs that deploy to staging, Teleport for persistent agent sessions that survive user disconnection, and Studio for team collaboration with a shared Kanban board. The talk argues that traditional diff-based verification is insufficient for AI-generated code and emphasizes the importance of outcome-aware model switching and autonomous coding workflows.
  29. Your Coding Agent Is 6 Months Out of Date — Jakub Hojsan, ExaThis post contributes to agent harness engineering by detailing Exa's solution to the knowledge-cutoff gap in coding agents. It explains why simply adding a web search tool is insufficient and proposes giving agents explicit rules for when to search. The talk also introduces query-dependent highlights, a technique that reduces a large page to a few hundred characters with no added latency, and previews Exa Agent, demonstrating its capabilities in finding conference speakers. The core technical contribution is the development of a search system specifically for coding and code-review agents that overcomes model knowledge limitations.
  30. MCP Doesn't Suck. Your Agent Does. — Jan Čurn, ApifyThis talk argues that the perceived failures of the MCP protocol are actually due to naive agent harness implementations. It proposes three key fixes: sub-agents for managing complexity, progressive tool discovery to avoid context bloat, and Code Mode for better code generation. The talk also makes a case for using MCP for remote, authenticated access and CLIs for local access, demonstrating this with mcpc, an open-source universal CLI client for MCP that supports stdio and remote servers, OAuth, persistent sessions, grep-based tool discovery, async tasks, and x402. Connector Evals show mcpc outperforming raw MCP and CLIs.
  31. Stop Renting Your AI's Memory — Dylan Couzon, QdrantThis talk by Dylan Couzon from Qdrant argues for owning AI memory, contrasting it with renting inference. It breaks down memory into write, retrieve, and forget operations, emphasizing the superiority of retrieval over simply dumping data into prompts. A live, offline demo showcases a drone building its own searchable memory with sub-millisecond queries in a 15MB footprint using Qdrant Edge. The talk also touches on the CPU/RAM/disk model for LLMs, context, and memory, and introduces the concept of a shared 'hive mind' memory for agents with opt-in consent.
  32. Get Out of the Model's Way — Kevin Hou, Google DeepMindThis talk details the evolution of AI coding tools from autocomplete to agent teams, introducing Google DeepMind's Antigravity 2.0. The core principle is to "get out of the model's way" by enabling dynamic subagents (led by Gemini 3.5 Flash via /teamwork), sidecars for plugin extensibility (listening to webhooks, cron jobs, messages), and generative UI for dynamic status views. This approach enabled the creation of an OS kernel that runs Doom using 93 subagents in 12 hours for under $1,000. It also highlights automating 90% of eval analysis with 100 parallel hypothesis agents, emphasizing that these building blocks are not fixed and can be dynamically composed.
  33. World Models Need Causality, Not Pretty Pixels — Christopher Manning, Moonlake AIThis talk argues that current generative video models, while capable of simulating observations, lack the semantic understanding necessary for planning and true embodied intelligence. The speaker, Christopher Manning, proposes a shift towards building action-conditioned world models in code, derived from single photos or short videos. These models allow for direct manipulation of objects and can augment their knowledge by researching objects online. A key technique involves a loop inspired by Claude Code, which compares rendered simulations against reality to minimize the sim-to-real gap, with the ultimate goal of replacing thousands of hours of teleoperation with simulation. The talk also touches upon the historical evolution of AI, from early cybernetics and robotics to modern LLMs, framing embodied AI as the future North Star.
  34. From VLM/VLA's to Embodied Agents — Armen Aghajanyan, Perceptron AIThis post introduces Perceptron AI's approach to embodied agents, focusing on a perceptive objective that learns relevant visual tokens rather than treating all pixels equally. It addresses context bloat with data-sparse mixture of experts, allowing models to selectively process tokens. A key contribution is the discovery of new scaling laws where extensive video pretraining can substitute for significantly less teleoperation data, reducing training costs. The talk also previews an embodied foundation model trained on a petabyte of diverse data, capable of agentic tasks like detection and control, with an open release planned.
  35. You’re Not Thinking Big Enough: Rebuilding Food Systems with AI Agents — Cody Menefee, FirecrawlThis talk proposes an AI agent-based system to optimize agricultural practices, specifically rotational grazing for livestock. The core idea is to replace manual decision-making by farmers with an LLM that analyzes environmental data (drought conditions, grass height) and animal locations to suggest optimal paddock movements. The system aims to overcome blockers by scraping knowledge from farmer-generated content (YouTube, research papers) using Firecrawl, developing a vision layer for biomass and biodiversity assessment, and advocating for open collar hardware to integrate with GPS tracking systems. The ultimate goal is to increase the efficiency and sustainability of livestock farming by addressing the labor bottleneck.
  36. The Dark Arts of Skill Engineering — Paul Bakaus, Renaissance GeekThis talk introduces nine techniques for 'skill engineering' that go beyond traditional prompt engineering. These include: 1. Using two blind sub-agents (e.g., design director and linter) for synthesis and self-correction. 2. Forcing divergence by naming preferred options and then discarding them to avoid safe predictions. 3. Routing skills like a mixture of experts. 4. Giving skills memory. 5. Using scripts that emit instructions on stdout for more reliable execution. 6. Implementing hooks that block writes rather than just complaining afterward. 7. Live wiring browser interactions. 8. The 'it worked on my machine' problem, addressed by designing for the weakest model. 9. Recognizing that every model is overfitted differently and designing for the weakest model. The overarching theme is that if a gate can be skipped, it will be, emphasizing the need for robust harness engineering.
  37. Total Recall: Agent Memory and Harness Engineering — Ignacio Martinez, OracleThis talk presents a novel seven-layer harness engineering framework for agents, arguing that control and memory are paramount, not just the language model itself. It highlights the critical role of physical memory location, advocating for databases over files for transactional consistency in parallel agent execution. The concept of 'Umwelt' is introduced to define an agent's perceptual lens, and 'skill promotion' is proposed as a mechanism for workflow optimization and retirement. A hysteresis variable for controlling model patience is also detailed.