Agentic Inference Cloud

10 videos · Sep 2026 – Oct 2026

What this trail covers

This post contributes to the field of agentic inference by detailing how to build and deploy AI applications on cloud GPUs with an iterative, local-like development experience. It introduces Runpod Flash, a system that eliminates the Docker build-push-run loop by enabling direct Python execution on GPUs, supporting serverless queues, load-balanced endpoints, inter-endpoint calls, network volumes with warm caching, and live serverless for instant iteration. This allows developers to deploy and iterate on AI applications, including text and image generation, with a single command, significantly accelerating the development cycle.

All AI Engineer trails

Oct 2026The Autonomous Computer: Infrastructure for Computer Use Agents — Ang Li, SimularThis talk introduces the concept of 'autonomous computers' and the infrastructure required for computer use agents. It demonstrates an agent (Agent S) that can perform tasks by combining adaptable vision agents with reusable automation code, checking the environment at each step. The research trajectory includes visual grounding, multiple models, and parallel attempts, with a focus on reliability across repeated runs with varying prompts and environments. The infrastructure discussion extends from personal desktops to fleets of cloud computers, with examples of legacy form entry and Android interactions. Key challenges highlighted include making everyday computer use reliable, understanding screen content, verification loops, irreversible actions, OS restrictions, and adapting to application changes.
Oct 2026Burn Your Flags: Interactive CLIs for Agents — Mark Lummus & Navinkumar Patil, PayPalThis post challenges the traditional approach of adding more command flags for agent interaction with CLIs. It highlights that interactive workflows inherently contain guidance and guardrails that agents can follow. The talk introduces three methods for enabling agent interaction with CLIs: tmux sessions for managing terminal state, IPC side channels for structured JSON communication, and context injection to make options and constraints explicit. It also emphasizes the benefits of using less expensive models and developing deterministic parsers for improved agent performance and accuracy in interactive CLI environments.
Oct 2026AI Apps in a Flash: Ship to GPUs Without Docker — Dean Quiñanola, RunpodThis post introduces Runpod Flash, a system designed to streamline the deployment and iteration of AI applications on cloud GPUs. It eliminates the traditional Docker build-push-run loop by allowing direct Python execution on GPUs, mimicking a local development environment. Key features include live serverless for instant iteration, endpoint-to-endpoint calls for function-like interactions, network volumes with warm caching for persistent state, and a single-command deployment process that packages and serves the application without Docker. The talk demonstrates evolving a function from text to image generation, installing dependencies on the fly, and deploying to production, as well as a hackathon project showcasing fan-out experiments across endpoints.
Oct 2026AI Agents Don't Read Your Policy Docs. They Hit Your APIs — GraviteeThis talk demonstrates the critical need for infrastructure-level governance of AI agents, arguing that policies should be enforced at the gateway rather than within agent prompts. It showcases five live scenarios using a hotel-booking agent on the Gravitee Gateway with a Databricks backend, Sonnet, and Groq. Key demonstrations include blocking destructive tool calls (e.g., 'delete all bookings'), preventing data exfiltration, rate-limiting runaway requests, utilizing semantic caching for repeated queries, and rejecting oversized prompts before they reach the LLM. The talk also touches upon telemetry, vendor decoupling, and trimming Multi-Cloud Platform (MCP) tool lists, highlighting the limitations of relying solely on agent instructions for governance and the importance of infrastructure-based controls for cost management and security.
Oct 2026Zero Cold Starts: Serverless AI Agents on Akamai Functions — Thorsten HansThis post contributes to the field of agentic inference by demonstrating the use of WebAssembly (via Spin) on a globally distributed serverless platform (Akamai Functions) to eliminate cold starts for AI agents and MCP servers. It showcases the development workflow using wasmcp for scaffolding and composition in TypeScript and Rust, testing with MCP Inspector, and one-command global deployment. The talk also touches upon using the Vercel AI SDK for agent creation and memory snapshots for maintaining application state.
Advertisement
Sep 2026Why LLM Recommenders Will Be AI's Biggest Consumer App — Devansh Tandon, MetaThis talk introduces the concept of LLM-native and agentic recommenders, proposing a three-step recipe for building them: 1. Tokenize content with semantic IDs (e.g., a Reel into ~10 tokens). 2. Pre-train the model to be bilingual in English and the catalog of semantic IDs. 3. Post-train the model as an LLM re-ranker with readable chain-of-thought reasoning. It also demonstrates steerable feeds (like Instagram's Your Algorithm) and explains the cost advantages of feed-based engagement over generative AI chat apps, positioning LLM recommenders as a significant future consumer AI application.
Sep 2026The Frontier AI Inference Cloud for Agents — Byung-Gon (Gon) Chun, FriendliAIThis post introduces FriendliAI's approach to building an inference cloud for AI agents. Key technical contributions include: 1. Prefix Caching: Computing shared prefixes of agent steps only once to avoid redundant computation. 2. Hierarchical KV Cache: Utilizing GPU, host memory, and disk for KV caching to manage large contexts efficiently. 3. Cache-Aware Routing: Directing agent requests to replicas that already hold the necessary prefix, preserving locality. 4. Agent-Aware Scheduling: Recognizing that individual calls are part of a larger agent program to optimize execution. These techniques are presented as a new paradigm for agentic inference, contrasting with traditional chat-based inference economics.
Sep 2026Large clusters for small models — Daniel Svonava, SuperlinkedThis post details how Superlinked addresses the bottleneck of serving many small models by inverting the routing model to a pull-based system, doubling cluster throughput, and using autoresearch to tune model configurations, including an 80-cent LoRA that improved retrieval on German legal text by 18%. It highlights the challenges of untuned open-source serving tools, stale worker state in top-down routers leading to low utilization, and the velocity killer of frequent LoRA/fine-tune conversations. The solution involves a gateway annotating requests, dropping them into a shared queue, and workers pulling tasks to form their own batches. A Rust sidecar abstracts adapters over three runtimes.
Sep 2026Vertical Mobility: Inference from MVP to Trillion-Parameter Workloads — Sitanshu Gupta, CoreWeaveThis post details CoreWeave's approach to serving diverse inference workloads, from MVP to trillion-parameter models. Key technical contributions include: prioritizing KV cache locality and least loaded GPUs in routing across heterogeneous hardware; offloading KV cache to high-bandwidth storage between chat turns to avoid re-paying for prefill; and leveraging 4-bit quantization and custom-trained speculators for improved acceptance lengths. The platform supports both serverless and dedicated consumption models, adapting to workload shapes like agentic, chat, voice/video streaming, and batch processing.
Sep 2026Operating Distributed Inference Systems at Scale — Nishant Gupta & Naman Ahuja, MetaThis post details Meta's approach to operating distributed inference systems at scale, highlighting the increasing complexity and coupling between different layers (routing, KV cache, batching, autoscaling). It introduces the concept of a request as a distributed transaction and the need for a dedicated control plane, analogous to Kubernetes for virtual machines. The discussion covers seven axes for an inference scheduler (GPU generation, memory headroom, KV cache state, weight warmth, tenant priority, latency budget, workflow context) and four optimization quadrants (avoid, share, move, delay). The primary metric is cost per successful task, and observability is framed as a control loop input to balance latency, cost, and throughput.