AI Security Vulnerability Benchmarking

12 posts · Jun 2025 – Aug 2026

What this trail covers

Vercel Sandbox provides hard isolation via Firecracker microVMs, credential security through a firewall proxy, dynamic network policies, and filesystem snapshotting for fast cold starts. This enables Notion to offer a developer platform for third-party data syncing, custom automations, and AI agent tools. Performance optimizations have been applied to snapshot restore times, reducing p75 latency from over 40 seconds to under one second. The platform now enforces network boundaries for untrusted code execution, enhancing AI agent security.

All Vercel trails

Aug 2026How v0 authenticates to Snowflake without exposing the user's OAuth tokenIntroduced a Snowflake request proxy for v0 sandboxes, built on the Vercel Sandbox firewall, to securely authenticate AI-generated applications to Snowflake. This proxy resolves user credentials server-side and injects them only into designated authentication fields within requests, preventing exposure of user OAuth tokens to untrusted code. The solution handles different Snowflake authentication flows (SQL API, login requests) and implements a 'fail closed' security model to reject requests with potential credential misuse.
Aug 2026$1 million hacker challenge for Vercel SandboxThis post announces a public HackerOne program to test Vercel Sandbox isolation, offering up to $1,000,000 USD in bounties for researchers who can escape the sandbox boundary. It details the focus on compute and network boundary breaches and reiterates the security model of Vercel Sandbox, which uses Firecracker microVMs and host-side network controls. The program aims to proactively identify and fix vulnerabilities before attackers do.
Aug 2026Everything hackable will get hackedThis post details the use of AI models, specifically Kimi K3, for offensive cybersecurity research, including mapping guest-kernel attack surfaces, identifying privilege escalation paths, and developing fuzzers. It highlights the capabilities of frontier models like OpenAI's Sol 5.6 for defensive cybersecurity tasks and introduces Deepsec, an open-source security harness for large-scale code analysis. The post emphasizes the temporary defensive advantage defenders have and Vercel's initiatives to enhance security, such as making the full egress firewall in Vercel Sandbox available on the Hobby plan and launching a HackerOne program focused on Vercel Sandbox vulnerabilities, incentivizing AI-assisted research.
Aug 2026A sandbox without a network boundary is only half a sandboxThis post details Vercel's approach to network isolation for sandboxes, emphasizing the need for egress control in addition to compute isolation. It introduces the Vercel Sandbox firewall, which polices outbound TCP connections and DNS queries on the host outside the microVM. The firewall inspects TLS handshakes (SNI) and checks against domain and CIDR policies. It also supports selective TLS termination for header injection and request forwarding, allowing for credential injection at the network boundary and programmable policy enforcement via custom proxies. The post highlights that credentials are never stored within the microVM and are disposed of when the sandbox stops. It also emphasizes that these security features are a baseline product, available on all Sandbox plans.
Jul 2026DeepsecBench: evaluating model performance in finding cybersecurity vulnerabilitiesIntroduced DeepsecBench, a benchmark for evaluating AI model performance in finding cybersecurity vulnerabilities. The benchmark uses a secret codebase with a golden set of 231 human-judged findings, running each model three times and publishing median results. It provides recall, precision, cost, and total time metrics, combining recall and precision into a single score weighted by F2. The post details model performance, cost-effectiveness, and time efficiency, and discusses strategies for building scanning programs using multiple models. It also highlights the role of Vercel's AI Gateway in managing diverse model access for security scans.
Advertisement
Jul 2026Claude Managed Agents on Vercel's Agentic Infrastructure - Ship 26 NYCThis talk introduces Claude Managed Agents, which separate the 'brains' (agent loop managed by Anthropic) from the 'hands' (bash commands and file operations running in self-hosted sandboxes controlled by the user). This partnership with Vercel Sandbox allows for bringing agents to private data securely, with three ownership layers for control. A demo showcases an EHR analyzer running on Claude Opus 4.8 within these self-hosted sandboxes.
May 2026Introducing deepsec: The security harness for finding vulnerabilities in your codebaseIntroduces Deepsec, an open-source security harness that uses coding agents and LLMs (Claude, Codex) to find vulnerabilities in codebases. It details the architecture involving static analysis, agent investigation, revalidation, and enrichment. The post highlights its ability to run on user infrastructure or scale with Vercel Sandboxes, its customization via plugins, and its effectiveness in identifying security-sensitive areas and potential issues, including edge cases in authentication. It also discusses false positive rates and best use cases.
Apr 2026Optimizing Vercel Sandbox snapshotsThis post details performance optimizations for Vercel Sandbox's filesystem snapshot restore process. Key improvements include parallelizing S3 downloads using the `Range` HTTP header and the AWS Go SDK's `transfermanager` API, parallelizing decompression across multiple goroutines, and streaming S3 download streams directly into decompression to eliminate intermediate disk writes. Additionally, a local NVMe disk cache using LRU eviction was implemented to store decompressed `.img` files, significantly reducing restore times for frequently used snapshots and achieving a 95% cache hit rate.
Feb 2026Security boundaries in agentic architecturesThis post introduces a security architecture for agentic systems by defining four actors (Agent, Agent Secrets, Generated Code Execution, Filesystem) and their respective trust levels. It proposes security boundaries to mitigate risks like prompt injection and credential theft. Architectures discussed include zero boundaries (default), secret injection proxy, and separating agent compute from sandbox compute. The latter architecture places agent and generated code in separate security contexts, with generated code unable to access secrets directly, and recommends scoped tool invocations for agents.
Jan 2026Run untrusted code with Vercel Sandbox, now generally availableVercel Sandbox is now generally available as the execution layer for agents, with the CLI and SDK open-sourced. It leverages Vercel's internal compute platform (Hive) powered by Firecracker microVMs for isolated, ephemeral, and fast-starting environments. Key features include sub-second starts, full isolation, ephemeral operation, snapshotting for instant environment restoration, and Fluid compute with Active CPU pricing. Teams like Roo Code and Blackbox AI are using Sandbox for AI agent development, enabling end-to-end testing, persistent collaboration via snapshots, and scalable, isolated execution of AI tasks.
Jun 2025Building secure AI agentsThis post details security considerations for building AI agents, focusing on prompt injection, data exfiltration, and designing for failure. It emphasizes scoping tools to user authority, treating model output as untrusted, sanitizing markdown, and avoiding secrets in prompts. It also references `harden-react-markdown` and `markdown-to-markdown-sanitizer` for output sanitization.