AI Security Vulnerability Benchmarking

7/27/2026

This post details how Sandstone built complex, multi-step agentic legal workflows using Vercel's AI SDK, showcasing its composability across various SDKs (Chat, Emulate, Agent Browser, Flags) for integrations with Slack, Okta, and local testing. It also highlights the use of Vercel Secure Compute for private data layer communication, simplifying enterprise security assurances.

7/27/2026

This post introduces DeepsecBench, a new benchmark for evaluating AI models' ability to find cybersecurity vulnerabilities. It details the benchmark's methodology, including the use of a golden set of findings, scoring based on recall-weighted F2, and measures to prevent models from memorizing solutions. The post presents performance data for various AI models, analyzing their scores against cost and time, and discusses strategies for building multi-model scanning programs. It also highlights the role of Vercel's AI Gateway in simplifying the execution of these scans.

5/4/2026

Introduces Deepsec, an open-source security harness that uses coding agents and LLMs (Claude, Codex) to find vulnerabilities in codebases. It details the architecture involving static analysis, agent investigation, revalidation, and enrichment. The post highlights its ability to run on user infrastructure or scale with Vercel Sandboxes, its customization via plugins, and its effectiveness in identifying security-sensitive areas and potential issues, including edge cases in authentication. It also discusses false positive rates and best use cases.

4/2/2026

This post details performance optimizations for Vercel Sandbox's filesystem snapshot restore process. Key improvements include parallelizing S3 downloads using the `Range` HTTP header and AWS Go SDK's `transfermanager`, fanning out decompression across multiple goroutines, and streaming download data directly into decompression to eliminate intermediate file writes. A local NVMe disk cache using LRU eviction was implemented, caching decompressed images to achieve a 95% cache hit rate and reduce boot times to be bounded only by microVM and container startup.

3/12/2026

This post details how Notion leverages Vercel Sandbox to run untrusted code for their Workers. Key technical contributions include the use of Firecracker microVMs for strong isolation, a firewall proxy for credential brokering to prevent secret exfiltration, dynamic network policies for runtime control, and filesystem snapshots for efficient state preservation and fast cold starts. The post also highlights the economic benefits of active-CPU billing combined with snapshots for predictable costs at scale.

1/30/2026

Vercel Sandbox is now generally available as the execution layer for agents, with the CLI and SDK open-sourced. It leverages Vercel's internal compute platform (Hive) powered by Firecracker microVMs for isolated, ephemeral, and fast-starting environments. Key features include sub-second starts, full isolation, ephemeral operation, snapshotting for instant environment restoration, and Fluid compute with Active CPU pricing. Teams like Roo Code and Blackbox AI are using Sandbox for AI agent development, enabling end-to-end testing, persistent collaboration via snapshots, and scalable, isolated execution of AI tasks.

6/9/2025

This post details security considerations for building AI agents, focusing on prompt injection, data exfiltration, and designing for failure. It emphasizes scoping tools to user authority, treating model output as untrusted, sanitizing markdown, and avoiding secrets in prompts. It also references `harden-react-markdown` and `markdown-to-markdown-sanitizer` for output sanitization.