AI-Powered Developer Tools

4 posts · Nov 2024 – Jun 2026

What this trail covers

This thread tracks the evolution of AI-powered developer tools at Slack, aimed at saving engineering time and improving workflows. Initial posts detail the development of a Slack bot for resolving escalations, leveraging LLMs for post classification and internal knowledge integration via Amazon Bedrock Knowledge Bases. Subsequent posts cover the development of custom AI assistants within Slack, a web application for advanced AI agent interaction, IDE integrations for AI assistance, and the exploration of AI agents for security investigations, breaking down complex tasks into sequential model invocations with structured outputs and distinct persona agents (Director, Expert, Critic) to improve control and consistency.

All Slack trails

Jun 2026Agentic Testing: Where Agents Fit in the E2E Testing StackThis post details an experiment comparing agent-driven E2E testing (using Playwright MCP, Playwright CLI) against generated deterministic Playwright tests. It analyzes reliability, execution speed, cost (token usage and turns), and adaptability across different flows (Thread Reply, Search Discovery) and input formats (NL, YAML). Key findings include that agent-native execution models like MCP offer better reliability on complex flows than CLI or generated tests, generated tests are faster but less reliable on complex flows, and cost is driven by turn count and context growth rather than model reasoning. The post introduces the concept of 'tests enforce journeys' vs. 'agents verify goals'.
Apr 2026Managing context in long-run agentic applicationsThis post details the technical implementation of context management for long-running agentic applications. It introduces three complementary context channels: Director's Journal (structured working memory with decision, observation, finding, question, action, hypothesis entry types), Critic's Review (annotated findings report with credibility scores), and Critic's Timeline (consolidated chronological findings). It also describes the Critic's tools (get_tool_call, get_tool_result, get_toolset_info, list_toolsets) for inspecting evidence and a five-level credibility scale (Trustworthy, Highly-plausible, Plausible, Speculative, Misguided) for scoring findings.
Dec 2025Streamlining Security Investigations with AgentsThis post details the development of an AI agent system for security investigations. It describes the transition from a single-prompt prototype to a structured, multi-agent approach. Key technical contributions include breaking down investigations into sequential model invocations, using structured outputs with JSON schemas for predictable agent behavior, and designing a persona-based architecture with Director, Expert (Access, Cloud, Code, Threat), and Critic agents. The 'knowledge pyramid' concept is introduced to strategically use different model costs. The service architecture includes a Hub, Workers, and a Dashboard for real-time observation and management. An example report highlights emergent behavior in discovering credential exposure.
Nov 2024Empowering Engineers with AIThis post details the implementation of an AI bot for resolving escalations in Slack channels, using LLMs for post classification into six categories (ANNOUNCEMENT, HOW_TO_REQUEST, HUMAN_ACTION, PULL_REQUEST_REVIEW, SERVICE_ISSUE_REQUEST, UNKNOWN) and Amazon Bedrock Knowledge Bases to import technical documentation, historical channel posts, and Canvas content. It measures 30% positive user ratings and estimates 3,000+ hours saved. It also describes custom AI assistants in Slack (14% internal LLM traffic), a web app for advanced agent interaction (37% internal LLM traffic), and IDE integrations for AI assistance. A unified LLM platform is detailed with features like multi-LLM access, a defined schema, a derived data system for metrics, and knowledge base APIs. Tools like CLI tools, an 'Evaluate' playground, and a 'Compare' tool are also described to assist in AI development efforts. The post estimates 10,000+ engineering hours saved by year-end, accounting for information discovery, documentation reading, context switching, comprehension, translation, and hallucination time.
Advertisement