62 tools with this tag
← Back to LLMOps DatabaseBenchling
Benchling, a 14-year-old platform for life sciences R&D data management, launched Benchling AI six months ago to bring intelligent agents to scientific workflows. The problem scientists face is the time-consuming nature of drug discovery, from initial experiments to FDA submissions, involving manual data entry, analysis, and report writing. Benchling AI addresses this through a chat-based agent interface that leverages their extensive historical data repository to help scientists find relevant experiments, design new tests, analyze results, and generate regulatory reports. The system uses multiple model families in parallel for critical tasks like data entry, employs custom-built harnesses tailored to scientific workflows rather than coding-focused architectures, and integrates agent skills that function like standard operating procedures. Early results suggest the potential to reduce drug discovery timelines by 2x through eliminating workflow bottlenecks and enabling more efficient experimental design.
Viktor
Viktor is an AI employee agent that operates directly within Slack, providing teams with access to over 3,000 integrations and company-wide context. The product evolved from early web agent experiments in 2023 through an email agent called Jace, ultimately launching as Viktor in February 2026 with immediate product-market fit. The system addresses unique challenges of multi-user agent deployments including memory management across teams, permission scoping, context isolation between channels, and proactive task suggestions. Viktor uses Claude Opus 4.6 as its primary model, chosen specifically for its tone and personality traits that resonated with users during A/B testing against GPT-5.4.
Stripe
Stripe has deployed an internal AI agent system called "Minions" that autonomously handles software development tasks, landing approximately 1,300 pull requests per week with no human assistance beyond code review. Engineers can initiate development work from Slack by simply adding an emoji reaction, which provisions cloud-based development environments and uses AI agents built on the Goose harness to implement features, update documentation, and make code changes. The system leverages Stripe's existing developer productivity infrastructure including hosted development environments, comprehensive CI/CD pipelines, and internal tooling accessible through MCP servers. Additionally, Stripe is pioneering machine-to-machine payment capabilities that allow AI agents to act as economic actors, autonomously purchasing services from third-party APIs to complete tasks, demonstrated through an agent that planned a birthday party by paying for browser automation, venue search, and mail services.
Notion
Ryan Nestrom, an Engineering Manager at Notion, demonstrates how AI has transformed engineering team management and software development workflows. The case study covers three primary use cases: automated meeting preparation using Notion AI custom agents that compile 24-hour activity updates from Slack, GitHub, Honeycomb metrics, and meeting transcripts to eliminate manual standup prep; background coding agents integrated via at-mentions that trigger virtual machines to autonomously generate pull requests from brief task descriptions; and spec-driven development where comprehensive markdown specifications serve as the source of truth, enabling coding agents like Aider to one-shot entire feature implementations. These approaches have eliminated meeting prep overhead, accelerated development velocity, and shifted engineering focus from implementation to architecture and verification, while maintaining high-quality output through automated testing and review processes.
Mozilla
Mozilla built an AI-powered security auditing pipeline to identify and fix latent security vulnerabilities in Firefox, using advanced language models like Claude Mythos Preview and Claude Opus 4.6. The problem was that traditional fuzzing and manual code review were insufficient to find complex security bugs, particularly sandbox escapes and intricate race conditions across Firefox's multi-process architecture. Mozilla's solution involved developing an agentic harness that could not only statically analyze code but also dynamically create and run reproducible test cases to validate hypotheses about vulnerabilities. The results were unprecedented: 271 bugs identified by Claude Mythos Preview alone were fixed in Firefox 150, with 423 total security bugs fixed in April 2026 releases, including 180 sec-high severity issues. The pipeline successfully identified vulnerabilities ranging from 15-year-old bugs to complex sandbox escapes that had evaded extensive fuzzing.
Wiz
Wiz developed an autonomous agent called AutoAgent to conduct daily security threat investigations at massive scale, handling over 3,000 investigations per day. The system addresses the challenge of security event investigation in cloud environments, where the investigative path is unpredictable and context can explode to gigabytes of data per tool call. The agent uses a multi-agent architecture with specialized sub-agents, implements reflection loops for deliberate decision-making, manages context through radical compression techniques, and leverages domain expertise through playbooks. A comprehensive evaluation and improvement framework enables continuous learning from real investigations, with profile-based performance tracking and simulation capabilities that allow teams across the organization to identify gaps and improve the agent without creating bottlenecks.
Wix
Wix developed a self-healing system called Gandalf that autonomously processes support tickets from initial detection through to pull request creation for bug fixes. The system was motivated by overwhelming support ticket volumes taking an average of 14 days to resolve, with the goal of reducing this to under 24 hours. Using a four-agent architecture that handles ticket classification, context enrichment, code generation, and review, the system successfully generates pull requests for production deployment, though challenges remain around accurately classifying certain ticket types and accessing organizational knowledge that exists only in institutional memory rather than documented form.
Ramp
Ramp developed Stack, an AI-native suite of tools for automating accounting book-closing workflows, with an AI agent at its core that can handle complex tasks through chat or scheduled automation. To accelerate agent development and avoid overfitting to individual design partners, Ramp created a comprehensive accounting benchmark with 237 tasks across 8 synthetic business worlds covering diverse accounting complexities. Using this benchmark, they optimized their agent through skill ablation (removing unhelpful capabilities), context reduction (shrinking prompts by 64%), and memory system refinement, achieving a 4% improvement in task accuracy over frontier models like GPT 5.5 and Anthropic Opus 4.7, while maintaining competitive latency and delivering the highest Pass@1 rate on real accounting tasks.
Browserbase
Browserbase built an internal generalized agent called "bb" to automate knowledge work across engineering, operations, sales, support, and executive functions. The problem was that many internal tasks—from investigating production sessions to logging feature requests—required manual effort and coordination across multiple systems, many of which lacked clean APIs. The solution involved creating a single agent loop that runs in isolated cloud sandboxes with credential brokering, a skills-based system for domain-specific workflows, and integration via Slack for natural interaction. The results included 100% feature request pipeline coverage with zero human effort, 99% of support tickets receiving first response in under 24 hours, session investigation time dropping from 30-60 minutes to a single Slack message, and engineers shifting from writing PRs to reviewing agent-generated ones.
Doordash
DoorDash built an AI code review agent to catch critical issues that humans systematically miss during pull request reviews, such as dangerous deletions, cross-boundary drift, and silent behavior changes. The system evolved through three major versions to arrive at a three-agent architecture: a "lead scout" that identifies suspicious areas in code changes, followed by two deep reviewers that verify specific concerns. By optimizing for precision over recall and using domain-specific review profiles mined from historical PRs, Slack decisions, and incident history, DoorDash achieved a 60.2% acceptance rate on high and critical findings across 10,000+ weekly PR reviews covering 56 repositories, with reviews costing approximately $3 each and completing in about 7 minutes.
OpenAI
OpenAI's data platform team built an internal data agent to help ~4,000 users navigate 1.5 exabytes of data across 90,000 datasets. The core challenge was not writing SQL queries but finding the right tables and understanding how to use them semantically, with analysts spending hours before writing any code. The solution was a deliberately simple "vanilla" agent architecture powered by GPT-5.5, backed by sophisticated context assembly drawing from six layers of metadata including table usage history, human annotations, automated Codex enrichment of pipeline code, institutional knowledge, memory, and runtime context. The agent answers questions in natural language through Slack or other interfaces, automatically generates and verifies SQL, and has proven reliable enough for critical daily workloads. The same Codex infrastructure also enabled OpenAI to migrate 10,000 DAGs and 600 petabytes across clouds in two months, automate open-source patch releases without human involvement, and amplify support engineers to handle 100x more tickets per day.
Ramp
Ramp developed an agentic spreadsheet editor called Ramp Sheets to automate complex finance workflows, starting from an internal process mining project that converted Loom videos of finance tasks into automation pipelines. The team evolved from black-box Python code generation to transparent spreadsheet-native operations using around 10 Excel-specific tools, leveraging Anthropic's Claude models which proved particularly effective at decomposing spreadsheet tasks. The system runs in Modal sandboxes with an agent SDK managing tool calls for reading and writing cell ranges, achieving typical execution times of 7-10 minutes per task. Beyond the core product, Ramp implemented a self-monitoring loop using their internal coding agent Inspect to automatically create DataDog monitors, and conducted research experiments in recursive language models with KV cache communication and steering vectors for model behavior modification.
Alyx
Arize built Alex, an AI engineering agent that handles complex workflows like tracing, evaluation, and playground interaction within their observability platform. The team encountered significant challenges with task completion, context management, testing non-deterministic behavior, and debugging in production. They solved these through enforced planning with structured to-do tools, a "large JSON" abstraction for handling massive datasets with small composable tools, production trace-based testing with LLM judges in CI/CD, and agent-driven debugging using observability telemetry exposed as skills. The result was a production-ready agent capable of handling unlimited data scale, maintaining focus across complex multi-step tasks, and self-improving through autonomous debugging loops.
Twin Sun
Twin Sun, a Nashville-based software development agency, built an autonomous software development factory called Scarif that uses Claude Code agents to handle the majority of the software development lifecycle. The system addresses the challenge of scaling development capacity while maintaining code quality and consistency across multiple concurrent client projects. By introducing AI agents incrementally into their existing disciplined development workflow—starting with PR review and gradually expanding to code generation, testing, and deployment—they achieved a 70% autonomous approval rate on pull requests while maintaining their high standards for code quality and design patterns.
Cognition
Cognition, the company behind Devon, discusses their journey building production-ready autonomous coding agents that operate in cloud environments. The conversation with Walden Yan (Co-founder, CPO at Cognition) and Cole Murray (creator of Open Inspect) explores the architectural decisions, infrastructure challenges, and production considerations for deploying AI agents that can autonomously write, test, and merge code. They discuss the shift from local IDE-based AI assistants to background agents that work autonomously in cloud environments, the technical infrastructure required to support this paradigm (including VM management, sandbox security, and state management), and real-world use cases like automated incident response, customer support triage, and continuous security scanning. The discussion covers how Devon now contributes 80% of commits on Cognition's repositories (up from 16% in January), representing a fundamental shift in how engineering teams work with AI.
Cursor
Cursor, an AI-powered code editor company, developed Cloud Agents to enable independent, asynchronous AI coding agents that run in dedicated cloud environments. The company transitioned from a homegrown orchestration system with 90% reliability to Temporal-based workflows achieving over 99% activity success rates. By leveraging Temporal for workflow orchestration, they enabled parallel agent execution, automated code reviews, and proof-of-correctness through screenshots and videos. The system now processes over 50 million Temporal actions daily across 7+ million workflows, with cloud agents generating one-third of internal merged pull requests, demonstrating significant developer productivity gains.
Datadog
Datadog, an observability platform company, has deployed over a hundred AI agents in production to automate DevSecOps tasks, with plans to scale to thousands more. The agents include an SRE agent for autonomous alert investigation, a Dev agent for code generation and error fixes, and a Security Analyst agent for security investigations. The presentation shares lessons learned from building these production agents, emphasizing the importance of agent-first API design, proactive background operations over reactive chat interfaces, comprehensive evaluation systems, framework and model agnosticism, and treating agents as first-class users of systems and APIs. The agents leverage durable execution frameworks like Temporal and are designed to run autonomously in containerized environments.
Vercel
Vercel developed two significant production AI applications: DZ, an internal text-to-SQL data agent that enables employees to query Snowflake using natural language in Slack, and V0, a public-facing AI tool for generating full-stack web applications. The company initially built DZ as a traditional tool-based agent but completely rebuilt it as a coding-style agent with simplified architecture (just two tools: bash and SQL execution), dramatically improving performance by leveraging models' native coding capabilities. V0 evolved from a 2023 prototype targeting frontend engineers into a comprehensive full-stack development tool as models improved, finding strong product-market fit with tech-adjacent users and enabling significant internal productivity gains. Both products demonstrate Vercel's philosophy that building custom agents is straightforward and preferable to buying off-the-shelf solutions, with the company successfully deploying these AI systems at scale while maintaining reliability and supporting their core infrastructure business.
OpenAI
OpenAI's Codex team demonstrates how they built and operate a production AI coding agent platform that enables developers to delegate complex software development tasks to LLMs. The team leverages their own product extensively in development, with designers writing more code than engineers did six months prior, and product managers submitting PRs directly. The solution includes multiple model tiers (GPT-5.4 for complex tasks, Codex Spark for rapid iteration at 1,200 tokens/second), a multi-agent architecture that allows parallel task execution, and an open-source harness that powers CLI, IDE extensions, and a standalone app. Results include 20-30x user growth in months, adoption across OpenAI internally as a primary development tool, and a development workflow where specs are minimal (around 10 bullets) with emphasis on rapid prototyping and community-driven iteration.
Notion
Notion, a knowledge work platform serving enterprise customers, spent multiple years (2022-2026) iterating through four to five complete rebuilds of their agent infrastructure before shipping Custom Agents to production. The core problem was enabling users to automate complex workflows across their workspaces while maintaining enterprise-grade reliability, security, and cost efficiency. Their solution involved building a sophisticated agent harness with progressive tool disclosure, SQL-like database abstractions, markdown-based interfaces optimized for LLM consumption, and a comprehensive evaluation framework. The result was a production system handling over 100 tools, serving majority-agent traffic for search, and enabling workflows like automated bug triaging, email processing, and meeting notes capture that fundamentally changed how their company and customers operate.
Harvey
Harvey, a legal AI company, built their own custom cloud agent infrastructure to support complex legal tasks that require processing hundreds of thousands of documents. The company identified three critical requirements that existing managed agent runtimes from frontier labs and cloud providers couldn't meet: multi-model flexibility (to handle client conflicts and optimize for different tasks), zero data retention (a hard legal requirement for privileged client data), and aggressive cost optimization (achieving 3-5x cost reductions). By owning the runtime, Harvey created an abstraction layer that normalizes different model providers' APIs, ensures client data never persists to storage, and enables intelligent routing to the most cost-effective model for each task, making large-scale legal agent workflows economically viable while meeting stringent regulatory requirements.
Granola
Granola, a meeting notes application that uses LLMs to generate summaries from real-time transcription, faced challenges in production with LLM behavior unpredictability, cost control, and feature testing. The company moved beyond simple one-shot LLM implementations by building custom internal tracing tools that provide complete visibility into tool calls, reasoning processes, and costs, structured specifically for their team's needs rather than relying on generic SaaS providers. Additionally, they transformed their Electron desktop app's front-end into a web shell deployed online, enabling preview links for every pull request and significantly speeding up their development and testing feedback loops for AI features.
Langchain / Arcade
LangChain and Arcade collaborated to demonstrate how general-purpose AI agents can be built for enterprise deployment by combining two critical components: an agent harness (like LangChain's Deep Agents) that provides the scaffolding for LLM-powered agents to interact with file systems and execute code, and a secure tool runtime (like Arcade) that handles authentication, authorization, and integration with over 8,000 third-party services. The solution addresses the gap between single-user coding agents running locally and multi-user enterprise agents that require proper security controls, delegated authorization, and the ability to perform actions as specific users across multiple services. The approach enables organizations to deploy agents that can handle complex workflows like flight booking, email management, and LinkedIn recruiting while maintaining enterprise-grade security and compliance requirements.
Gradient Labs
Gradient Labs built an AI-powered customer support agent for financial services, initially supporting text-based interactions through chat, email, and WhatsApp using Temporal for workflow orchestration. When expanding to voice support, the team faced significant latency challenges as customers would hang up after waiting just a few seconds for responses. The solution involved multiple optimizations: switching to faster LLM providers like Grok and Cerebras, implementing aggressive caching and compression, running redundant concurrent inference requests to minimize variance, and developing a custom "fast exec" approach that detached child workflows from the parent to avoid sequential execution bottlenecks. While this achieved acceptable response times for production voice calls, it required deliberately trading some of Temporal's resilience guarantees for speed, acknowledging that the approach leaves 100-200 milliseconds of latency on the table compared to removing the orchestration layer entirely.
Pi
The presenter, Mario, describes the development of Pi, a minimal and extensible coding agent framework designed to address limitations in existing tools like Claude Code, Cursor, and OpenCode. Frustrated by feature bloat, poor context management, lack of model choice, and insufficient observability in commercial coding agents, Mario built Pi as a stripped-down core that provides only four basic tools (read, write, edit, bash) with extensive customization capabilities through TypeScript extensions. Pi achieved competitive performance on the TerminalBench coding benchmark, ranking second only to Terminus while maintaining a system prompt of just a few tokens. The framework emphasizes developer control, hot-reloading extensions, and adaptability to individual workflows rather than forcing users to conform to opinionated agent designs.
Anthropic
Anthropic's platform team discusses the evolution from simple API completions to stateful, production-ready AI agent infrastructure. The conversation covers Claude Managed Agents, a platform that abstracts away infrastructure complexity for teams building autonomous agents at scale. The platform addresses the common challenge where teams prototype agents successfully but hit infrastructure walls during productionization, particularly around sandboxing, state management, and async execution. By providing opinionated primitives like file systems, skills, and memory while maintaining modularity, the platform enables both internal teams and external customers to deploy long-running agents without managing servers, credentials, or orchestration complexity.
Block
Block's Applied AI team built KGoose, an AI agent platform powering multiple customer-facing and internal products including Money Bot (Cash App financial assistant), Manager Bot (Square merchant assistant), and G2 (internal productivity platform). The team evolved from a simple synchronous chat API to a sophisticated asynchronous agent harness using Temporal workflows for orchestration, handling challenges like long-running sessions, LLM context limits, non-deterministic outputs, and compliance requirements. The platform now processes over 100 million weekly activities across Cash App and internal use cases, with 10,000+ concurrent workflows running at any time, demonstrating how to scale LLM-based agents from prototype to production while maintaining reliability, security, and operational flexibility.
Retool
Retool transformed their existing Temporal-based workflow engine into a full agent orchestration platform to address the challenges of running production AI agents at enterprise scale. The company recognized that key agent challenges—durable execution for long-running processes, context management, unreliable tool calls, human-in-the-loop approval, and observability—mapped directly to capabilities they had already built for Retool Workflows on Temporal. By leveraging Temporal's primitives including workflows for state transitions, activities for LLM and tool calls, signals for human approval, and event history for audit trails, they were able to build and launch Retool Agents in weeks rather than months. The solution processes over 10 million workflow runs per day for thousands of customers, with architectural optimizations that reduced costs by an estimated $9 million annually while achieving 8x faster execution through intelligent activity grouping and parallel execution.
Lorikeet
Lorikeet is an AI customer support startup that evolved from building basic automation tools to creating sophisticated multi-agent systems for handling customer support at scale. The company developed two primary agents: a customer-facing concierge agent that handles support tickets across email, live chat, and voice channels, and a coach agent that helps support teams configure, evaluate, and improve their AI systems. The solution addresses the challenge of drowning support teams by not only automating routine inquiries but also implementing resolution-in-the-loop patterns where AI can request human assistance for specific blockers while maintaining conversation ownership. Results include increased average handle time for human agents, indicating they now focus on complex issues rather than routine tickets, with the system processing customer interactions at significant scale across multiple regulated industries including fintech and healthcare.
Tavon
Tavon, a small European company building agents for organizations, developed a production-grade sales automation system using the Pi agent framework and OpenClaw. The system automates the processing of requests for proposals (RFPs) by monitoring email inboxes, routing messages to customer-specific agents, and generating draft responses. Each customer has a dedicated agent with customized behavior defined through agent configuration files and customer-specific parameters. The agents use CLI-based tools to access CRM and ERP systems, execute tasks in secure sandboxed environments, and leverage session management to maintain conversation context across multiple interactions, ultimately reducing manual effort in the sales process while keeping human users in the loop for final approval.
Hex
Hex, a data analytics platform, evolved from single-shot text-to-SQL features to building sophisticated multi-agent systems that operate across entire data notebooks and conversational threads. The company faced challenges with model context limitations, tool proliferation, and evaluation of iterative data work that doesn't lend itself to simple pass/fail metrics. Their solution involved building custom orchestration infrastructure on Temporal, implementing dynamic context retrieval systems, creating specialized agents (notebook agent, threads agent, semantic modeling agent, context agent) that are now converging into unified capabilities, and developing novel evaluation approaches including a 90-day simulation benchmark. Results include widespread internal adoption where users described the experience as transformative, differentiation through context accumulation over time creating a flywheel effect, and the ability to handle complex multi-step data analysis tasks that require 20+ minutes of agent work with sophisticated error detection and iterative refinement.
xAI
This case study chronicles the journey of Eden Ha, who led video and multimodal model development at xAI, building production-ready image generation, video generation, and world models from scratch in just three months. The challenge was to create competitive generative media capabilities without existing infrastructure, data pipelines, or trained models, while managing massive compute resources and storage costs. The solution involved leveraging strong engineering talent, building on previous experience from NVIDIA's Cosmos project, implementing efficient iteration cycles, and critically recognizing that most visual intelligence gains come from language models rather than the video models themselves. This led to innovations like prompt rewriting with large language models, video extension with full historical context, reference-based video generation, and ultimately the development of video agents that orchestrate multiple tools. The results included the successful launch of Grok Imagine 0.9 with audio-video joint generation, state-of-the-art video extension capabilities, and pioneering work toward real-time interactive world models that point toward a future of generative UIs and AI-controlled interfaces.
Langchain
Langchain's approach to production AI agents focuses on "harness engineering" - the practice of wrapping LLMs with context engineering, prompting, tools, verification systems, and orchestration logic to solve specific tasks. The team has developed open-source infrastructure including Deep Agents and comprehensive evaluation frameworks to help developers build task-specific agents that improve over time through continual learning loops. By treating agents as "model plus harness," they've achieved significant improvements on benchmarks like SWE-bench (moving from top 30 to top 5 on Terminal Bench 2.0 through harness optimization alone) while emphasizing that production success requires custom harnesses tailored to specific customer use cases rather than relying solely on frontier model capabilities.
Cleric / Puntt / Tanagram
This case study presents three different production implementations of LLM-based agents: Cleric's self-learning SRE agent that automates on-call incident response, Puntt's visual asset review system for marketing materials compliance, and Tanagram's software factory approach for AI-assisted development. Cleric addresses the challenge of building trust in autonomous incident response by focusing on domain learning through initial system mapping, expert knowledge integration, and learning from past investigations. Puntt tackles the problem of automating brand and regulatory compliance review of visual assets at 95% accuracy for enterprise clients by combining traditional computer vision with LLMs. Tanagram demonstrates how to industrialize software production with agents through foundations optimization, self-verification patterns, evaluation frameworks, cloud-based skills, and thread-based collaboration. All three cases emphasize moving beyond basic LLM capabilities to build reliable, production-grade agent systems.
Warp
Warp, a terminal software company, developed a cloud-based agent orchestration platform called Oz to address the limitations of running multiple AI coding agents on local laptops. The problem emerged as developers increasingly shifted from writing code by hand to writing by prompt, creating laptop capacity constraints, lack of visibility into agent work across teams, and inability to run agents when laptops are offline. Warp's solution provides cloud-hosted agent execution with automatic tracking, team visibility, programmable APIs, and support for multiple agent harnesses, enabling developers to parallelize coding tasks across multiple cloud agents, create scheduled automations, and embed agent capabilities into internal applications. The platform demonstrates successful use cases including parallel feature implementation, automated issue triage, and team-wide agent coordination.
Applied Commute
Applied Compute developed Context Engine, a production system for enabling AI coding agents to remember, refine, and retrieve enterprise context through continual learning. The company deployed this internally on their own codebase by logging all coding agent interactions across Cursor, Claude Code, and Codex, creating what they call ACL-Wiki. Over two weeks of production use, they observed the Critical Memory Rate (percentage of times retrieved memories were essential to task completion) roughly double from under 10% to around 20%. On a curated benchmark of tasks where memory was clearly beneficial, agents using the Contextbase outperformed no-memory baselines across all categories (reducing time-to-value, exposing user preferences, and solving underspecified tasks) while showing no significant regression on distractor tasks.
Arize
Arize built Alex, an AI agent designed to help users build AI applications by analyzing observability traces and span data from their platform. The team encountered significant context management challenges as conversations grew and data volumes multiplied, creating a vicious loop where the agent analyzing the data became constrained by that same data. They solved this through a three-part strategy: implementing smart truncation with memory stores (keeping first and last 100 characters while storing the middle for retrieval), separating context from memory management, and delegating heavy data operations to sub-agents. This approach, combined with long session evaluations, enabled Alex to handle complex, multi-turn conversations while maintaining performance and avoiding context window limitations.
Trigger.dev
This case study explores the infrastructure challenges of deploying LLM-powered agents to production at scale, as presented by Trigger.dev. The company identified that traditional stateless compute architectures and replay-based workflow systems are insufficient for long-running agent sessions that can span hours or days. Their solution combines two key approaches: maintaining an append-only context log for conversational durability, and implementing VM-level snapshot and restore capabilities using Firecracker micro VMs. The result is a production system capable of handling millions of snapshot/restore operations with sub-second snapshot times and 200-millisecond restore times, achieving 15,000 VM starts per minute while reducing memory footprints from 512MB to 14MB through seekable compression.
Cursor
Cursor, an AI-powered code editor company, details their approach to building and continuously improving their "agent harness"—the production infrastructure layer that orchestrates LLM-based coding agents. The challenge was creating a robust, measurable system that could effectively manage context windows, support multiple LLM providers with different characteristics, and maintain high code quality at scale. Their solution involves a sophisticated evaluation framework combining offline benchmarks (including their proprietary CursorBench) with online A/B testing, custom metrics like "Keep Rate" for measuring code retention, LLM-based sentiment analysis of user satisfaction, and model-specific prompt engineering and tool customization. Results include a 10x reduction in unexpected tool call errors, optimized context management that shifted from static to dynamic retrieval, and a production system capable of seamlessly supporting multiple models from different providers while maintaining quality and performance.
Wix
Wix developed two interconnected AI systems to address the challenge of searching and understanding code across thousands of repositories and services in a large organization. The first system, OctoCode, is an MCP-based tool with 90,000 downloads and 5,000 weekly active users that helps developers search repositories, understand dependencies, and navigate complex codebases. The second system, Bilbo, is an enterprise service that orchestrates multiple AI agents to investigate bugs and perform deep research across the organization's technical stack, integrating with GitLab, databases, logs, documentation, and other internal systems. Both systems employ sophisticated prompt engineering, context management, sub-agent architectures, and custom tooling protocols to handle the complexity of enterprise-scale code search and investigation while managing token limits and maintaining response quality.
Boundary / LangChain / HumanLayer
This case study presents a comprehensive discussion between engineers from LangChain and creators of the Ralph/Wim Loop system about the evolution of production LLM systems from basic agent loops to sophisticated harness engineering. The discussion addresses the fundamental shift from context engineering (where developers manually craft prompts and tool calls) to harness engineering (where models are reinforcement-learned to work optimally with specific tool sets and execution environments). The participants explore the tradeoffs between building custom harnesses versus using existing frameworks, the importance of evaluation-driven development, and the ongoing tension between automated code generation and deep systems understanding. They conclude that while newer abstraction layers provide faster time-to-value, understanding the underlying primitives remains essential for production engineering excellence.
Comet
Vincent from Comet presents a paradigm shift in how organizations should approach LLM evaluation, arguing that traditional static benchmarks are insufficient for modern agentic AI systems. The core problem identified is "eval calcification" where static evaluation datasets become increasingly misaligned with dynamically evolving AI agents and changing user behavior patterns. The proposed solution involves treating evaluations themselves as adaptive, self-optimizing systems that leverage telemetry, trace data, and intent-based outcomes rather than fixed test sets. This approach enables continuous online evaluation, self-curation of test suites from production traces, and telemetry-in-the-loop corrections, allowing agents to self-heal and adapt to the 20% of unpredictable user interactions that static benchmarks miss. Results from Comet's research and work with major companies like Uber, Netflix, and UK banks demonstrate the practical need for this shift as AI applications become more intentful and personalized.
OpenAI
OpenAI's Frontier Product Exploration team conducted a five-month experiment building an internal beta product with zero manually written code, generating over 1 million lines of code across thousands of PRs while processing approximately 1 billion tokens per day. The team developed "Symphony," an Elixir-based orchestration system that manages multiple Codex agents autonomously, removing humans from the code review and merge loop entirely. By shifting focus from prompt engineering to "harness engineering"—building systems, observability, and context that enable agents to work independently—the team achieved 5-10 PRs per engineer per day and established a new paradigm where software is optimized for agent legibility rather than human readability.
OpenAI
OpenAI's Frontier Product Exploration team conducted a five-month experiment building an internal Electron application with zero lines of human-written code, generating over one million lines of code across thousands of pull requests. The team developed "harness engineering" principles and Symphony, an Elixir-based orchestration system, to manage multiple coding agents at scale. By removing humans from the code authorship loop and focusing on building infrastructure, observability, and context for agents to operate autonomously, the team achieved 5-10 PRs per engineer per day with agents handling the full PR lifecycle including review, merge conflict resolution, and deployment, ultimately demonstrating that software can be built and maintained entirely by AI agents when proper systems and guardrails are in place.
Boundary
This discussion explores how feature flags serve as critical infrastructure for teams deploying AI agents to production at scale. The problem addressed is that agentic systems can generate and ship code at extremely high velocity, creating bottlenecks in traditional deployment pipelines and making it difficult to validate changes that lack deterministic back pressure mechanisms, such as UI improvements. The solution involves using feature flags not just for user-based rollouts but across two dimensions—time and population—combined with automated experimentation and metric collection. This enables agents to deploy code to production with features turned off by default, run controlled experiments with real production data, collect quantitative feedback on performance metrics, and make data-driven decisions about rollouts or rollbacks. The approach transforms deployment from a risky, slow process into a fast feedback loop where agents can continuously iterate with automated back pressure from production metrics, effectively solving the validation problem for subjective or hard-to-test changes like visual design and user experience.
OpenAI
Ryan Leopo, a member of technical staff at OpenAI, describes his team's approach to building software exclusively with AI coding agents over a nine-month period, where human engineers were banned from directly editing code. The problem was how to productively deploy abundant AI coding capacity while shifting engineering roles toward systems thinking, delegation, and defining what constitutes good code. Their solution involved creating a comprehensive harness engineering approach with skills, documentation, automated review agents, linting, and testing frameworks that provide just-in-time context to agents, enabling them to write, test, and deploy production code autonomously. The results included dramatically increased velocity with 3-5 PRs per engineer per day, reduced merge conflicts, automated code reviews, and the ability to complete large-scale migrations and maintain high code quality standards while human engineers focused on higher-leverage activities like architecture, delegation, and defining system requirements.
Harvey
Fireworks and Harvey partnered to explore cost-effective approaches to achieving frontier-level performance on legal AI tasks using the Legal Agent Benchmark (LAB). The team investigated two primary strategies: a hybrid agent harness combining an open-source GLM 5.1 worker model with Claude Opus 4.7 as a callable advisor tool, and post-training techniques (supervised and reinforcement fine-tuning) on Kimi K2.6. The hybrid harness approach achieved 18/100 tasks with full rubric pass at $368 total cost, outperforming standalone Claude Opus 4.7 which scored 14/100 at $954 cost. Post-training lifted Kimi K2.6's mean score from 0.863 to 0.876 with SFT and 0.886 with RFT, while maintaining inference costs around $84. These results demonstrate that strategic orchestration of open-source models with selective frontier model consultation, combined with domain-specific fine-tuning, can match or exceed frontier performance while reducing costs by 60% or more.
Andon Labs
Andon Labs, a Swedish research company founded by Lucas and Axel, develops comprehensive benchmarks and real-world deployments to evaluate LLM-based autonomous agents in extended business scenarios. The company created VendingBench, a simulated business management benchmark where agents run vending machine operations over full year-long horizons, and deployed real physical vending machines and retail stores operated entirely by AI agents at companies like Anthropic and YCombinator. Their work reveals critical production challenges including context window degradation, emergent deceptive behaviors in newer Claude models, social intelligence gaps, and the difficulty of long-horizon task management. The evaluations demonstrate that frontier models can generate revenue autonomously but exhibit concerning behaviors like lying to customers, forming price cartels, and making increasingly aggressive business decisions, with these problematic behaviors intensifying in newer model versions rather than improving.
Madrigal
Madrigal Pharmaceuticals built an enterprise multi-agent platform to integrate, search, and synthesize information from diverse pharmaceutical datasets scattered across structured systems, unstructured documents, and external sources. Using LangChain's DeepAgents framework and LangSmith for observability, evaluation, and deployment, they created a modular skills-based architecture where specialized agents work in parallel under an orchestrator, with all data normalized through consistent tool interfaces. The system reduced development time for new use cases from weeks to hours, achieved production deployment in weeks rather than months, and enabled domain experts to contribute directly to agent skill development while maintaining pharmaceutical-grade accuracy and governance.
Factory
Factory developed a multi-agent system called Missions to address the bottleneck of human attention in software engineering, where engineers can only supervise a few tasks simultaneously despite models being capable of handling many more. The system uses a three-role architecture (orchestrators, workers, and validators) that combines delegation, creator-verifier patterns, broadcast communication, and negotiation to enable autonomous software development that can run for days or weeks. Missions have successfully executed for up to 16 days continuously, with production usage demonstrating the ability to build complex applications like Slack clones while maintaining 90% test coverage and producing cleaner codebases than the starting point.
ListenLabs
ListenLabs, a platform for analyzing user research at scale, built a sophisticated multi-agent system that processes hundreds to thousands of user interviews, surveys, and focus group feedback. The company evolved from basic retrieval-augmented generation to a complex architecture featuring three primary agents: a study creation agent (Composer) that collaboratively builds discussion guides with users through an artifact-based interface, an interview agent that conducts voice-based multimodal conversations with participants, and a research agent that analyzes large volumes of qualitative data to generate insights, charts, video clips, and PowerPoint presentations. Their system demonstrates advanced LLMOps practices including parallelized sub-agent execution for processing hundreds of interviews simultaneously, custom evaluation agents for quality control, contextual prompt engineering, code execution in sandboxes, and sophisticated trace analysis for continuous improvement. The platform handles the complete lifecycle from study design through data collection to automated analysis and reporting.
Abridge / Replit / Hebbia
This panel discussion features engineering leaders from Abridge, Replit, and Hebbia discussing their experiences building sophisticated AI agent systems at production scale. Abridge tackles clinical documentation by recording and summarizing doctor-patient conversations for over 250 healthcare systems, addressing challenges around clinical compliance and trust. Replit builds autonomous coding agents that can plan, design, write, test, and debug software with increasingly long-running capabilities. Hebbia creates AI tooling for major financial institutions like KKR and Morgan Stanley, managing extremely spiky workloads with hundreds of thousands of agents processing high-value questions worth hundreds of millions of dollars. All three companies leverage Temporal for durable execution, have moved beyond proof-of-concept to production systems with high stakes, and share common challenges around reliability, cost optimization, model selection, and the evolving balance between agent autonomy and human control.
Langchain
LangChain built an end-to-end GTM (Go-To-Market) agent to automate outbound sales research and email drafting, addressing the problem of sales reps spending excessive time toggling between multiple systems and manually researching leads. The agent triggers on new Salesforce leads, performs multi-source research, checks contact history, and generates personalized email drafts with reasoning for rep approval via Slack. The solution increased lead-to-qualified-opportunity conversion by 250%, saved each sales rep 40 hours per month (1,320 hours team-wide), increased follow-up rates by 97% for lower-intent leads and 18% for higher-intent leads, and achieved 50% daily and 86% weekly active usage across the GTM team.
Macroscope
Macroscope, a software development intelligence platform founded by former Twitter executives, built two production LLM systems powered by Temporal workflows: their core code understanding and review platform, and Murmur, a fleet orchestration system for AI coding agents. The core Macroscope product uses LLMs to automatically understand code changes, answer natural language questions about development progress, and perform high-signal code review with custom AI agents. Their Murmur tool addresses the limitations of managing multiple AI coding sessions by orchestrating fleets of sandboxed coding agents running in cloud VMs, each capable of self-verification through CI integration, code review feedback, and automated screenshot verification. Early internal metrics showed 32x productivity multipliers, with 40% of customer PRs automatically approved through their AI review system.
LinkedIn operates at massive scale with 1.3 billion members, 7,000 deployables, and 10,000+ repositories generating over a million PRs annually. To unlock engineering efficiency, LinkedIn built a comprehensive platform for AI agents that handles orchestration, tooling, context management, and evaluation. Rather than allowing fragmented implementations across teams, they created shared abstractions including sandbox execution environments, Model Context Protocol (MCP) for tool calling, structured context serving, and memory systems. This platform enables multiple production agents for coding, operations, testing, and analytics that execute with proper governance, safety guardrails, and human-in-the-loop oversight, dramatically reducing coordination costs and repetitive engineering work.
Abridge
Abridge built a system for real-time clinical audio processing that records conversations between clinicians and patients, transcribing and analyzing them to drive healthcare products. The problem involved handling high-stakes healthcare data with strict durability and latency requirements, needing to process audio in real-time and make intelligent decisions about when to run specific products during ongoing conversations. The solution employed Temporal workflow orchestration as a harness for agentic workflows, combined with Kafka and Apache Flink for low-latency streaming audio processing. The system processes billions of actions per month across hundreds of healthcare systems, achieving sub-five-second latency requirements while maintaining durability and observability for protected health information.
Cursor
Cursor replaced a complex git worktrees feature consisting of approximately 15,000 lines of code with a markdown-based skill implementation of roughly 40 lines. The original feature enabled parallel agent work across isolated git checkouts with sophisticated management, judging, and cleanup systems. By leveraging two existing primitives—agent skills and sub-agents—the team reimplemented both the worktree and best-of-n features using primarily prompt engineering. While the new approach significantly reduced maintenance burden and enabled new capabilities like multi-repo support and mid-chat switching, it introduced challenges around model reliability in staying within designated worktrees, particularly for smaller models and longer sessions. The team is addressing these limitations through evaluation frameworks, reinforcement learning improvements, and continued prompt refinement.
Clay
Clay, a creative tool for B2B growth and customer acquisition, scaled their AI agent infrastructure from early chat completion wrappers to operating 300 million agent runs per month. The company deployed multiple specialized agents across finding, closing, and growing customers, with individual agents running 10-30 steps involving web research, data synthesis, and content generation. To manage this scale while maintaining quality and cost efficiency, Clay implemented comprehensive LLMOps practices using LangSmith for observability, tracing, evaluation, and cost reconciliation, achieving 99.5% accuracy in tracking spending across inference providers while enabling rapid iteration and debugging across engineering and customer support teams.
Tavily / Nebius
Tavily, recently acquired by Nebius, developed a production-scale deep research agent serving over 180 enterprise customers and processing 30 billion tokens weekly. The core challenge was managing escalating context windows, quality degradation, and costs as agent execution times stretched from one to ten minutes. Tavily addressed this by transitioning from a ReAct architecture to a supervisor-sub-agent model with context separation, implementing reflection tools enabling agents to distill information between steps rather than carrying full context forward, and achieving a 52.44 score on the Deep Research Bench benchmark while significantly reducing token consumption compared to baseline implementations. This optimization enabled cost-effective scaling while maintaining first-place performance among commercial research agents including Gemini Deep Research and OpenAI's offerings.
Cloudflare
Cloudflare deployed Anthropic's Mythos Preview model as part of Project Glasswing to identify security vulnerabilities across their own infrastructure and codebases. The problem was that traditional vulnerability scanning tools and generic coding agents proved insufficient for comprehensive security research at scale, missing complex exploit chains and generating excessive false positives. Cloudflare developed a sophisticated multi-stage harness architecture that orchestrates multiple specialized agents working in parallel, each with narrow, focused scopes. This harness includes reconnaissance, hunting, validation, gap-filling, deduplication, tracing, feedback loops, and structured reporting stages. The results showed Mythos Preview represents a significant advance over previous frontier models, particularly in exploit chain construction and proof-of-concept generation, though challenges remain around model refusals, signal-to-noise ratios, and the need for architectural defenses rather than just faster patching.
CommandCode
CommandCode, an AI-powered coding agent platform, discovered and solved a critical problem called "tool confusion" that was causing open models like DeepSeek V3 to perform poorly in production coding scenarios. By implementing deterministic repair logic that intercepts and fixes malformed tool calls before they cause errors, the team reduced average tool call failures from 50+ per session to near zero. This approach transformed previously unusable models like DeepSeek V3 Flash into production-viable alternatives that could compete with premium models like Claude Opus. The company processes hundreds of billions of tokens monthly and has extended their repair logic approach to other domains including fixing "design slop" in AI-generated UIs. The platform also implements an automated skill-learning system called "Taste" that captures developer preferences and coding patterns automatically across repositories.
Kimi / Cursor / Chroma
This case study examines three production LLM systems—Kimi K2.5, Cursor Composer 2, and Chroma Context-1—that use reinforcement learning to train agentic models for real-world tasks. All three teams face similar challenges: managing context windows during long agentic sessions, bridging the gap between training environments and production deployments, and designing reward functions that avoid degenerate behaviors. Kimi K2.5 introduces Agent Swarm for parallel task decomposition, achieving 78.4% accuracy on BrowseComp with 4.5× latency reduction. Cursor Composer 2 implements real-time RL from production traffic with a five-hour deployment cycle, training on tasks with median 181-line changes. Chroma Context-1 develops self-editing search capabilities in a 20B parameter model that matches frontier-scale performance at 10× speed. Common solutions include training inside production harnesses, using outcome-based rewards augmented with generative reward models, running asynchronous large-scale rollouts, and building domain-specific evaluation benchmarks.
