28 tools with this tag
← Back to LLMOps DatabaseSnorkel
Snorkel developed a specialized benchmark dataset for evaluating AI agents in insurance underwriting, leveraging their expert network of Chartered Property and Casualty Underwriters (CPCUs). The benchmark simulates an AI copilot that assists junior underwriters by reasoning over proprietary knowledge, using multiple tools including databases and underwriting guidelines, and engaging in multi-turn conversations. The evaluation revealed significant performance variations across frontier models (single digits to ~80% accuracy), with notable error modes including tool use failures (36% of conversations) and hallucinations from pretrained domain knowledge, particularly from OpenAI models which hallucinated non-existent insurance products 15-45% of the time.
Swedish Tax Authority
The Swedish Tax Authority (Skatteverket) has been on a multi-decade digitalization journey, progressively incorporating AI and large language models into production systems to automate and enhance tax services. The organization has developed various NLP applications including text categorization, transcription, OCR pipelines, and question-answering systems using RAG architectures. They have tested both open-source models (Llama 3.1, Mixtral 7B, Cohere) and commercial solutions (GPT-3.5), finding that open-source models perform comparably for simpler queries while commercial models excel at complex questions. The Authority operates within a regulated environment requiring on-premise deployment for sensitive data, adopting Agile/SAFe methodologies and building reusable AI infrastructure components that can serve multiple business domains across different public sector silos.
Australian Epilepsy Project
The Australian Epilepsy Project (AEP) developed a cloud-based precision medicine platform on AWS that integrates multimodal patient data (MRI scans, neuropsychological assessments, genetic data, and medical histories) to support epilepsy diagnosis and treatment planning. The platform leverages various AI/ML techniques including machine learning models for automated brain region analysis, large language models for medical text processing through RAG approaches, and generative AI for patient summaries. This resulted in a 70% reduction in diagnosis time for language area mapping prior to surgery, 10% higher lesion detection rates, and improved patient outcomes including 9% better work productivity and 8% reduction in seizures over two years.
Toyota
Toyota Motor North America (TMNA) and Toyota Connected built a generative AI platform to help dealership sales staff and customers access accurate vehicle information in real-time. The problem was that customers often arrived at dealerships highly informed from internet research, while sales staff lacked quick access to detailed vehicle specifications, trim options, and pricing. The solution evolved from a custom RAG-based system (v1) using Amazon Bedrock, SageMaker, and OpenSearch to retrieve information from official Toyota data sources, to a planned agentic platform (v2) using Amazon Bedrock AgentCore with Strands agents and MCP servers. The v1 system achieved over 7,000 interactions per month across Toyota's dealer network, with citation-backed responses and legal compliance built in, while v2 aims to enable more dynamic actions like checking local vehicle availability.
Wealthsimple
Wealthsimple, a Canadian FinTech company, developed a comprehensive LLM platform to securely leverage generative AI while protecting sensitive financial data. They built an LLM gateway with built-in security features, PII redaction, and audit trails, eventually expanding to include self-hosted models, RAG capabilities, and multi-modal inputs. The platform achieved widespread adoption with over 50% of employees using it monthly, leading to improved productivity and operational efficiencies in client service workflows.
Hexagon
Hexagon's Asset Lifecycle Intelligence division developed HxGN Alix, an AI-powered digital worker to enhance user interaction with their Enterprise Asset Management products. They implemented a secure solution using AWS services, custom infrastructure, and RAG techniques. The solution successfully balanced security requirements with AI capabilities, deploying models on Amazon EKS with private subnets, implementing robust guardrails, and solving various RAG-related challenges to provide accurate, context-aware responses while maintaining strict data privacy standards.
Datastax
Datastax developed UnReel, a multiplayer movie trivia game that combines AI-generated questions with real-time gaming. The system uses RAG to generate movie-related questions and fake movie quotes, implemented through Langflow, with data storage in Astra DB and real-time multiplayer functionality via PartyKit. The project demonstrates practical challenges in production AI deployment, particularly in fine-tuning LLM outputs for believable content generation and managing distributed system state.
Stripe
Stripe, processing approximately 1.3% of global GDP, has evolved from traditional ML-based fraud detection to deploying transformer-based foundation models for payments that process every transaction in under 100ms. The company built a domain-specific foundation model treating charges as tokens and behavior sequences as context windows, ingesting tens of billions of transactions to power fraud detection, improving card-testing detection from 59% to 97% accuracy for large merchants. Stripe also launched the Agentic Commerce Protocol (ACP) jointly with OpenAI to standardize how agents discover and purchase from merchant catalogs, complemented by internal AI adoption reaching 8,500 employees daily using LLM tools, with 65-70% of engineers using AI coding assistants and achieving significant productivity gains like reducing payment method integrations from 2 months to 2 weeks.
Contextual
Contextual has developed an end-to-end context engineering platform designed to address the challenges of building production-ready RAG and agentic systems across multiple domains including e-commerce, code generation, and device testing. The platform combines multimodal ingestion, hierarchical document processing, hybrid search with reranking, and dynamic agents to enable effective reasoning over large document collections. In a recent context engineering hackathon, Contextual's dynamic agent achieved competitive results on a retail dataset of nearly 100,000 documents, demonstrating the value of constrained sub-agents, turn limits, and intelligent tool selection including MCP server management.
Airtrain
Two case studies demonstrate significant cost reduction through LLM fine-tuning. A healthcare company reduced costs and improved privacy by fine-tuning Mistral-7B to match GPT-3.5's performance for patient intake, while an e-commerce unicorn improved product categorization accuracy from 47% to 94% using a fine-tuned model, reducing costs by 94% compared to using GPT-4.
Beekeeper
Beekeeper, a digital workplace platform for frontline workers, faced the challenge of selecting and optimizing LLMs and prompts across rapidly evolving models while personalizing responses for different users and use cases. They built an Amazon Bedrock-powered system that continuously evaluates multiple model/prompt combinations using synthetic test data and real user feedback, ranks them on a live leaderboard based on quality, cost, and speed metrics, and automatically routes requests to the best-performing option. The system also mutates prompts based on user feedback to create personalized variations while using drift detection to ensure quality standards are maintained. This approach resulted in 13-24% better ratings on responses when aggregated per tenant, reduced manual labor in model selection, and enabled rapid adaptation to new models and user preferences.
Anaconda
Anaconda developed a systematic approach called Evaluations Driven Development (EDD) to improve their AI coding assistant's performance through continuous testing and refinement. Using their in-house "llm-eval" framework, they achieved dramatic improvements in their assistant's ability to handle Python debugging tasks, increasing success rates from 0-13% to 63-100% across different models and configurations. The case study demonstrates how rigorous evaluation, prompt engineering, and automated testing can significantly enhance LLM application reliability in production.
Roots
Roots, an insurance AI company, developed and deployed fine-tuned 7B Mistral models in production using the vLLM framework to process insurance documents for entity extraction, classification, and summarization. The company evaluated multiple inference frameworks and selected vLLM for its performance advantages, achieving up to 130 tokens per second throughput on A100 GPUs with the ability to handle 32 concurrent requests. Their fine-tuned models outperformed GPT-4 on specialized insurance tasks while providing cost-effective processing at $30,000 annually for handling 20-30 million documents, demonstrating the practical benefits of self-hosting specialized models over relying on third-party APIs.
Kantar Worldpanel
Kantar Worldpanel, a market research company, needed to modernize their product description matching system to better link paper receipt descriptions with product barcode names. They leveraged Databricks Mosaic AI to experiment with various LLMs (including Llama, Mistral, and GPT models) to generate high-quality training data, achieving 94% accuracy in matching product descriptions. This automated approach generated 120,000 training pairs in just hours, allowing them to fine-tune smaller models for production use while freeing up human resources for more complex tasks.
Vannevar Labs
Vannevar Labs needed to improve their sentiment analysis capabilities for defense intelligence across multiple languages, finding that GPT-4 provided insufficient accuracy (64%) and high costs. Using Databricks Mosaic AI, they successfully fine-tuned a Mistral 7B model on domain-specific data, achieving 76% accuracy while reducing latency by 75%. The entire process from development to deployment took only two weeks, enabling efficient processing of multilingual content for defense-related applications.
DoorDash
DoorDash faced challenges in scaling personalization and maintaining product catalogs as they expanded beyond restaurants into new verticals like grocery, retail, and convenience stores, dealing with millions of SKUs and cold-start scenarios for new customers and products. They implemented a layered approach combining traditional machine learning with fine-tuned LLMs, RAG systems, and LLM agents to automate product knowledge graph construction, enable contextual personalization, and provide recommendations even without historical user interaction data. The solution resulted in faster, more cost-effective catalog processing, improved personalization for cold-start scenarios, and the foundation for future agentic shopping experiences that can adapt to real-time contexts like emergency situations.
QuantumBlack
QuantumBlack presented two distinct LLM applications: molecular discovery for pharmaceutical research and call center analytics for banking. The molecular discovery system used chemical language models and RAG to analyze scientific literature and predict molecular properties. The call center analytics solution processed audio files through a pipeline of diarization, transcription, and LLM analysis to extract insights from customer calls, achieving 60x performance improvement through domain-specific optimizations and efficient resource utilization.
eBay
eBay developed Mercury, an internal agentic framework designed to scale LLM-powered recommendation experiences across its massive marketplace of over two billion active listings. The platform addresses the challenge of transforming vast amounts of unstructured data into personalized product recommendations by integrating Retrieval-Augmented Generation (RAG) with a custom Listing Matching Engine that bridges the gap between LLM-generated text outputs and eBay's dynamic inventory. Mercury enables rapid development through reusable, plug-and-play components following object-oriented design principles, while its near-real-time distributed queue-based execution platform handles cost and latency requirements at industrial scale. The system combines multiple retrieval mechanisms, semantic search using embedding models, anomaly detection, and personalized ranking to deliver contextually relevant shopping experiences to hundreds of millions of users.
Meta / AWS / NVIDIA / ConverseNow
This panel discussion features leaders from Meta, AWS, NVIDIA, and ConverseNow discussing real-world challenges and solutions for deploying LLMs in production environments. The conversation covers the trade-offs between small and large language models, with ConverseNow sharing their experience building voice AI systems for restaurants that require high accuracy and low latency. Key themes include the importance of fine-tuning small models for production use cases, the convergence of training and inference systems, optimization techniques like quantization and alternative architectures, and the challenges of building reliable, cost-effective inference stacks for mission-critical applications.
Various (Alation, GrottoAI, Nvidia, OLX)
This panel discussion brings together experts from Nvidia, OLX, Alation, and GrottoAI to discuss practical considerations for deploying agentic AI systems in production. The conversation explores when to choose open source versus closed source tooling, the challenges of standardizing agent frameworks across enterprise organizations, and the tradeoffs between abstraction levels in agent orchestration platforms. Key themes include starting with closed source models for rapid prototyping before transitioning to open source for compliance and cost reasons, the importance of observability across heterogeneous agent frameworks, the difficulty of enabling non-technical users to build agents, and the critical difference between internal tooling with lower precision requirements versus customer-facing systems demanding 95%+ accuracy.
Zoro UK
Zoro UK, an e-commerce subsidiary of Grainger with 3.5 million products from 300+ suppliers, faced challenges normalizing and sorting product attributes across 75,000 different attribute types. Using DSPy (a framework for optimizing LLM prompts programmatically), they built a production system that automatically determines whether attributes require alpha-numeric sorting or semantic sorting. The solution employs a two-tier architecture: Mistral 8B for initial classification and GPT-4 for complex semantic sorting tasks. The DSPy approach eliminated manual prompt engineering, provided LLM-agnostic compatibility, and enabled automated prompt optimization using genetic algorithm-like iterations, resulting in improved product discoverability and search experience for their 1 million monthly active users.
Co-op
Co-op, a major UK retailer, developed a GenAI-powered virtual assistant to help store employees quickly access essential operational information from over 1,000 policy and procedure documents. Using RAG and the Databricks Data Intelligence Platform, the solution aims to handle 50,000-60,000 weekly queries more efficiently than their previous keyword-based search system. The project, currently in proof-of-concept stage, demonstrates promising results in improving information retrieval speed and reducing support center workload.
Capital One
Capital One developed enhanced input guardrails to protect LLM-powered conversational assistants from adversarial attacks and malicious inputs. The company used chain-of-thought prompting combined with supervised fine-tuning (SFT) and alignment techniques like Direct Preference Optimization (DPO) and Kahneman-Tversky Optimization (KTO) to improve the accuracy of LLM-as-a-Judge moderation systems. Testing on four open-source models (Mistral 7B, Mixtral 8x7B, Llama2 13B, and Llama3 8B) showed significant improvements in F1 scores and attack detection rates of over 50%, while maintaining low false positive rates, demonstrating that effective guardrails can be achieved with small training datasets and minimal computational resources.
Various
A tech company needed to improve their developer documentation accessibility and understanding. They implemented a self-hosted LLM solution using retrieval augmented generation (RAG), with guard rails for content safety. The team optimized performance using vLLM for faster inference and Ray Serve for horizontal scaling, achieving significant improvements in latency and throughput while maintaining cost efficiency. The solution helped developers better understand and adopt the company's products while keeping proprietary information secure.
Coinbase
Coinbase, a cryptocurrency exchange serving millions of users across 100+ countries, faced challenges scaling customer support amid volatile market conditions, managing complex compliance investigations, and improving developer productivity. They built a comprehensive Gen AI platform integrating multiple LLMs through standardized interfaces (OpenAI API, Model Context Protocol) on AWS Bedrock to address these challenges. Their solution includes AI-powered chatbots handling 65% of customer contacts automatically (saving ~5 million employee hours annually), compliance investigation tools that synthesize data from multiple sources to accelerate case resolution, and developer productivity tools where 40% of daily code is now AI-generated or influenced. The implementation uses a multi-layered agentic architecture with RAG, guardrails, memory systems, and human-in-the-loop workflows, resulting in significant cost savings, faster resolution times, and improved quality across all three domains.
Tinder
Tinder implemented a comprehensive LLM-based trust and safety system to combat various forms of harmful content at scale. The solution involves fine-tuning open-source LLMs using LoRA (Low-Rank Adaptation) for different types of violation detection, from spam to hate speech. Using the Lorax framework, they can efficiently serve multiple fine-tuned models on a single GPU, achieving real-time inference with high precision and recall while maintaining cost-effectiveness. The system demonstrates superior generalization capabilities against adversarial behavior compared to traditional ML approaches.
Salesforce
Salesforce's AI platform team faced operational challenges deploying customized large language models (fine-tuned versions of Llama, Qwen, and Mistral) for their Agentforce agentic AI applications. The deployment process was time-consuming, requiring months of optimization for instance families, serving engines, and configurations, while also proving expensive due to GPU capacity reservations for peak usage. By adopting Amazon Bedrock Custom Model Import, Salesforce integrated a unified API for model deployment that minimized infrastructure management while maintaining backward compatibility with existing endpoints. The results included a 30% reduction in deployment time, up to 40% cost savings through pay-per-use pricing, and maintained scalability without sacrificing performance.
Flipkart
Flipkart faced the challenge of evaluating AI-generated opinion summaries of customer reviews, where traditional metrics like ROUGE failed to align with human judgment and couldn't comprehensively assess summary quality across multiple dimensions. The company developed OP-I-PROMPT, a novel single-prompt framework that uses LLMs as evaluators across seven critical dimensions (fluency, coherence, relevance, faithfulness, aspect coverage, sentiment consistency, and specificity), along with SUMMEVAL-OP, a new benchmark dataset with 2,912 expert annotations. The solution achieved a 0.70 Spearman correlation with human judgments, significantly outperforming previous approaches especially on open-source models like Mistral-7B, while demonstrating that high-quality summaries directly impact business metrics like conversion rates and product return rates.
