25 tools in this industry
← Back to LLMOps DatabaseSnorkel
Snorkel developed a specialized benchmark dataset for evaluating AI agents in insurance underwriting, leveraging their expert network of Chartered Property and Casualty Underwriters (CPCUs). The benchmark simulates an AI copilot that assists junior underwriters by reasoning over proprietary knowledge, using multiple tools including databases and underwriting guidelines, and engaging in multi-turn conversations. The evaluation revealed significant performance variations across frontier models (single digits to ~80% accuracy), with notable error modes including tool use failures (36% of conversations) and hallucinations from pretrained domain knowledge, particularly from OpenAI models which hallucinated non-existent insurance products 15-45% of the time.
Snorkel
Snorkel developed a comprehensive benchmark dataset and evaluation framework for AI agents in commercial insurance underwriting, working with Chartered Property and Casualty Underwriters (CPCUs) to create realistic scenarios for small business insurance applications. The system leverages LangGraph and Model Context Protocol to build ReAct agents capable of multi-tool reasoning, database querying, and user interaction. Evaluation across multiple frontier models revealed significant challenges in tool use accuracy (36% error rate), hallucination issues where models introduced domain knowledge not present in guidelines, and substantial variance in performance across different underwriting tasks, with accuracy ranging from single digits to 80% depending on the model and task complexity.
Allianz
Allianz Benelux tackled their complex insurance claims process by implementing an AI-powered chatbot using Landbot. The system processed over 92,000 unique search terms, categorized insurance products, and implemented a real-time feedback loop with Slack and Trello integration. The solution achieved 90% positive ratings from 18,000+ customers while significantly simplifying the claims process and improving operational efficiency.
Prudential
Prudential Financial, in partnership with AWS GenAI Innovation Center, built a scalable multi-agent platform to support 100,000+ financial advisors across insurance and financial services. The system addresses fragmented workflows where advisors previously had to navigate dozens of disconnected IT systems for client engagement, underwriting, product information, and servicing. The solution features an orchestration agent that routes requests to specialized sub-agents (quick quote, forms, product, illustration, book of business) while maintaining context and enforcing governance. The platform-based microservices architecture reduced time-to-value from 6-8 weeks to 3-4 weeks for new agent deployments, enabled cross-business reusability, and provided standardized frameworks for authentication, LLM gateway access, knowledge management, and observability while handling the complexity of scaling multi-agent systems in a regulated financial services environment.
Verisk
Verisk developed PAAS AI, a generative AI-powered conversational assistant to help premium auditors efficiently search and retrieve information from their vast repository of insurance documentation. Using a RAG architecture built on Amazon Bedrock with Claude, along with ElastiCache, OpenSearch, and custom evaluation frameworks, the system reduced document processing time by 96-98% while maintaining high accuracy. The solution demonstrates effective use of hybrid search, careful data chunking, and comprehensive evaluation metrics to ensure reliable AI-powered customer support.
HDI
HDI, a German insurance company, implemented a RAG-based chatbot system to help customer service agents quickly find and access information across multiple knowledge bases. The system processes complex insurance documents, including tables and multi-column layouts, using various chunking strategies and vector search optimizations. After 120 experiments to optimize performance, the production system now serves 800+ users across multiple business lines, handling 26 queries per second with 88% recall rate and 6ms query latency.
Vouch
Vouch Insurance implemented a production machine learning system using Metaflow to handle risk classification and document processing for their technology-focused insurance business. The system combines traditional data warehousing with LLM-powered predictions, processing structured and unstructured data through hourly pipelines. They built a comprehensive stack that includes data transformation, LLM integration via OpenAI, and a FastAPI service layer with an SDK for easy integration by product engineers.
Anzen
The case study explores how Anzen builds robust LLM applications for processing insurance documents in environments where accuracy is critical. They employ a multi-model approach combining specialized models like LayoutLM for document structure analysis with LLMs for content understanding, implement comprehensive monitoring and feedback systems, and use fine-tuned classification models for initial document sorting. Their approach demonstrates how to effectively handle LLM hallucinations and build production-grade systems with high accuracy (99.9% for document classification).
Canada Life
Canada Life, a leading financial services company serving 14 million customers (one in three Canadians), faced significant contact center challenges including 5-minute average speed to answer, wait times up to 40 minutes, complex routing, high transfer rates, and minimal self-service options. The company migrated 21 business units from a legacy system to Amazon Connect in 7 months, implementing AI capabilities including chatbots, call summarization, voice-to-text, automated authentication, and proficiency-based routing. Results included 94% reduction in wait time, 10% reduction in average handle time, $7.5 million savings in first half of 2025, 92% reduction in average speed to answer (now 18 seconds), 83% chatbot containment rate, and 1900 calls deflected per week. The company plans to expand AI capabilities including conversational AI, agent assist, next best action, and fraud detection, projecting $43 million in cost savings over five years.
Liberty IT
Liberty IT, the technology division of Fortune 100 insurance company Liberty Mutual, embarked on a large-scale deployment of generative AI tools across their global workforce of over 5,000 developers and 50,000+ employees. The initiative involved rolling out custom GenAI platforms including Liberty GPT (an internal ChatGPT variant) to 70% of employees and GitHub Copilot to over 90% of IT staff within the first year. The company faced challenges including rapid technology evolution, model availability constraints, cost management, RAG implementation complexity, and achieving true adoption beyond basic usage. Through building a centralized AI platform with governance controls, implementing comprehensive learning programs across six streams, supporting 28 different models optimized for various use cases, and developing custom dashboards for cost tracking and observability, Liberty IT successfully navigated these challenges while maintaining enterprise security and compliance requirements.
Travelers Insurance
Travelers Insurance developed an automated email classification system using Amazon Bedrock and Anthropic's Claude models to categorize millions of service request emails into 13 different categories. Through advanced prompt engineering techniques and without model fine-tuning, they achieved 91% classification accuracy, potentially saving tens of thousands of manual processing hours. The system combines email text analysis, PDF processing using Amazon Textract, and foundation model-based classification in a serverless architecture.
Santalucía Seguros
Santalucía Seguros implemented a GenAI-based Virtual Assistant to improve customer service and agent productivity in their insurance operations. The solution uses a RAG framework powered by Databricks and Microsoft Azure, incorporating MLflow for LLMOps and Mosaic AI Model Serving for LLM deployment. They developed a sophisticated LLM-based evaluation system that acts as a judge for quality assessment before new releases, ensuring consistent performance and reliability of the virtual assistant.
Wakam
Wakam, a European digital insurance leader with 250 employees across 5 countries, faced critical knowledge silos that hampered productivity across insurance operations, business development, customer service, and legal teams. After initially attempting to build custom AI chatbots in-house with their data science team, they pivoted to implementing Dust, a commercial AI agent platform, to unlock organizational knowledge trapped across Notion, SharePoint, Slack, and other systems. Through strategic executive sponsorship, comprehensive employee enablement, and empowering workers to build their own agents, Wakam achieved 70% employee adoption and deployed 136 AI agents within two months, resulting in a 50% reduction in legal contract analysis time and dramatic improvements in self-service data intelligence across the organization.
Marsh McLennan
Marsh McLennan, a global professional services firm, implemented a comprehensive LLM-based assistant solution reaching 87% of their 90,000 employees worldwide, processing 25 million requests annually. Initially focused on productivity enhancement through API access and RAG, they evolved their strategy from using out-of-the-box models to incorporating fine-tuned models for specific tasks, achieving better accuracy than GPT-4 while maintaining cost efficiency. The implementation has conservatively saved over a million hours annually across the organization.
Roots
Roots, an insurance AI company, developed and deployed fine-tuned 7B Mistral models in production using the vLLM framework to process insurance documents for entity extraction, classification, and summarization. The company evaluated multiple inference frameworks and selected vLLM for its performance advantages, achieving up to 130 tokens per second throughput on A100 GPUs with the ability to handle 32 concurrent requests. Their fine-tuned models outperformed GPT-4 on specialized insurance tasks while providing cost-effective processing at $30,000 annually for handling 20-30 million documents, demonstrating the practical benefits of self-hosting specialized models over relying on third-party APIs.
GEICO
GEICO explored using LLMs for customer service chatbots through a hackathon initiative in 2023. After discovering issues with hallucinations and "overpromising" in their initial implementation, they developed a comprehensive RAG (Retrieval Augmented Generation) solution enhanced with their novel "RagRails" approach. This method successfully reduced incorrect responses from 12 out of 20 to zero in test cases by providing structured guidance within retrieved context, demonstrating how to safely deploy LLMs in a regulated insurance environment.
Verisk
Verisk developed a generative AI companion for their Mozart platform to automate insurance policy document comparison and change detection. Using Amazon Bedrock, OpenSearch, and Anthropic's Claude 3 Sonnet model, they built a system that reduces policy review time from days to minutes. The solution combines embedding-based retrieval, sophisticated prompt engineering, and document chunking strategies to achieve over 90% accuracy in change summaries while maintaining cost efficiency and security compliance.
Northwestern Mutual
Northwestern Mutual implemented a GenAI-powered developer support system to address challenges with their internal developer support chat system, which suffered from long response times and repetitive basic queries. Using Amazon Bedrock Agents, they developed a multi-agent system that could automatically handle common developer support requests, documentation queries, and user management tasks. The system went from pilot to production in just three months and successfully reduced support engineer workload while maintaining strict compliance with internal security and risk management requirements.
CDL
CDL, a UK-based insurtech company, has developed a comprehensive AI agent system using Amazon Bedrock to handle insurance policy management tasks in production. The solution includes a supervisor agent architecture that routes customer intents to specialized domain agents, enabling customers to manage their insurance policies through conversational AI interfaces available 24/7. The implementation addresses critical production concerns through rigorous model evaluation processes, guardrails for safety, and comprehensive monitoring, while preparing their APIs to be AI-ready for future digital assistant integrations.
Allianz Direct
Allianz Direct implemented a GenAI-powered agent assist tool using RAG to help contact center agents quickly and accurately answer customer questions about insurance policies. Built on the Databricks Data Intelligence Platform using Mosaic AI tools, the solution improved answer accuracy by 10-15% compared to their previous system, while allowing agents to focus more on customer relationships rather than searching through documentation.
Dataworkz
Insurance companies face challenges with call center efficiency and customer satisfaction. Dataworkz addresses this by implementing a RAG-based solution that converts call recordings into searchable vectors using Amazon Transcribe, Cohere, and MongoDB Atlas Vector Search. The system processes audio recordings through speech-to-text conversion, vectorization, and storage, enabling real-time access to relevant information for customer service agents. This approach aims to improve response accuracy and reduce resolution times.
nib
nib, an Australian health insurance provider covering approximately 2 million people, transformed both customer and agent experiences using AWS generative AI capabilities. The company faced challenges around contact center efficiency, agent onboarding time, and customer service scalability. Their solution involved deploying a conversational AI chatbot called "Nibby" built on Amazon Lex, implementing call summarization using large language models to reduce after-call work, creating an internal knowledge-based GPT application for agents, and developing intelligent document processing for claims. These initiatives resulted in approximately 60% chat deflection, $22 million in savings from Nibby alone, and a reported 50% reduction in after-call work time through automated call summaries, while significantly improving agent onboarding and overall customer experience.
InsuranceDekho
InsuranceDekho addressed the challenge of slow response times in insurance agent queries by implementing a RAG-based chat assistant using Amazon Bedrock and Anthropic's Claude Haiku. The solution eliminated the need for constant SME consultation, cached frequent responses using Redis, and leveraged OpenSearch for vector storage, resulting in an 80% reduction in response times for customer queries about insurance plans.
Lemonade
A comprehensive analysis of common challenges and solutions in implementing RAG (Retrieval Augmented Generation) pipelines at Lemonade, an insurance technology company. The case study covers issues ranging from missing content and retrieval problems to reranking challenges, providing practical solutions including data cleaning, prompt engineering, hyperparameter tuning, and advanced retrieval strategies.
Anzen
Anzen, a small insurance company with under 20 people, leveraged LLMs to compete with larger insurers by automating their underwriting process. They implemented a document classification system using BERT and AWS Textract for information extraction, achieving 95% accuracy in document classification. They also developed a compliance document review system using sentence embeddings and question-answering models to provide immediate feedback on legal documents like offer letters.
