339 tools with this tag
← Back to LLMOps DatabaseLinkedIn's Hiring Assistant, an AI agent for recruiters, faced significant latency challenges when generating long structured outputs (1,000+ tokens) from thousands of input tokens including job descriptions and candidate profiles. To address this, LinkedIn implemented n-gram speculative decoding within their vLLM serving stack, a technique that drafts multiple tokens ahead and verifies them in parallel without compromising output quality. This approach proved ideal for their use case due to the structured, repetitive nature of their outputs (rubric-style summaries with ratings and evidence) and high lexical overlap with prompts. The implementation resulted in nearly 4× higher throughput at the same QPS and SLA ceiling, along with a 66% reduction in P90 end-to-end latency, all while maintaining identical output quality as verified by their evaluation pipelines.
Codeium
Codeium addressed the limitations of traditional embedding-based retrieval in code generation by developing a novel approach called M-query, which leverages vertical integration and custom infrastructure to run thousands of parallel LLM calls for context analysis. Instead of relying solely on vector embeddings, they implemented a system that can process entire codebases efficiently, resulting in more accurate and contextually aware code generation. Their approach has led to improved user satisfaction and code generation acceptance rates while maintaining rapid response times.
Pinterest enhanced their homefeed recommendation system through several advancements in embedding-based retrieval. They implemented sophisticated feature crossing techniques using MaskNet and DHEN frameworks, adopted pre-trained ID embeddings with careful overfitting mitigation, upgraded their serving corpus with time-decay mechanisms, and introduced multi-embedding retrieval and conditional retrieval approaches. These improvements led to significant gains in user engagement metrics, with increases ranging from 0.1% to 1.2% across various metrics including engaged sessions, saves, and clicks.
Grammarly
Grammarly, a leading AI-powered writing assistant, tackled the challenge of improving grammatical error correction (GEC) by moving beyond traditional neural machine translation approaches that optimize n-gram metrics but sometimes produce semantically inconsistent corrections. The team developed a novel generative adversarial network (GAN) framework where a sequence-to-sequence generator produces grammatical corrections, and a sentence-pair discriminator evaluates whether the generated correction is the most appropriate rewrite for the given input sentence. Through adversarial training with policy gradients, the discriminator provides task-specific rewards to the generator, enabling better distributional alignment between generated and human corrections. Experiments showed that adversarially trained models (both RNN-based and transformer-based) consistently outperformed their standard counterparts on GEC benchmarks, striking a better balance between grammatical correctness, semantic preservation, and natural phrasing while serving millions of users in production.
Zoom
Zoom developed AI Companion 3.0, an agentic AI system that transforms meeting conversations into actionable outcomes through automated planning, reasoning, and execution. The system addresses the challenge of turning hours of meeting content across distributed teams into coordinated action by implementing a federated AI approach combining small language models (SLMs) with large language models (LLMs), deployed on AWS infrastructure including Bedrock and OpenSearch. The solution enables users to automatically generate meeting summaries, perform cross-meeting analysis, schedule meetings with intelligent calendar management, and prepare meeting agendas—reducing what typically takes days of administrative work to minutes while maintaining low latency and cost-effectiveness at scale.
Snorkel
Snorkel developed a specialized benchmark dataset for evaluating AI agents in insurance underwriting, leveraging their expert network of Chartered Property and Casualty Underwriters (CPCUs). The benchmark simulates an AI copilot that assists junior underwriters by reasoning over proprietary knowledge, using multiple tools including databases and underwriting guidelines, and engaging in multi-turn conversations. The evaluation revealed significant performance variations across frontier models (single digits to ~80% accuracy), with notable error modes including tool use failures (36% of conversations) and hallucinations from pretrained domain knowledge, particularly from OpenAI models which hallucinated non-existent insurance products 15-45% of the time.
Commonwealth Bank of Australia
Commonwealth Bank of Australia (CBA) partnered with AWS ProServe to modernize legacy Windows 2012 applications and migrate them to cloud at scale. Facing challenges with time-consuming manual processes, missing documentation, and significant technical debt, CBA developed "Lumos," an internal multi-agent AI platform that orchestrates the entire modernization lifecycle—from application analysis and design through code transformation, testing, deployment, and operations. By integrating AI agents with deterministic engines and AWS services (Bedrock, ECS, OpenSearch, etc.), CBA increased their modernization velocity from 10 applications per year to 20-30 applications per quarter, while maintaining security, compliance, and quality standards through human-in-the-loop validation and multi-agent review processes.
AstraZeneca
AstraZeneca partnered with AWS to deploy agentic AI systems across their clinical development and commercial operations to accelerate their goal of delivering 20 new medicines by 2030. The company built two major production systems: a Development Assistant serving over 1,000 users across 21 countries that integrates 16 data products with 9 agents to enable natural language queries across clinical trials, regulatory submissions, patient safety, and quality domains; and an AZ Brain commercial platform that uses 500+ AI models and agents to provide precision insights for patient identification, HCP engagement, and content generation. The implementation reduced time-to-market for various workflows from months to weeks, with field teams using the commercial assistant generating 2x more prescriptions, and reimbursement dossier authoring timelines dramatically shortened through automated agent workflows.
Loka
Loka, an AWS partner specializing in generative AI solutions, and Domo, a business intelligence platform, demonstrate production implementations of agentic AI systems across multiple industries. Loka showcases their drug discovery assistant (ADA) that integrates multiple AI models and databases to accelerate pharmaceutical research workflows, while Domo presents agentic solutions for call center optimization and financial analysis. Both companies emphasize the importance of systematic approaches to AI implementation, moving beyond simple chatbots to multi-agent systems that can take autonomous actions while maintaining human oversight through human-in-the-loop architectures.
FSI
Digital asset market makers face the challenge of rapidly analyzing news events and social media posts to adjust trading strategies within seconds to avoid adverse selection and inventory risk. Traditional dictionary-based and statistical machine learning approaches proved too slow or required extensive labeled data. The solution involved building an agentic LLM-based platform on AWS that processes streaming news in near real-time, using fine-tuned embeddings for deduplication, reasoning models for sentiment analysis and impact assessment, and optimized inference infrastructure. Through progressive optimization from SageMaker JumpStart to VLLM to SGLNG, the team achieved 180 output tokens per second, enabling end-to-end latency under 10 seconds and doubling news processing capacity compared to initial deployment.
Meta
Meta developed AI Lab, a pre-production framework for continuously testing and optimizing machine learning workflows, with a focus on minimizing Time to First Batch (TTFB). The system enables both proactive improvements and automatic regression prevention for ML infrastructure changes. Using AI Lab, Meta was able to achieve up to 40% reduction in TTFB through the implementation of the Python Cinder runtime, while ensuring no regressions occurred during the rollout process.
Swedish Tax Authority
The Swedish Tax Authority (Skatteverket) has been on a multi-decade digitalization journey, progressively incorporating AI and large language models into production systems to automate and enhance tax services. The organization has developed various NLP applications including text categorization, transcription, OCR pipelines, and question-answering systems using RAG architectures. They have tested both open-source models (Llama 3.1, Mixtral 7B, Cohere) and commercial solutions (GPT-3.5), finding that open-source models perform comparably for simpler queries while commercial models excel at complex questions. The Authority operates within a regulated environment requiring on-premise deployment for sensitive data, adopting Agile/SAFe methodologies and building reusable AI infrastructure components that can serve multiple business domains across different public sector silos.
Meta
Meta developed an AI-assisted root cause analysis system to streamline incident investigations in their large-scale systems. The system combines heuristic-based retrieval with LLM-based ranking to identify potential root causes of incidents. Using a fine-tuned Llama 2 model and a novel ranking approach, the system achieves 42% accuracy in identifying root causes for investigations at creation time in their web monorepo, significantly reducing the investigation time and helping responders make better decisions.
Geminus
Geminus addresses the challenge of optimizing large industrial machinery operations by combining traditional ML models with high-fidelity simulations to create fast, trustworthy digital twins. Their solution reduces model development time from 24 months to just days, while building operator trust through probabilistic approaches and uncertainty bounds. The system provides optimization advice through existing control systems, ensuring safety and reliability while significantly improving machine performance.
Amazon
Amazon developed Dialogue Boost, an AI-powered audio processing technology that enhances dialogue clarity in TV shows, movies, and podcasts by suppressing background music and sound effects. The system uses deep neural networks for sound source separation and runs directly on-device (Echo smart speakers and Fire TV devices) thanks to breakthroughs in model compression and knowledge distillation. Originally launched on Prime Video in 2022 using cloud-based processing, the technology was compressed to less than 1% of its original size while maintaining nearly identical performance, enabling real-time processing across multiple streaming platforms including Netflix, YouTube, and Disney+. Research shows over 86% of participants preferred Dialogue-Boost-enhanced audio, with 100% approval among users with hearing loss, significantly reducing listening effort and improving accessibility for millions of viewers globally.
Veradigm
Veradigm, a healthcare IT company, partnered with AWS to integrate generative AI into their Practice Fusion electronic health record (EHR) system to address clinician burnout caused by excessive documentation tasks. The solution leverages AWS HealthScribe for autonomous AI scribing that generates clinical notes from patient-clinician conversations, and AWS HealthLake as a FHIR-based data foundation to provide patient context at scale. The implementation resulted in clinicians saving approximately 2 hours per day on charting, 65% of users requiring no training to adopt the technology, and high satisfaction with note quality. The system processes 60 million patient visits annually and enables ambient documentation that allows clinicians to focus on patient care rather than typing, with a clear path toward zero-edit note generation.
Heidi Health
Heidi Health developed an ambient AI scribe to reduce the administrative burden on healthcare clinicians by automatically generating clinical notes from patient consultations. The company faced significant LLMOps challenges including building confidence in non-deterministic AI outputs through "clinicians in the loop" evaluation processes, scaling clinical validation beyond small teams using synthetic data generation and LLM-as-judge approaches, and managing global expansion across regions with different data sovereignty requirements, model availability constraints, and regulatory compliance needs. Their solution involved standardizing infrastructure-as-code deployments across AWS regions, using a hybrid approach of Amazon Bedrock for immediate availability and EKS for self-hosted model control, and integrating clinical ambassadors in each region to validate medical accuracy and local practice patterns. The platform now serves over 370,000 clinicians processing 10 million consultations per month globally.
Cresta / OpenAI
Cresta, founded in 2017 by Stanford PhD students with OpenAI research experience, developed an AI copilot system for contact center agents that provides real-time suggestions during customer conversations. The company tackled the challenge of transforming academic NLP and reinforcement learning research into production-grade enterprise software by building domain-specific models fine-tuned on customer conversation data. Starting with Intuit as their first customer through an unconventional internship arrangement, they demonstrated measurable ROI through A/B testing, showing improved conversion rates and agent productivity. The solution evolved from custom LSTM and transformer models to leveraging pre-trained foundation models like GPT-3/4 with fine-tuning, ultimately serving Fortune 500 customers across telecommunications, airlines, and banking with demonstrated value including a pilot generating $100 million in incremental revenue.
Roblox
Roblox moderates billions of pieces of user-generated content daily across 28 languages using a sophisticated AI-driven system that combines large transformer-based models with human oversight. The platform processes an average of 6.1 billion chat messages and 1.1 million hours of voice communication per day, requiring ML models that can make moderation decisions in milliseconds. The system achieves over 750,000 requests per second for text filtering, with specialized models for different violation types (PII, profanity, hate speech). The solution integrates GPU-based serving infrastructure, model quantization and distillation for efficiency, real-time feedback mechanisms that reduce violations by 5-6%, and continuous model improvement through diverse data sampling strategies including synthetic data generation via LLMs, uncertainty sampling, and AI-assisted red teaming.
DoorDash
DoorDash developed SafeChat, an AI-powered content moderation system to handle millions of daily messages, hundreds of thousands of images, and voice calls exchanged between delivery drivers (Dashers) and customers. The platform employs a multi-layered architecture that evolved from using three external LLMs to a more efficient two-layer approach combining an internally trained model with a precise external LLM, processing text, images, and voice communications in real-time. Since launch, SafeChat has achieved a 50% reduction in low to medium-severity safety incidents while maintaining low latency (under 300ms for most messages) and cost-effectiveness by intelligently routing only 0.2% of content to expensive, high-precision models.
Australian Epilepsy Project
The Australian Epilepsy Project (AEP) developed a cloud-based precision medicine platform on AWS that integrates multimodal patient data (MRI scans, neuropsychological assessments, genetic data, and medical histories) to support epilepsy diagnosis and treatment planning. The platform leverages various AI/ML techniques including machine learning models for automated brain region analysis, large language models for medical text processing through RAG approaches, and generative AI for patient summaries. This resulted in a 70% reduction in diagnosis time for language area mapping prior to surgery, 10% higher lesion detection rates, and improved patient outcomes including 9% better work productivity and 8% reduction in seizures over two years.
Providence
Providence Health System automated the processing of over 40 million annual faxes using GenAI and MLflow on Databricks to transform manual referral workflows into real-time automated triage. The system combines OCR with GPT-4.0 models to extract referral data from diverse document formats and integrates seamlessly with Epic EHR systems, eliminating months-long backlogs and freeing clinical staff to focus on patient care across 1,000+ clinics.
Delivery Hero
Delivery Hero built a comprehensive AI-powered image generation system to address the problem that 86% of food products lacked images, which significantly impacted conversion rates. The solution involved implementing both text-to-image generation and image inpainting workflows using Stable Diffusion models, with extensive optimization for cost efficiency and quality assurance. The system successfully generated over 100,000 production images, achieved 6-8% conversion rate improvements, and reduced costs to under $0.003 per image through infrastructure optimization and model fine-tuning.
Feedzai
Feedzai developed TrustScore, an AI-powered fraud detection system that addresses the limitations of traditional rule-based and custom AI models in financial crime detection. The solution leverages a Mixture of Experts (MoE) architecture combined with federated learning to aggregate fraud intelligence from across Feedzai's network of financial institutions processing $8.02T in yearly transactions. Unlike traditional systems that require months of historical data and constant manual updates, TrustScore provides a zero-day, ready-to-use solution that continuously adapts to emerging fraud patterns while maintaining strict data privacy. Real-world deployments have demonstrated significant improvements in fraud detection rates and reductions in false positives compared to traditional out-of-the-box rule systems.
Volkswagen
Volkswagen Group Services partnered with AWS to build a production-scale generative AI platform for automotive marketing content generation and compliance evaluation. The problem was a slow, manual content supply chain that took weeks to months, created confidentiality risks with pre-production vehicles, and faced massive compliance bottlenecks across 10 brands and 200+ countries. The solution involved fine-tuning diffusion models on proprietary vehicle imagery (including digital twins from CAD), automated prompt enhancement using LLMs, and multi-stage image evaluation using vision-language models for both component-level accuracy and brand guideline compliance. Results included massive time savings (weeks to minutes), automated compliance checks across legal and brand requirements, and a reusable shared platform supporting multiple use cases across the organization.
Amazon
Amazon developed an AI-driven compliance screening system to handle approximately 2 billion daily transactions across 160+ businesses globally, ensuring adherence to sanctions and regulatory requirements. The solution employs a three-tier approach: a screening engine using fuzzy matching and vector embeddings, an intelligent automation layer with traditional ML models, and an AI-powered investigation system featuring specialized agents built on Amazon Bedrock AgentCore Runtime. These agents work collaboratively to analyze matches, gather evidence, and make recommendations following standardized operating procedures. The system achieves 96% accuracy with 96% precision and 100% recall, automating decision-making for over 60% of case volume while reserving human intervention only for edge cases requiring nuanced judgment.
LyricLens
LyricLens, developed by Music Smatch, is a production AI system that extracts semantic meaning, themes, entities, cultural references, and sentiment from music lyrics at scale. The platform analyzes over 11 million songs using Amazon Bedrock's Nova family of foundation models to provide real-time insights for brands, artists, developers, and content moderators. By migrating from a previous provider to Amazon Nova models, Music Smatch achieved over 30% cost savings while maintaining accuracy, processing over 2.5 billion tokens. The system employs a multi-level semantic engine with knowledge graphs, supports content moderation with granular PG ratings, and enables natural language queries for playlist generation and trend analysis across demographics, genres, and time periods.
Cedars Sinai
Cedars Sinai and various academic institutions have implemented AI and machine learning solutions to improve neurosurgical outcomes across multiple areas. The applications include brain tumor classification using CNNs achieving 95% accuracy (surpassing traditional radiologists), hematoma prediction and management using graph neural networks with 80%+ accuracy, and AI-assisted surgical planning and intraoperative guidance. The implementations demonstrate significant improvements in patient outcomes while highlighting the importance of balanced innovation with appropriate regulatory oversight.
Golden State Warriors
The Golden State Warriors implemented a recommendation engine powered by Google Cloud's Vertex AI to personalize content delivery for their fans across multiple platforms. The system integrates event data, news content, game highlights, retail inventory, and user analytics to provide tailored recommendations for both sports events and entertainment content at Chase Center. The solution enables personalized experiences for 18,000+ venue seats while operating with limited technical resources.
LinkedIn transformed their traditional keyword-based job search into an AI-powered semantic search system to serve 1.2 billion members. The company addressed limitations of exact keyword matching by implementing a multi-stage LLM architecture combining retrieval and ranking models, supported by synthetic data generation, GPU-optimized embedding-based retrieval, and cross-encoder ranking models. The solution enables natural language job queries like "Find software engineer jobs that are mostly remote with above median pay" while maintaining low latency and high relevance at massive scale through techniques like model distillation, KV caching, and exhaustive GPU-based nearest neighbor search.
LinkedIn deployed a sophisticated machine learning pipeline to extract and map skills from unstructured content across their platform (job postings, profiles, resumes, learning courses) to power their Skills Graph. The solution combines token-based and semantic skill tagging using BERT-based models, multitask learning frameworks for domain-specific scoring, and knowledge distillation to serve models at scale while meeting strict latency requirements (100ms for 200 profile edits/second). Product-driven feedback loops from recruiters and job seekers continuously improve model performance, resulting in measurable business impact including 0.46% increase in predicted confirmed hires for job recommendations and 0.76% increase in PPC revenue for job search.
Furuno
Furuno, a marine electronics company known for inventing the first fish finder in 1948, is addressing sustainable fishing challenges by combining traditional fishermen's knowledge with AI and LLMs. They've developed an ensemble model approach that combines image recognition, classification models, and a unique knowledge model enhanced by LLMs to help identify fish species and make better fishing decisions. The system is being deployed as a $300 monthly subscription service, with initial promising results in improving fishing efficiency while promoting sustainability.
Google Docs implemented automatic document summary generation to help users manage the volume of documents they receive daily. The challenge was to create concise, high-quality summaries that capture document essence while maintaining writer control over the final output. Google developed a solution based on Pegasus, a Transformer-based abstractive summarization model with custom pre-training, combined with careful data curation focusing on quality over quantity, knowledge distillation to optimize serving efficiency (distilling to a Transformer encoder + RNN decoder hybrid), and TPU-based serving infrastructure. The feature was launched for Google Workspace business customers, providing 1-2 sentence suggestions that writers can accept, edit, or ignore, helping both document creators and readers navigate content more efficiently.
Edmunds
Edmunds transformed their dealer review moderation process from a manual system taking up to 72 hours to an automated GenAI solution using GPT-4 through Databricks Model Serving. The solution processes over 300 daily dealer quality-of-service reviews, reducing moderation time from days to minutes and requiring only two moderators instead of a larger team. The implementation included careful prompt engineering and integration with Databricks Unity Catalog for improved data governance.
Quotient AI
Quotient AI addresses the challenge of manually improving AI agents in production by building an infrastructure platform that automatically transforms real-world telemetry data into reinforcement learning signals. The platform ingests agent traces with minimal code integration, analyzes production behavior using specialized models, and generates custom fine-tuned models that perform better at specific tasks than the original base models. The solution reduces the improvement cycle from weeks or months to approximately one hour (with plans to optimize to 20 minutes), enabling developers to deploy continuously improving agents without the manual testing and analysis overhead typically required in traditional LLMOps workflows.
NVIDIA
NVIDIA engineers developed a novel approach to automatically generate optimized GPU attention kernels using the DeepSeek-R1 language model combined with inference-time scaling. They implemented a closed-loop system where the model generates code that is verified and refined through multiple iterations, achieving 100% accuracy for Level-1 problems and 96% for Level-2 problems in Stanford's KernelBench benchmark. This approach demonstrates how additional compute resources during inference can improve code generation capabilities of LLMs.
JetBlue
JetBlue faced challenges in manually tuning prompts across complex, multi-stage LLM pipelines for applications like customer feedback classification and RAG-powered predictive maintenance chatbots. The airline adopted DSPy, a framework for building self-optimizing LLM pipelines, integrated with Databricks infrastructure including Model Serving and Vector Search. By leveraging DSPy's automatic optimization capabilities and modular architecture, JetBlue achieved 2x faster RAG chatbot deployment compared to their previous Langchain implementation, eliminated manual prompt engineering, and enabled automatic optimization of pipeline quality metrics using LLM-as-a-judge evaluations, resulting in more reliable and efficient LLM applications at scale.
Shopify
Shopify tackled the challenge of automatically understanding and categorizing millions of products across their platform by implementing a multi-step Vision LLM solution. The system extracts structured product information including categories and attributes from product images and descriptions, enabling better search, tax calculation, and recommendations. Through careful fine-tuning, evaluation, and cost optimization, they scaled the solution to handle tens of millions of predictions daily while maintaining high accuracy and managing hallucinations.
VSL Labs
VSL Labs is developing an automated system for translating English into American Sign Language (ASL) using generative AI models. The solution addresses the significant challenges faced by the deaf community, including limited availability and high costs of human interpreters. Their platform uses a combination of in-house and GPT-4 models to handle text processing, cultural adaptation, and generates precise signing instructions including facial expressions and body movements for realistic avatar-based sign language interpretation.
Meta
Meta developed TestGen-LLM, a tool that leverages large language models to automatically improve unit test coverage for Android applications written in Kotlin. The system uses an Assured Offline LLM-Based Software Engineering approach to generate additional test cases while maintaining strict quality controls. When deployed at Meta, particularly for Instagram and Facebook platforms, the tool successfully enhanced 10% of the targeted classes with reliable test improvements that were accepted by engineers for production use.
DDI
DDI, a leadership development company, transformed their manual behavioral simulation assessment process by implementing LLMs and MLOps practices using Databricks. They reduced report generation time from 48 hours to 10 seconds while improving assessment accuracy through prompt engineering and model fine-tuning. The solution leveraged DSPy for prompt optimization and achieved significant improvements in recall and F1 scores, demonstrating the successful automation of complex behavioral analyses at scale.
Heidelberg University
Researchers at Heidelberg University developed a novel approach to address the growing workload of radiologists by automating the generation of detailed radiology reports from medical images. They implemented a system using Vision Transformers for image analysis combined with a fine-tuned Llama 3 model for report generation. The solution achieved promising results with a training loss of 0.72 and validation loss of 1.36, demonstrating the potential for efficient, high-quality report generation while running on a single GPU through careful optimization techniques.
MediaRadar | Vivvix
MediaRadar | Vivvix faced challenges with manual video ad classification and fragmented workflows that couldn't keep up with growing ad volumes. They implemented a solution using Databricks Mosaic AI and Apache Spark Structured Streaming to automate ad classification, combining GenAI models with their own classification systems. This transformation enabled them to process 2,000 ads per hour (up from 800), reduced experimentation time from 2 days to 4 hours, and significantly improved the accuracy of insights delivered to customers.
UK MetOffice
The UK Met Office partnered with AWS to automate the generation of the Shipping Forecast, a 100-year-old maritime weather forecast that traditionally required expert meteorologists several hours daily to produce. The solution involved fine-tuning Amazon Nova foundation models (both LLM and vision-language model variants) to convert complex multi-dimensional weather data into structured text forecasts. Within four weeks of prototyping, they achieved 52-62% accuracy using vision-language models and 62% accuracy using text-based LLMs, reducing forecast generation time from hours to under 5 minutes. The project demonstrated scalable architectural patterns for data-to-text conversion tasks involving massive datasets (45GB+ per forecast run) and established frameworks for rapid experimentation with foundation models in production weather services.
British Telecom
British Telecom (BT) partnered with AWS to deploy agentic AI systems for autonomous network operations across their 5G standalone mobile network infrastructure serving 30 million subscribers. The initiative addresses major operational challenges including high manual operations costs (up to 20% of revenue), complex failure diagnosis in containerized networks with 20,000 macro sites generating petabytes of data, and difficulties in change impact analysis with 11,000 weekly network changes. The solution leverages AWS Bedrock Agent Core, Amazon SageMaker for multivariate anomaly detection, Amazon Neptune for network topology graphs, and domain-specific community agents for root cause analysis and service impact assessment. Early results focus on cost reduction through automation, improved service level agreements, faster customer impact identification, and enhanced change efficiency, with plans to expand coverage optimization, dynamic network slicing, and further closed-loop automation across all network domains.
Samsung
Samsung is implementing a comprehensive LLMOps system for autonomous semiconductor fabrication, using multi-modal LLMs and reinforcement learning to transform manufacturing processes. The system combines sensor data analysis, knowledge graphs, and LLMs to automate equipment control, defect detection, and process optimization. Early results show significant improvements in areas like RF matching efficiency and anomaly detection, though challenges remain in real-time processing and time series prediction accuracy.
Outerbounds / AWS
The key lesson from this meetup is that we're seeing a fundamental shift in how organizations can approach large-scale ML training and deployment. Through the combination of purpose-built hardware (AWS Trainium/Inferentia) and modern MLOps frameworks (Metaflow), teams can now achieve enterprise-grade ML infrastructure without requiring deep expertise in distributed systems. The traditional approach of having ML experts manually manage infrastructure is being replaced by more automated, standardized workflows that integrate with existing software delivery practices. This democratization is enabled by significant cost reductions (up to 50-80% compared to traditional GPU deployments), simplified deployment patterns through tools like Optimum Neuron, and the ability to scale from small experiments to massive distributed training with minimal code changes. Perhaps most importantly, the barrier to entry for sophisticated ML infrastructure has been lowered to the point where even small teams can leverage these tools effectively.
Instacart
Instacart built a centralized contextual retrieval system powered by BERT-like transformer models to provide real-time product recommendations across multiple shopping surfaces including search, cart, and item detail pages. The system replaced disparate legacy retrieval systems that relied on ad-hoc combinations of co-occurrence, similarity, and popularity signals with a unified approach that predicts next-product probabilities based on in-session user interaction sequences. The solution achieved a 30% lift in user cart additions for cart recommendations, 10-40% improvement in Recall@K metrics over randomized sequence baselines, and enabled deprecation of multiple legacy ad-hoc retrieval systems while serving both ads and organic recommendation surfaces.
Microsoft
A discussion with Raj Ricky, Principal Product Manager at Microsoft, about the development and deployment of AI agents in production. He shares insights on how to effectively evaluate agent frameworks, develop MVPs, and implement testing strategies. The conversation covers the importance of starting with constrained environments, keeping humans in the loop during initial development, and gradually scaling up agent capabilities while maintaining clear success criteria.
MNP
MNP, a Canadian professional services firm, faced challenges with their conventional data analytics platforms and needed to modernize to support advanced LLM applications. They partnered with Databricks to implement a lakehouse architecture that integrated Mixtral 8x7B using RAG for delivering contextual insights to clients. The solution was deployed in under 6 weeks, enabling secure, efficient processing of complex data queries while maintaining data isolation through Private AI standards.
Perplexity
Perplexity developed Pro Search, an advanced AI answer engine that handles complex, multi-step queries by breaking them down into manageable steps. The system combines careful prompt engineering, step-by-step planning and execution, and an interactive UI to deliver precise answers. The solution resulted in a 50% increase in query search volume, demonstrating its effectiveness in handling complex research questions efficiently.
Qualtrics
Qualtrics built Socrates, an enterprise-level ML platform, to power their experience management solutions. The platform leverages Amazon SageMaker and Bedrock to enable the full ML lifecycle, from data exploration to model deployment and monitoring. It includes features like the Science Workbench, AI Playground, unified GenAI Gateway, and managed inference APIs, allowing teams to efficiently develop, deploy, and manage AI solutions while achieving significant cost savings and performance improvements through optimized inference capabilities.
Swiggy
Swiggy implemented various generative AI solutions to enhance their food delivery platform, focusing on catalog enrichment, review summarization, and vendor support. They developed a platformized approach with a middle layer for GenAI capabilities, addressing challenges like hallucination and latency through careful model selection, fine-tuning, and RAG implementations. The initiative showed promising results in improving customer experience and operational efficiency across multiple use cases including image generation, text descriptions, and restaurant partner support.
IncludedHealth
IncludedHealth built Wordsmith, a comprehensive platform for GenAI applications in healthcare, starting in early 2023. The platform includes a proxy service for multi-provider LLM access, model serving capabilities, training and evaluation libraries, and prompt engineering tools. This enabled multiple production applications including automated documentation, coverage checking, and clinical documentation, while maintaining security and compliance in a regulated healthcare environment.
Databricks
Databricks developed an AI-generated documentation feature for automatically documenting tables and columns in Unity Catalog. After initially using SaaS LLMs that faced challenges with quality, performance, and cost, they built a custom fine-tuned 7B parameter model in just one month with two engineers and less than $1,000 in compute costs. The bespoke model achieved better quality than cheaper SaaS alternatives, 10x cost reduction, and higher throughput, now powering 80% of table metadata updates on their platform.
Grab
Grab developed a custom lightweight vision LLM to address the challenges of extracting information from diverse user-submitted documents like ID cards and driver's licenses across Southeast Asia. Traditional OCR systems struggled with the variety of document templates and languages, while proprietary LLMs had high latency and poor SEA language support. The team fine-tuned and ultimately built a custom ~1B parameter vision LLM from scratch, achieving performance comparable to larger 2B models while significantly reducing latency. The solution involved a four-stage training process using synthetic OCR datasets, an auto-labeling framework called Documint, and full-parameter fine-tuning, resulting in dramatic accuracy improvements (+70pp for Thai, +40pp for Vietnamese) and establishing a unified model to replace traditional OCR pipelines.
Slack
Slack developed a generic recommendation API to serve multiple internal use cases for recommending channels and users. They started with a simple API interface hiding complexity, used hand-tuned models for cold starts, and implemented strict privacy controls to protect customer data. The system achieved over 10% improvement when switching from hand-tuned to ML models while maintaining data privacy and gaining internal customer trust through rapid iteration cycles.
Shopify
Shopify addressed the challenge of fragmented product data across millions of merchants by building a Global Catalogue using multimodal LLMs to standardize and enrich billions of product listings. The system processes over 10 million product updates daily through a four-layer architecture involving product data foundation, understanding, matching, and reconciliation. By fine-tuning open-source vision language models and implementing selective field extraction, they achieve 40 million LLM inferences daily with 500ms median latency while reducing GPU usage by 40%. The solution enables improved search, recommendations, and conversational commerce experiences across Shopify's ecosystem.
Roblox
Roblox underwent a three-phase transformation of their AI infrastructure to support rapidly growing ML inference needs across 250+ production models. They built a comprehensive ML platform using Kubeflow, implemented a custom feature store, and developed an ML gateway with vLLM for efficient large language model operations. The system now processes 1.5 billion tokens weekly for their AI Assistant, handles 1 billion daily personalization requests, and manages tens of thousands of CPUs and over a thousand GPUs across hybrid cloud infrastructure.
OpenRouter
OpenRouter was founded in early 2023 to address the fragmented landscape of large language models by creating a unified API marketplace that aggregates over 400 models from 60+ providers. The company identified that the LLM inference market would not be winner-take-all, and built infrastructure to normalize different model APIs, provide intelligent routing, caching, and uptime guarantees. Their platform enables developers to switch between models with near-zero switching costs while providing better prices, uptime, and choice compared to using individual model providers directly.
OpenRouter
OpenRouter was founded in 2023 to address the challenge of choosing between rapidly proliferating language models by creating a unified API marketplace that aggregates over 400 models from 60+ providers. The platform solves the problem of model selection, provider heterogeneity, and high switching costs by providing normalized access, intelligent routing, caching, and real-time performance monitoring. Results include 10-100% month-over-month growth, sub-30ms latency, improved uptime through provider aggregation, and evidence that the AI inference market is becoming multi-model rather than winner-take-all.
Cursor
Cursor built a modern AI-enhanced code editor by forking VS Code and incorporating advanced LLM capabilities. Their approach focused on creating a more responsive and predictive coding environment that goes beyond simple autocompletion, using techniques like mixture of experts (MoE) models, speculative decoding, and sophisticated caching strategies. The editor aims to eliminate low-entropy coding actions and predict developers' next actions, while maintaining high performance and low latency.
Cursor
Cursor, founded by MIT graduates, developed an AI-powered code editor that goes beyond simple code completion to reimagine how developers interact with AI while coding. By focusing on innovative features like instructed edits and codebase indexing, along with developing custom models for specific tasks, they achieved rapid growth to $100M in revenue. Their success demonstrates how combining frontier LLMs with custom-trained models and careful UX design can transform developer productivity.
Meta
Meta developed an AI-powered system for automatically translating and lip-syncing video content across multiple languages. The system combines Meta's Seamless universal translator model with custom lip-syncing technology to create natural-looking translated videos while preserving the original speaker's voice characteristics and emotions. The solution includes comprehensive safety measures, complex model orchestration, and handles challenges like background noise and timing alignment. Early alpha testing shows 90% eligibility rates for submitted content and meaningful increases in content impressions due to expanded language accessibility.
Cursor
Cursor developed Composer, a specialized coding agent model designed to balance speed and intelligence for real-world software engineering tasks. The challenge was creating a model that could perform at near-frontier levels while being four times more efficient at token generation than comparable models, moving away from the "airplane Wi-Fi" problem where agents were either too slow for synchronous work or required long async waits. The solution involved extensive reinforcement learning (RL) training in an environment that closely mimicked production, using custom kernels for low-precision training, parallel tool calling capabilities, semantic search with custom embeddings, and a fleet of cloud VMs to simulate the real Cursor IDE environment. The result was a model that performs close to frontier models like GPT-4.5 and Claude Sonnet 3.5 on coding benchmarks while maintaining significantly faster token generation, enabling developers to stay in flow state rather than context-switching during long agent runs.
Autodesk
Autodesk built a machine learning platform from scratch using Metaflow as the foundation for their managed training infrastructure. The platform enables data scientists to construct end-to-end ML pipelines, with particular focus on distributed training of large language models. They successfully integrated AWS services, implemented security measures, and created a user-friendly interface that supported both experimental and production workflows. The platform has been rolled out to 50 users and demonstrated successful fine-tuning of large language models, including a 6B parameter model in 50 minutes using 16 A10 GPUs.
Malt
Malt's implementation of a retriever-ranker architecture for their freelancer recommendation system, leveraging a vector database (Qdrant) to improve matching speed and scalability. The case study highlights the importance of carefully selecting and integrating vector databases in LLM-powered systems, emphasizing performance benchmarking, filtering capabilities, and deployment considerations to achieve significant improvements in response times and recommendation quality.
Exa.ai
Exa.ai has built the first search engine specifically designed for AI agents rather than human users, addressing the fundamental problem that existing search engines like Google are optimized for consumer clicks and keyword-based queries rather than semantic understanding and agent workflows. The company trained its own models, built its own index, and invested heavily in compute infrastructure (including purchasing their own GPU cluster) to enable meaning-based search that returns raw, primary data sources rather than listicles or summaries. Their solution includes both an API for developers building AI applications and an agentic search tool called Websites that can find and enrich complex, multi-criteria queries. The results include serving hundreds of millions of queries across use cases like sales intelligence, recruiting, market research, and research paper discovery, with 95% inbound growth and expanding from 7 to 28+ employees within a year.
zeb
zeb developed SuperInsight, a generative AI-powered self-service reporting engine that transforms natural language data requests into actionable insights. Using Databricks' DBRX model and combining fine-tuning with RAG approaches, they created a system that reduced data analyst workload by 80-90% while increasing report generation requests by 72%. The solution integrates with existing communication platforms and can generate reports, forecasts, and ML models based on user queries.
Dropbox
Dropbox is transforming from a file storage company to an AI-powered universal search and organization platform. Through their Dash product, they are implementing LLM-powered search and organization capabilities across enterprise content, while maintaining strict data privacy and security. The engineering approach combines open-source LLMs, custom inference stacks, and hybrid architectures to deliver AI features to 700M+ users cost-effectively.
Weights & Biases
A case study of building an open-source Alexa alternative using LLMs, demonstrating the journey from prototype to production. The project used Llama 2 and Mistral models running on affordable hardware, combined with Whisper for speech recognition. Through iterative improvements including prompt engineering and fine-tuning with QLoRA, the system's accuracy improved from 0% to 98%, while maintaining real-time performance requirements.
The Browser Company
The Browser Company transitioned from their Arc browser to building Dia, an AI-native browser, requiring a fundamental shift in how they approached product development and LLMOps. The company invested heavily in tooling for rapid prototyping, evaluation systems, and automated prompt optimization using techniques like Jeba (a sample-efficient prompt optimization method). They created a "model behavior" discipline to define and ship desired LLM behaviors, treating it as a craft analogous to product design. Additionally, they built security considerations into the product design from the ground up, particularly addressing prompt injection vulnerabilities through user confirmation workflows. The result was a browser that provides an AI assistant alongside users, personalizing experiences and helping with tasks, while enabling their entire company—from CEO to strategy team members—to iterate on AI features.
Cursor
Cursor, an AI-powered code editor startup, entered an extremely competitive market dominated by Microsoft's GitHub Copilot and well-funded competitors like Poolside, Augment, and Magic.dev. Despite initial skepticism from advisors about competing against Microsoft's vast resources and distribution, Cursor succeeded by focusing on the right short-term product decisions—specifically deep IDE integration through forking VS Code and delivering immediate value through "Cursor Tab" code completion. The company differentiated itself through rapid iteration, concentrated talent, bottom-up adoption among developers, and eventually building their own fast agent models. Cursor demonstrated that startups can compete against tech giants by moving quickly, dog-fooding their own product, and correctly identifying what developers need in the near term rather than betting solely on long-term agent capabilities.
Cursor
Cursor, an AI-powered IDE built by Anysphere, faced the challenge of scaling from zero to serving billions of code completions daily while handling 1M+ queries per second and 100x growth in load within 12 months. The solution involved building a sophisticated architecture using TypeScript and Rust, implementing a low-latency sync engine for autocomplete suggestions, utilizing Merkle trees and embeddings for semantic code search without storing source code on servers, and developing Anyrun, a Rust-based orchestrator service. The results include reaching $500M+ in annual revenue, serving more than half of the Fortune 500's largest tech companies, and processing hundreds of millions of lines of enterprise code written daily, all while maintaining privacy through encryption and secure indexing practices.
FactSet
FactSet, a financial data and analytics provider, faced challenges with fragmented LLM development approaches across teams, leading to collaboration barriers and inconsistent quality. They implemented a standardized LLMOps framework using Databricks Mosaic AI and MLflow, enabling unified governance, efficient model development, and improved deployment capabilities. This transformation resulted in significant performance improvements, including a 70% reduction in response time for code generation and 60% reduction in end-to-end latency for formula generation, while maintaining high accuracy and enabling cost-effective use of fine-tuned open-source models alongside commercial LLMs.
Doordash
The ML Platform team at Doordash shares their exploration and strategy for building an enterprise LLMOps stack, discussing the unique challenges of deploying LLM applications at scale. The presentation covers key components needed for production LLM systems, including gateway services, prompt management, RAG implementations, and fine-tuning capabilities, while drawing insights from industry leaders like LinkedIn and Uber's approaches to LLMOps architecture.
LinkedIn developed Hiring Assistant, an AI agent designed to transform the recruiting workflow by automating repetitive tasks like candidate sourcing, evaluation, and engagement across 1.2+ billion profiles. The system addresses the challenge of recruiters spending excessive time on pattern-recognition tasks rather than high-value decision-making and relationship building. Using a plan-and-execute agent architecture with specialized sub-agents for intake, sourcing, evaluation, outreach, screening, and learning, Hiring Assistant combines real-time conversational interfaces with large-scale asynchronous execution. The solution leverages LinkedIn's Economic Graph for talent insights, custom fine-tuned LLMs for candidate evaluation, and cognitive memory systems that learn from recruiter behavior over time. The result is a globally available agentic product that enables recruiters to work with greater speed, scale, and intelligence while maintaining human-in-the-loop control for critical decisions.
ADP
ADP, a major HR and payroll services provider, is developing ADP Assist, a generative AI initiative to make their platforms more interactive and user-friendly while maintaining security and quality. They're implementing a comprehensive AI strategy through their "One AI" and "One Data" platforms, partnering with Databricks to address key challenges in quality assurance, IP protection, data structuring, and cost control. The solution employs RAG and various MLOps tools to ensure reliable, secure, and cost-effective AI deployment across their global operations serving over 41 million workers.
HealthInsuranceLLM
Development of an LLM-based system to help generate health insurance appeals, deployed on-premise with limited resources. The system uses fine-tuned models trained on publicly available medical review board data to generate appeals for insurance claim denials. The implementation includes Kubernetes deployment, GPU inference, and a Django frontend, all running on personal hardware with multiple internet providers for reliability.
Propel
Propel developed a sophisticated evaluation framework for testing and benchmarking LLM performance in handling SNAP (food stamp) benefit inquiries. The company created two distinct evaluation approaches: one for benchmarking current base models on SNAP topics, and another for product development. They implemented automated testing using Promptfoo and developed innovative ways to evaluate model responses, including using AI models as judges for assessing response quality and accessibility.
Replit
Replit, a software development platform, aimed to democratize coding by developing their own code completion LLM. Using Databricks' Mosaic AI Training infrastructure, they successfully built and deployed a multi-billion parameter model in just three weeks, enabling them to launch their code completion feature on time with a small team. The solution allowed them to abstract away infrastructure complexity and focus on model development, resulting in a production-ready code generation system that serves their 25 million users.
Mistral
Mistral, a European AI company, evolved from developing academic LLMs to building and deploying enterprise-grade language models. They started with the successful launch of Mistral-7B in September 2023, which became one of the top 10 most downloaded models on Hugging Face. The company focuses not just on model development but on providing comprehensive solutions for enterprise deployment, including custom fine-tuning, on-premise deployment infrastructure, and efficient inference optimization. Their approach demonstrates the challenges and solutions in bringing LLMs from research to production at scale.
LinkedIn developed a comprehensive LLM-based system for extracting and mapping skills from various content sources across their platform to power their Skills Graph. The system uses a multi-step AI pipeline including BERT-based models for semantic understanding, with knowledge distillation techniques for production deployment. They successfully implemented this at scale with strict latency requirements, achieving significant improvements in job recommendations and skills matching while maintaining performance with 80% model size reduction.
Figma
Figma implemented AI-powered search features to help users find designs and components across their organization using text descriptions or visual references. The solution leverages the CLIP multimodal embedding model, with infrastructure built to handle billions of embeddings while keeping costs down. The system combines traditional lexical search with vector similarity search, using AWS services including SageMaker, OpenSearch, and DynamoDB to process and index designs at scale. Key optimizations included vector quantization, software rendering, and cluster autoscaling to manage computational and storage costs.
GitHub
GitHub shares the three-year journey of developing GitHub Copilot, an LLM-powered code completion tool, from concept to general availability. The team followed a "find it, nail it, scale it" framework to identify the problem space (helping developers code faster), create a smooth product experience through rapid iteration and A/B testing, and scale to enterprise readiness. Starting with a focused problem of function-level code completion in IDEs, they leveraged OpenAI's LLMs and Microsoft Azure infrastructure, implementing techniques like neighboring tabs processing, caching for consistency, and security filters. Through technical previews and community feedback, they achieved a 55% faster coding speed and 74% reduction in developer frustration, while addressing responsible AI concerns through code reference tools and vulnerability filtering.
Discord
Discord shares their comprehensive approach to building and deploying LLM-powered features, from ideation to production. They detail their process of identifying use cases, defining requirements, prototyping with commercial LLMs, evaluating prompts using AI-assisted evaluation, and ultimately scaling through either hosted or self-hosted solutions. The case study emphasizes practical considerations around latency, quality, safety, and cost optimization while building production LLM applications.
CircleCI
CircleCI shares their experience building AI-enabled applications like their error summarizer tool, focusing on the challenges of testing and evaluating LLM-powered applications in production. They discuss implementing model-graded evals, handling non-deterministic outputs, managing costs, and building robust testing strategies that balance thoroughness with practicality. The case study provides insights into applying traditional software development practices to AI applications while addressing unique challenges around evaluation, cost management, and scaling.
Cursor
Cursor's AI research team built Composer, an agent-based LLM designed for coding that combines frontier-level intelligence with four times faster token generation than comparable models. The problem they addressed was creating an agentic coding assistant that feels fast enough for interactive use while maintaining high intelligence for realistic software engineering tasks. Their solution involved training a large mixture-of-experts model using reinforcement learning (RL) at scale, developing custom low-precision training kernels, and building infrastructure that integrates their production environment directly into the training loop. The result is a model that performs nearly as well as the best frontier models on their internal benchmarks while delivering edits and tool calls in seconds rather than minutes, fundamentally changing how developers interact with AI coding assistants.
Stripe
Stripe, processing approximately 1.3% of global GDP, has evolved from traditional ML-based fraud detection to deploying transformer-based foundation models for payments that process every transaction in under 100ms. The company built a domain-specific foundation model treating charges as tokens and behavior sequences as context windows, ingesting tens of billions of transactions to power fraud detection, improving card-testing detection from 59% to 97% accuracy for large merchants. Stripe also launched the Agentic Commerce Protocol (ACP) jointly with OpenAI to standardize how agents discover and purchase from merchant catalogs, complemented by internal AI adoption reaching 8,500 employees daily using LLM tools, with 65-70% of engineers using AI coding assistants and achieving significant productivity gains like reducing payment method integrations from 2 months to 2 weeks.
Windsurf
Codeium's journey in building their AI-powered development tools showcases how investing early in enterprise-ready infrastructure, including containerization, security, and comprehensive deployment options, enabled them to scale from individual developers to large enterprise customers. Their "go slow to go fast" approach in building proprietary infrastructure for code completion, retrieval, and agent-based development culminated in Windsurf IDE, demonstrating how thoughtful early architectural decisions can create a more robust foundation for AI tools in production.
Tzafon
Tzafon, a research lab focused on training foundation models for computer use agents, tackled the challenge of enabling LLMs to autonomously interact with computers through visual understanding and action execution. The company identified fundamental limitations in existing models' ability to ground visual information and coordinate actions, leading them to develop custom infrastructure (Waypoint) for data generation at scale, fine-tune vision encoders on screenshot data, and ultimately pre-train models from scratch with specialized computer interaction capabilities. While initial approaches using supervised fine-tuning and reinforcement learning on successful trajectories showed limited generalization, their focus on solving the grounding problem through improved vision-language integration and domain-specific pre-training has positioned them to release models and desktop applications for autonomous computer use, though performance on benchmarks like OS World remains a challenge across the industry.
GitHub
GitHub developed GitHub Copilot by integrating OpenAI's large language models, starting with GPT-3 and evolving through multiple iterations of the Codex model. The problem was creating an effective AI-powered code generation tool that could work seamlessly within developer IDEs. The solution involved extensive prompt crafting to create optimal "pseudo-documents" that guide the model toward better completions, fine-tuning on specific codebases, and implementing contextual improvements such as incorporating code from neighboring editor tabs and file paths. The results included dramatic improvements in code acceptance rates, with the multilingual model eventually solving over 90% of test problems compared to about 50% initially, and noticeable quality improvements particularly for non-top-five programming languages when new model versions were deployed.
Faber Labs
Faber Labs developed Gora (Goal-Oriented Retrieval Agents), a system that transforms subjective relevance ranking using cutting-edge technologies. The system optimizes for specific KPIs like conversion rates and average order value in e-commerce, or minimizing surgical engagements in healthcare. They achieved this through a combination of real-time user feedback processing, unified goal optimization, and high-performance infrastructure built with Rust, resulting in consistent 200%+ improvements in key metrics while maintaining sub-second latency.
Cline
Cline's head of AI presents their experience operating a model-agnostic AI coding agent platform, arguing that the industry has over-invested in "clever scaffolding" like RAG and tool-calling frameworks when frontier models can succeed with simpler approaches. The real bottleneck to progress, they contend, isn't prompt engineering or agent architecture but rather the quality of benchmarks and RL environments used to train models. Cline developed an automated "RL environments factory" system that transforms real-world coding tasks captured from actual user interactions into standardized, containerized training environments. They announce Cline Bench, an open-source benchmark derived from genuine software development work, inviting the community to contribute by simply working on open-source projects with Cline and opting into the initiative, thereby creating a shared substrate for improving frontier models.
Bud Financial / Scotts Miracle-Gro
This case study explores how Bud Financial and Scotts Miracle-Gro leverage Google Cloud's AI capabilities to create personalized customer experiences. Bud Financial developed a conversational AI solution for personalized banking interactions, while Scotts Miracle-Gro implemented an AI assistant called MyScotty for gardening advice and product recommendations. Both companies utilize various Google Cloud services including Vertex AI, GKE, and AI Search to deliver contextual, regulated, and accurate responses to their customers.
eBay
eBay developed a hybrid system for pricing recommendations and similar item search in their marketplace, specifically focusing on sports trading cards. They combined semantic similarity models with direct price prediction approaches, using transformer-based architectures to create embeddings that balance both price accuracy and item similarity. The system helps sellers price their items accurately by finding similar items that have sold recently, while maintaining semantic relevance.
Vercel
This AWS re:Invent 2025 session explores the challenges organizations face moving AI projects from proof-of-concept to production, addressing the statistic that 46% of AI POC projects are canceled before reaching production. AWS Bedrock team members and Vercel's director of AI engineering present a comprehensive framework for production AI systems, focusing on three critical areas: model switching, evaluation, and observability. The session demonstrates how Amazon Bedrock's unified APIs, guardrails, and Agent Core capabilities combined with Vercel's AI SDK and Workflow Development Kit enable rapid development and deployment of durable, production-ready agentic systems. Vercel showcases real-world applications including V0 (an AI-powered prototyping platform), Vercel Agent (an AI code reviewer), and various internal agents deployed across their organization, all powered by Amazon Bedrock infrastructure.
Block (Square)
Block (Square) implemented a comprehensive LLMOps strategy across multiple business units using a combination of retrieval augmentation, fine-tuning, and pre-training approaches. They built a scalable architecture using Databricks' platform that allowed them to manage hundreds of AI endpoints while maintaining operational efficiency, cost control, and quality assurance. The solution enabled them to handle sensitive data securely, optimize model performance, and iterate quickly while maintaining version control and monitoring capabilities.
Anterior
This case study examines Anterior's experience building LLM-powered products for healthcare prior authorization over three years. The company faced the challenge of building production systems around rapidly evolving AI capabilities, where approaches designed around current model limitations could quickly become obsolete. Through experimentation with techniques like hierarchical query reasoning, finetuning, domain knowledge injection, and expert review systems, they learned which approaches compound with model progress versus those that compete with it. The result was a framework for "Sour Lesson-pilled" product development that emphasizes building systems that benefit from model improvements rather than being made redundant by them, with key surviving techniques including dynamic domain knowledge injection and scalable expert review infrastructure.
Replit
Replit tackled the challenge of automating code repair in their IDE by developing a specialized 7B parameter LLM that integrates directly with their Language Server Protocol (LSP) diagnostics. They created a production-ready system that can automatically fix Python code errors by processing real-time IDE events, operational transformations, and project snapshots. Using DeepSeek-Coder-Instruct-v1.5 as their base model, they implemented a comprehensive data pipeline with serverless verification, structured input/output formats, and GPU-accelerated inference. The system achieved competitive results against much larger models like GPT-4 and Claude-3, with their finetuned 7B model matching or exceeding the performance of these larger models on both academic benchmarks and real-world error fixes. The production system features low-latency inference, load balancing, and real-time code application, demonstrating successful deployment of an LLM system in a high-stakes development environment where speed and accuracy are crucial.
Gitlab
Gitlab's ModelOps team developed a sophisticated code completion system using multiple LLMs, implementing a continuous evaluation and improvement pipeline. The system combines both open-source and third-party LLMs, featuring a comprehensive architecture that includes continuous prompt engineering, evaluation benchmarks, and reinforcement learning to consistently improve code completion accuracy and usefulness for developers.
Upwork
Upwork developed Uma, their "mindful AI" assistant, by rejecting off-the-shelf LLM solutions in favor of building custom-trained models using proprietary platform data and in-house AI research. The company hired expert freelancers to create high-quality training datasets, generated synthetic data anchored in real platform interactions, and fine-tuned open-source LLMs specifically for hiring workflows. This approach enabled Uma to handle complex, business-critical tasks including crafting job posts, matching freelancers to opportunities, autonomously coordinating interviews, and evaluating candidates. The strategy resulted in models that substantially outperform generic alternatives on domain-specific tasks while reducing costs by up to 10x and improving reliability in production environments. Uma now operates as an increasingly agentic system that takes meaningful actions across the full hiring lifecycle.
Various
Climate tech startups are leveraging Amazon SageMaker HyperPod to build specialized foundation models that address critical environmental challenges including weather prediction, sustainable material discovery, ecosystem monitoring, and geological modeling. Companies like Orbital Materials and Hum.AI are training custom models from scratch on massive environmental datasets, achieving significant breakthroughs such as tenfold performance improvements in carbon capture materials and the ability to see underwater from satellite imagery. These startups are moving beyond traditional LLM fine-tuning to create domain-specific models with billions of parameters that process multimodal environmental data including satellite imagery, sensor networks, and atmospheric measurements at scale.
Rolls-Royce
Rolls-Royce implemented a cloud-based generative AI approach using GANs (Generative Adversarial Networks) to support preliminary engineering design tasks. The system combines geometric parameters and simulation data to generate and validate new design concepts, with a particular focus on aerospace applications. By leveraging Databricks' cloud infrastructure, they reduced training time from one week to 4-6 hours while maintaining data security through careful governance and transfer learning approaches.
Philips
Philips partnered with AWS to transform medical imaging and diagnostics by moving their entire healthcare informatics portfolio to the cloud, with particular focus on digital pathology. The challenge was managing petabytes of medical imaging data across multiple modalities (radiology, cardiology, pathology) stored in disparate silos, making it difficult for clinicians to access comprehensive patient information efficiently. Philips leveraged AWS Health Imaging and other cloud services to build a scalable, cloud-native integrated diagnostics platform that reduces workflow time from 11+ hours to 36 minutes in pathology, enables real-time collaboration across geographies, and supports AI-assisted diagnosis. The solution now manages 134 petabytes of data covering 34 million patient exams and 11 billion medical records, with 95 of the top 100 US hospitals using Philips healthcare informatics solutions.
Github
Github describes their robust evaluation framework for testing and deploying new LLM models in their Copilot product. The team runs over 4,000 offline tests, including automated code quality assessments and chat capability evaluations, before deploying any model changes to production. They use a combination of automated metrics, LLM-based evaluation, and manual testing to assess model performance, quality, and safety across multiple programming languages and frameworks.
Windsurf
Windsurf, an AI coding toolkit company, addresses the challenge of generating contextually relevant code for individual developers and organizations. While generating generic code has become straightforward, the real challenge lies in producing code that fits into existing large codebases, adheres to organizational standards, and aligns with personal coding preferences. Windsurf's solution centers on a sophisticated context management system that combines user behavioral heuristics (cursor position, open files, clipboard content, terminal activity) with hard evidence from the codebase (code, documentation, rules, memories). Their approach optimizes for relevant context selection rather than simply expanding context windows, leveraging their background in GPU optimization to efficiently find and process relevant context at scale.
Various
A panel discussion featuring experts from Neva, Intercom, Prompt Layer, and OctoML discussing strategies for optimizing costs and performance when running LLMs in production. The panel explores various approaches from using API services to running models in-house, covering topics like model compression, hardware selection, latency optimization, and monitoring techniques. Key insights include the trade-offs between API usage and in-house deployment, strategies for cost reduction, and methods for performance optimization.
Airtrain
Two case studies demonstrate significant cost reduction through LLM fine-tuning. A healthcare company reduced costs and improved privacy by fine-tuning Mistral-7B to match GPT-3.5's performance for patient intake, while an e-commerce unicorn improved product categorization accuracy from 47% to 94% using a fine-tuned model, reducing costs by 94% compared to using GPT-4.
Lmsys
Intel PyTorch Team collaborated with the SGLang project to develop a cost-effective CPU-only deployment solution for large Mixture of Experts (MoE) models like DeepSeek R1, addressing the challenge of high memory requirements that typically necessitate multiple expensive AI accelerators. Their solution leverages Intel Xeon 6 processors with Advanced Matrix Extensions (AMX) and implements highly optimized kernels for attention mechanisms and MoE computations, achieving 6-14x speedup in time-to-first-token (TTFT) and 2-4x speedup in time-per-output-token (TPOT) compared to llama.cpp, while supporting multiple quantization formats including BF16, INT8, and FP8.
Sixt
Sixt, a mobility service provider with over €4 billion in revenue, transformed their customer service operations using generative AI to handle the complexity of multiple product lines across 100+ countries. The company implemented "Project AIR" (AI-based Replies) to automate email classification, generate response proposals, and deploy chatbots across multiple channels. Within five months of ideation, they moved from proof-of-concept to production, achieving over 90% classification accuracy using Amazon Bedrock with Anthropic Claude models (up from 70% with out-of-the-box solutions), while reducing classification costs by 70%. The solution now handles customer inquiries in multiple languages, integrates with backend reservation systems, and has expanded from email automation to messaging and chatbot services deployed across all corporate countries by Q1 2025.
Defense Innovation Unit
The Defense Innovation Unit developed a system to detect illegal, unreported, and unregulated fishing vessels using satellite-based synthetic aperture radar (SAR) imagery and machine learning. They created a large annotated dataset of SAR images, developed ML models for vessel detection, and deployed the system to over 100 countries through a platform called SeaVision. The system successfully identifies "dark vessels" that turn off their AIS transponders to hide illegal fishing activities, enabling better maritime surveillance and law enforcement.
Nvidia
NVIDIA implemented a data flywheel approach to optimize their internal employee support AI agent, addressing the challenge of maintaining accuracy while reducing inference costs. The system continuously collects user feedback and production data to fine-tune smaller, more efficient models that can replace larger, expensive foundational models. Through this approach, they achieved comparable accuracy (94-96%) with significantly smaller models (1B-8B parameters instead of 70B), resulting in 98% cost savings and 70x lower latency while maintaining the agent's effectiveness in routing employee queries across HR, IT, and product documentation domains.
Nvidia
Financial institutions including Capital One, Royal Bank of Canada (RBC), and Visa are deploying agentic AI systems in production to handle real-time financial transactions and complex workflows. These multi-agent systems go beyond simple generative AI by reasoning through problems and taking action autonomously, requiring 100-200x more computational resources than traditional single-shot inference. The implementations focus on use cases like automotive purchasing assistance, investment research automation, and fraud detection, with organizations building proprietary models using open-source foundations (like Llama or Mistral) combined with bank-specific data to achieve 60-70% accuracy improvements. The results include 60% cycle time improvements in report generation, 10x more data analysis capacity, and enhanced fraud detection capabilities, though these gains require substantial investment in AI infrastructure and talent development.
OpenAI
OpenAI addresses the challenge of verifying AI-generated code at scale by deploying an autonomous code reviewer built on GPT-5-Codex and GPT-5.1-Codex-Max. As autonomous coding systems produce code volumes that exceed human oversight capacity, the risk of severe bugs and vulnerabilities increases. The solution involves training a dedicated agentic code reviewer with repository-wide tool access and code execution capabilities, optimizing for precision over recall to maintain developer trust and minimize false alarms. The system now reviews over 100,000 external PRs daily, with authors making code changes in response to 52.7% of comments internally, demonstrating actionable impact while maintaining a low "alignment tax" on developer workflows.
Bainbridge Capital
A data scientist shares their experience transitioning from traditional ML to implementing LLM-based recommendation systems at a private equity company. The case study focuses on building a recommendation system for boomer-generation users, requiring recommendations within the first five suggestions. The implementation involves using OpenAI APIs for data cleaning, text embeddings, and similarity search, while addressing challenges of production deployment on AWS.
eBay
eBay tackled the challenge of incorporating LLMs into their e-commerce platform by developing e-Llama, a domain-adapted version of Llama 3.1. Through continued pre-training on a mix of e-commerce and general domain data, they created 8B and 70B parameter models that achieved 25% improvement in e-commerce tasks while maintaining strong general performance. The training was completed efficiently using 480 NVIDIA H100 GPUs and resulted in production-ready models aligned with human feedback and safety requirements.
LinkedIn developed a family of domain-adapted foundation models (EON models) to enhance their GenAI capabilities across their platform serving 1B+ members. By adapting open-source models like Llama through multi-task instruction tuning and safety alignment, they created cost-effective models that maintain high performance while being 75x more cost-efficient than GPT-4. The EON-8B model demonstrated significant improvements in production applications, including a 4% increase in candidate-job-requirements matching accuracy compared to GPT-4o mini in their Hiring Assistant product.
Ebay
eBay developed customized large language models by adapting Meta's Llama 3.1 models (8B and 70B parameters) to the e-commerce domain through continued pretraining on a mixture of proprietary eBay data and general domain data. This hybrid approach allowed them to infuse domain-specific knowledge while avoiding the resource intensity of training from scratch. Using 480 NVIDIA H100 GPUs and advanced distributed training techniques, they trained the models on 1 trillion tokens, achieving approximately 25% improvement on e-commerce benchmarks for English (30% for non-English) with only 1% degradation on general domain tasks. The resulting "e-Llama" models were further instruction-tuned and aligned with human feedback to power various AI initiatives across the company in a cost-effective, scalable manner.
Articul8
Articul8 developed a generative AI platform to address enterprise challenges in manufacturing and supply chain management, particularly for a European automotive manufacturer. The platform combines public AI models with domain-specific intelligence and proprietary data to create a comprehensive knowledge graph from vast amounts of unstructured data. The solution reduced incident response time from 90 seconds to 30 seconds (3x improvement) and enabled automated root cause analysis for manufacturing defects, helping experts disseminate daily incidents and optimize production processes that previously required manual analysis by experienced engineers.
Deepgram
Deepgram tackles the challenge of building efficient language AI products for call centers by advocating for small, domain-specific language models instead of large foundation models. They demonstrate this by creating a 500M parameter model fine-tuned on call center transcripts, which achieves better performance in call center tasks like conversation continuation and summarization while being more cost-effective and faster than larger models.
Doordash
DoorDash's Summer 2025 interns developed multiple LLM-powered production systems to solve operational challenges. The first project automated never-delivered order feature extraction using a custom DistilBERT model that processes customer-Dasher conversations, achieving 0.8289 F1 score while reducing manual review burden. The second built a scalable chatbot-as-a-service platform using RAG architecture, enabling any team to deploy knowledge-based chatbots with centralized embedding management and customizable prompt templates. These implementations demonstrate practical LLMOps approaches including model comparison, data balancing techniques, and infrastructure design for enterprise-scale conversational AI systems.
Uber
Uber developed DragonCrawl, an innovative AI-powered mobile testing system that uses a small language model (110M parameters) to automate app testing across multiple languages and cities. The system addressed critical challenges in mobile testing, including high maintenance costs and scalability issues across Uber's global operations. Using an MPNet-based architecture with a retriever-ranker approach, DragonCrawl achieved 99%+ stability in production, successfully operated in 85 out of 89 tested cities, and demonstrated remarkable adaptability to UI changes without requiring manual updates. The system proved particularly valuable by blocking ten high-priority bugs from reaching customers while significantly reducing developer maintenance time. Most notably, DragonCrawl exhibited human-like problem-solving behaviors, such as retrying failed operations and implementing creative solutions like app restarts to overcome temporary issues.
Tastewise
This appears to be the Dutch footer section of YouTube's interface, showcasing the platform's localization and content management system. However, without more context about specific LLMOps implementation details, we can only infer that YouTube likely employs language models for content translation, moderation, and user interface localization.
Meta / Ray Ban
Meta Reality Labs developed a production AI system for Ray-Ban Meta smart glasses that brings AI capabilities directly to wearable devices through a four-part architecture combining on-device processing, smartphone connectivity, and cloud-based AI services. The system addresses unique challenges of wearable AI including power constraints, thermal management, connectivity limitations, and real-time performance requirements while enabling features like visual question answering, photo capture, and voice commands with sub-second response times for on-device operations and under 3-second response times for cloud-based AI interactions.
Wayve
Wayve is developing self-driving technology that works across multiple vehicle types and global markets by leveraging end-to-end foundation models trained on driving data rather than traditional rule-based systems. The company moved away from intermediate representations like object detection to a more holistic approach where a single neural network learns to drive from examples, similar to how large language models learn language. This architecture enabled rapid global expansion from primarily driving in London to operating across 500 cities in Japan, Europe, the UK, and the US within a year. The system uses foundation models for multiple tasks including driving, simulation, scenario classification, and even natural language explanations of driving decisions, with all components compressed into a single 75-watt model deployable in production vehicles.
Pinterest improved their ads engagement modeling by implementing a Multi-gate Mixture-of-Experts (MMoE) architecture combined with knowledge distillation techniques. The system faced challenges with short data retention periods and computational efficiency, which they addressed through mixed precision inference and lightweight gate layers. The solution resulted in significant improvements in both offline accuracy and online metrics while achieving a 40% reduction in inference latency.
Credal
A comprehensive analysis of how enterprises adopt and scale AI/LLM technologies, based on observations from multiple companies. The journey typically progresses through four stages: early experimentation, chat with docs workflows, enterprise search, and core operations integration. The case study explores key challenges including data security, use case discovery, and technical implementation hurdles, while providing insights into critical decisions around build vs. buy, platform selection, and LLM provider strategy.
Rubrik
Predibase, a fine-tuning and model serving platform, announced its acquisition by Rubrik, a data security and governance company, with the goal of combining Predibase's generative AI capabilities with Rubrik's secure data infrastructure. The integration aims to address the critical challenge that over 50% of AI pilots never reach production due to issues with security, model quality, latency, and cost. By combining Predibase's post-training and inference capabilities with Rubrik's data security posture management, the merged platform seeks to provide an end-to-end solution that enables enterprises to deploy generative AI applications securely and efficiently at scale.
Barclays
A senior leader in industry discusses the key challenges and opportunities in deploying LLMs at enterprise scale, highlighting the differences between traditional MLOps and LLMOps. The presentation covers critical aspects including cost management, infrastructure needs, team structures, and organizational adaptation required for successful LLM deployment, while emphasizing the importance of leveraging existing MLOps practices rather than completely reinventing the wheel.
AstraZeneca / Adobe / Allianz Technology
A panel discussion featuring leaders from AstraZeneca, Adobe, and Allianz Technology sharing their experiences implementing GenAI in production. The case study covers how these enterprises prioritized use cases, managed legal considerations, and scaled AI adoption. Key successes included AstraZeneca's viral research assistant tool, Adobe's approach to legal frameworks for AI, and Allianz's code modernization efforts. The discussion highlights the importance of early legal engagement, focusing on impactful use cases, and treating AI implementation as a cultural transformation rather than just a tool rollout.
Various
A panel discussion featuring leaders from multiple enterprises sharing their experiences implementing LLMs in production. The discussion covers key challenges including data privacy, security, cost management, and enterprise integration. Speakers from Box discuss content management challenges, Glean covers enterprise search implementations, Tyace shares content generation experiences, Security AI addresses data safety, and Citibank provides CIO perspective on enterprise-wide AI deployment. The panel emphasizes the importance of proper data governance, security controls, and the need for systematic approach to move from POCs to production.
Thomson Reuters
Thomson Reuters developed Open Arena, an enterprise-wide LLM playground, in under 6 weeks using AWS services. The platform enables non-technical employees to experiment with various LLMs in a secure environment, combining open-source and in-house models with company data. The solution saw rapid adoption with over 1,000 monthly users and helped drive innovation across the organization by allowing safe experimentation with generative AI capabilities.
IBM
IBM's Watson X platform addresses enterprise LLMOps challenges by providing a comprehensive solution for model access, deployment, and customization. The platform offers both open-source and proprietary models, focusing on specialized use cases like banking and insurance, while emphasizing API optimization for LLM interactions and robust evaluation capabilities. The case study highlights how enterprises are implementing LLMOps at scale with particular attention to data security, model evaluation, and efficient API design for LLM consumption.
Cisco
At Cisco, the challenge of integrating LLMs into enterprise-scale applications required developing new DevSecOps workflows and practices. The presentation explores how Cisco approached continuous delivery, monitoring, security, and on-call support for LLM-powered applications, showcasing their end-to-end model for LLMOps in a large enterprise environment.
DeepL
DeepL, a translation company founded in 2017, has built a successful enterprise-focused business using neural machine translation models to tackle the language barrier problem at scale. The company handles hundreds of thousands of customers by developing specialized neural translation models that balance accuracy and fluency, training them on curated parallel and monolingual corpora while leveraging context injection rather than per-customer fine-tuning for scalability. By building their own GPU infrastructure early on and developing custom frameworks for inference optimization, DeepL maintains a competitive edge over general-purpose LLMs and established players like Google Translate, demonstrating strong product-market fit in high-stakes enterprise use cases where translation quality directly impacts legal compliance, customer experience, and business operations.
Anomalo
Anomalo addresses the critical challenge of unstructured data quality in enterprise AI deployments by building an automated platform on AWS that processes, validates, and cleanses unstructured documents at scale. The solution automates OCR and text parsing, implements continuous data observability to detect anomalies, enforces governance and compliance policies including PII detection, and leverages Amazon Bedrock for scalable LLM-based document quality analysis. This approach enables enterprises to transform their vast collections of unstructured text data into trusted assets for production AI applications while reducing operational burden, optimizing costs, and maintaining regulatory compliance.
Activeloop
Activeloop developed a solution for processing and generating patents using enterprise-grade memory agents and their Deep Lake vector database. The system handles 600,000 annual patent filings and 80 million total patents, reducing the typical 2-4 week patent generation process through specialized AI agents for different tasks like claim search, abstract generation, and question answering. The solution combines vector search, lexical search, and their proprietary Deep Memory technology to improve information retrieval accuracy by 5-10% without changing the underlying vector search architecture.
Memorial Sloan Kettering / McLeod Health / UCLA
This panel discussion features three major healthcare systems—McLeod Health, Memorial Sloan Kettering Cancer Center, and UCLA Health—discussing their experiences deploying generative AI-powered ambient clinical documentation (AI scribes) at scale. The organizations faced challenges in vendor evaluation, clinician adoption, and demonstrating ROI while addressing physician burnout and documentation burden. Through rigorous evaluation processes including randomized controlled trials, head-to-head vendor comparisons, and structured pilots, these systems successfully deployed AI scribes to hundreds to thousands of physicians. Results included significant reductions in burnout (20% at UCLA), improved patient satisfaction scores (5-6% increases at McLeod), time savings of 1.5-2 hours per day, and positive financial ROI through improved coding and RVU capture. Key learnings emphasized the importance of robust training, encounter-based pricing models, workflow integration, and managing expectations that AI scribes are not a universal solution for all specialties and clinicians.
Wesco
Wesco, a B2B supply chain and industrial distribution company, presents a comprehensive case study on deploying enterprise-grade AI applications at scale, moving from POC to production. The company faced challenges in transitioning from traditional predictive analytics to cognitive intelligence using generative AI and agentic systems. Their solution involved building a composable AI platform with proper governance, MLOps/LLMOps pipelines, and multi-agent architectures for use cases ranging from document processing and knowledge retrieval to fraud detection and inventory management. Results include deployment of 50+ use cases, significant improvements in employee productivity through "everyday AI" applications, and quantifiable ROI through transformational AI initiatives in supply chain optimization, with emphasis on proper observability, compliance, and change management to drive adoption.
Marsh McLennan
Marsh McLennan, a global professional services firm, implemented a comprehensive LLM-based assistant solution reaching 87% of their 90,000 employees worldwide, processing 25 million requests annually. Initially focused on productivity enhancement through API access and RAG, they evolved their strategy from using out-of-the-box models to incorporating fine-tuned models for specific tasks, achieving better accuracy than GPT-4 while maintaining cost efficiency. The implementation has conservatively saved over a million hours annually across the organization.
OpenAI
OpenAI's applied evaluation team presented best practices for implementing LLMs in production through two case studies: Morgan Stanley's internal document search system for financial advisors and Grab's computer vision system for Southeast Asian mapping. Both companies started with simple evaluation frameworks using just 5 initial test cases, then progressively scaled their evaluation systems while maintaining CI/CD integration. Morgan Stanley improved their RAG system's document recall from 20% to 80% through iterative evaluation and optimization, while Grab developed sophisticated vision fine-tuning capabilities for recognizing road signs and lane counts in Southeast Asian contexts. The key insight was that effective evaluation systems enable rapid iteration cycles and clear communication between teams and external partners like OpenAI for model improvement.
Weights & Biases
Weights & Biases documented their journey refactoring Wandbot, their LLM-powered documentation assistant, achieving significant improvements in both accuracy (72% to 81%) and latency (84% reduction). The team initially attempted a "refactor-first, evaluate-later" approach but discovered the necessity of systematic evaluation throughout the process. Through methodical testing and iterative improvements, they replaced multiple components including switching from FAISS to ChromaDB for vector storage, transitioning to LangChain Expression Language (LCEL) for better async operations, and optimizing their RAG pipeline. Their experience highlighted the importance of continuous evaluation in LLM system development, with the team conducting over 50 unique evaluations costing approximately $2,500 to debug and optimize their refactored system.
Writer
Writer, an enterprise AI platform company, evolved their retrieval-augmented generation (RAG) system from traditional vector search to a sophisticated graph-based approach to address limitations in handling dense, specialized enterprise data. Starting with keyword search and progressing through vector embeddings, they encountered accuracy issues with chunking and struggled with concentrated enterprise data where documents shared similar terminology. Their solution combined knowledge graphs with fusion-in-decoder techniques, using specialized models for graph structure conversion and storing graph data as JSON in Lucene-based search engines. This approach resulted in improved accuracy, reduced hallucinations, and better performance compared to seven different vector search systems in benchmarking tests.
NVIDA / Lepton
This lecture transcript from Yangqing Jia, VP at NVIDIA and founder of Lepton AI (acquired by NVIDIA), explores the evolution of AI system design from an engineer's perspective. The talk covers the progression from research frameworks (Caffe, TensorFlow, PyTorch) to production AI infrastructure, examining how LLM applications are built and deployed at scale. Jia discusses the emergence of "neocloud" infrastructure designed specifically for AI workloads, the challenges of GPU cluster management, and practical considerations for building consumer and enterprise LLM applications. Key insights include the trade-offs between open-source and closed-source models, the importance of RAG and agentic AI patterns, infrastructure design differences between conventional cloud and AI-specific platforms, and the practical challenges of operating LLMs in production, including supply chain management for GPUs and cost optimization strategies.
Hitachi
Hitachi's journey in implementing AI across industrial applications showcases the evolution from traditional machine learning to advanced generative AI solutions. The case study highlights how they transformed from focused applications in maintenance, repair, and operations to a more comprehensive approach integrating LLMs, focusing particularly on reliability, small data scenarios, and domain expertise. Key implementations include repair recommendation systems for fleet management and fault tree extraction from manuals, demonstrating the practical challenges and solutions in industrial AI deployment.
Github
The case study details GitHub's journey in developing GitHub Copilot by working with OpenAI's large language models. Starting with GPT-3 experimentation in 2020, the team evolved from basic code generation testing to creating an interactive IDE integration. Through multiple iterations of model improvements, prompt engineering, and fine-tuning techniques, they enhanced the tool's capabilities, ultimately leading to features like multi-language support, context-aware suggestions, and the development of GitHub Copilot X.
Faire
Faire, a wholesale marketplace, evolved their ML model deployment infrastructure from a monolithic approach to a streamlined platform. Initially struggling with slow deployments, limited testing, and complex workflows across multiple systems, they developed an internal Machine Learning Model Management (MMM) tool that unified model deployment processes. This transformation reduced deployment time from 3+ days to 4 hours, enabled safe deployments with comprehensive testing, and improved observability while supporting various ML workloads including LLMs.
Various
The U.S. federal government agencies are working to move AI applications from pilots to production, focusing on scalable and responsible deployment. The Department of Energy (DOE) has implemented Energy GPT using open models in their environment, while the Department of State is utilizing LLMs for diplomatic cable summarization. The U.S. Navy's Project AMMO showcases successful MLOps implementation, reducing model retraining time from six months to one week for underwater vehicle operations. Agencies are addressing challenges around budgeting, security compliance, and governance while ensuring user-friendly AI implementations.
Mercado Libre
Mercado Libre (MELI) faced the challenge of categorizing millions of financial transactions across Latin America in multiple languages and formats as Open Finance unlocked access to customer financial data. Starting with a brittle regex-based system in 2021 that achieved only 60% accuracy and was difficult to maintain, they evolved through three generations: first implementing GPT-3.5 Turbo in 2023 to achieve 80% accuracy with 75% cost reduction, then transitioning to GPT-4o-mini in 2024, and finally developing custom BERT-based semantic embeddings trained on regional financial text to reach 90% accuracy with an additional 30% cost reduction. This evolution enabled them to scale from processing tens of millions of transactions per quarter to tens of millions per week, while enabling near real-time categorization that powers personalized financial insights across their ecosystem.
Impel
Impel, an automotive retail AI company, migrated from a third-party LLM to a fine-tuned Meta Llama model deployed on Amazon SageMaker to power their Sales AI product, which provides 24/7 personalized customer engagement for dealerships. The transition addressed cost predictability concerns and customization limitations, resulting in 20% improved accuracy across core features including response personalization, conversation summarization, and follow-up generation, while achieving better security and operational control.
Roots
Roots, an insurance AI company, developed and deployed fine-tuned 7B Mistral models in production using the vLLM framework to process insurance documents for entity extraction, classification, and summarization. The company evaluated multiple inference frameworks and selected vLLM for its performance advantages, achieving up to 130 tokens per second throughput on A100 GPUs with the ability to handle 32 concurrent requests. Their fine-tuned models outperformed GPT-4 on specialized insurance tasks while providing cost-effective processing at $30,000 annually for handling 20-30 million documents, demonstrating the practical benefits of self-hosting specialized models over relying on third-party APIs.
Swisscom
Swisscom, a leading telecommunications provider in Switzerland, partnered with AWS to deploy fine-tuned large language models in their customer service contact centers to enable personalized, fast, and efficient customer interactions. The problem they faced was providing 24/7 customer service with high accuracy, low latency (critical for voice interactions), and the ability to handle hundreds of requests per minute during peak times while maintaining control over the model lifecycle. Their solution involved using AWS SageMaker to fine-tune a smaller LLM (Llama 3.1 8B) using synthetic data generated by a larger teacher model, implementing LoRA for efficient training, and deploying the model with infrastructure-as-code using AWS CDK. The results achieved median latency below 250 milliseconds in production, accuracy comparable to larger models, cost-efficient scaling with hourly infrastructure charging instead of per-token pricing, and successful handling of 50% of production traffic with the ability to scale for unexpected peaks.
Mercari
Mercari tackled the challenge of extracting dynamic attributes from user-generated marketplace listings by fine-tuning a 2B parameter LLM using QLoRA. The team successfully created a model that outperformed GPT-3.5-turbo while being 95% smaller and 14 times more cost-effective. The implementation included careful dataset preparation, parameter efficient fine-tuning, and post-training quantization using llama.cpp, resulting in a production-ready model with better control over hallucinations.
Faire
Faire, an e-commerce marketplace, tackled the challenge of evaluating search relevance at scale by transitioning from manual human labeling to automated LLM-based assessment. They first implemented a GPT-based solution and later improved it using fine-tuned Llama models. Their best performing model, Llama3-8b, achieved a 28% improvement in relevance prediction accuracy compared to their previous GPT model, while significantly reducing costs through self-hosted inference that can handle 70 million predictions per day using 16 GPUs.
Kantar Worldpanel
Kantar Worldpanel, a market research company, needed to modernize their product description matching system to better link paper receipt descriptions with product barcode names. They leveraged Databricks Mosaic AI to experiment with various LLMs (including Llama, Mistral, and GPT models) to generate high-quality training data, achieving 94% accuracy in matching product descriptions. This automated approach generated 120,000 training pairs in just hours, allowing them to fine-tune smaller models for production use while freeing up human resources for more complex tasks.
Cosine
Cosine, a company building enterprise coding agents, faced the challenge of deploying high-performance AI systems in highly constrained environments including on-premise and air-gapped deployments where large frontier models were not viable. They developed a multi-agent architecture using specialized orchestrator and worker models, leveraging model distillation, supervised fine-tuning, preference optimization, and reinforcement fine-tuning to create smaller models that could match or exceed the performance of much larger models. The result was a 31% performance increase on the SWE-bench Freelancer benchmark, 3X latency improvement, 60% reduction in GPU footprint, and 20% fewer errors in generated code, all while operating on as few as 4 H100 GPUs and maintaining full deployment flexibility across cloud, VPC, and on-premise environments.
Large Gaming Company
AWS Professional Services helped a major gaming company build an automated toxic speech detection system by fine-tuning Large Language Models. Starting with only 100 labeled samples, they experimented with different BERT-based models and data augmentation techniques, ultimately moving from a two-stage to a single-stage classification approach. The final solution achieved 88% precision and 83% recall while reducing operational complexity and costs compared to the initial proof of concept.
Vannevar Labs
Vannevar Labs needed to improve their sentiment analysis capabilities for defense intelligence across multiple languages, finding that GPT-4 provided insufficient accuracy (64%) and high costs. Using Databricks Mosaic AI, they successfully fine-tuned a Mistral 7B model on domain-specific data, achieving 76% accuracy while reducing latency by 75%. The entire process from development to deployment took only two weeks, enabling efficient processing of multilingual content for defense-related applications.
Apoidea Group
Apoidea Group tackled the challenge of efficiently processing banking documents by developing a solution using multimodal large language models. They fine-tuned the Qwen2-VL-7B-Instruct model using LLaMA-Factory on Amazon SageMaker HyperPod to enhance visual information extraction from complex banking documents. The solution significantly improved table structure recognition accuracy from 23.4% to 81.1% TEDS score, approaching the performance of more advanced models while maintaining computational efficiency. This enabled reduction of financial spreading process time from 4-6 hours to just 10 minutes.
Nubank
Nubank developed a sophisticated approach to customer behavior modeling by combining transformer-based transaction embeddings with tabular data through supervised fine-tuning and joint fusion training. Starting with self-supervised pre-trained foundation models for transaction data, they implemented a DCNv2-based architecture that incorporates numerical and categorical feature embeddings to blend sequential transaction data with traditional tabular features. This joint fusion approach, which simultaneously optimizes the transformer and blending model during fine-tuning, outperforms both late fusion methods and standalone LightGBM models, achieving measurable improvements in AUC across multiple benchmark tasks while eliminating the need for manual feature engineering from sequential transaction data.
Meta
Meta developed GEM (Generative Ads Recommendation Model), an LLM-scale foundation model trained on thousands of GPUs to enhance ads recommendation across Facebook and Instagram. The model addresses challenges of sparse signals in billions of daily user-ad interactions, diverse multimodal data, and efficient large-scale training. GEM achieves 4x efficiency improvement over previous models through novel architecture innovations including stackable factorization machines, pyramid-parallel sequence processing, and cross-feature learning. The system employs sophisticated post-training knowledge transfer techniques achieving 2x the effectiveness of standard distillation, propagating learnings across hundreds of vertical models. Since launch in early 2025, GEM delivered a 5% increase in ad conversions on Instagram and 3% on Facebook Feed in Q2, with Q3 architectural improvements doubling performance gains from additional compute and data.
Netflix
Netflix developed a foundation model approach to centralize and scale their recommendation system, transitioning from multiple specialized models to a unified architecture. The system processes hundreds of billions of user interactions, employing sophisticated tokenization, sparse attention mechanisms, and incremental training to handle cold-start problems and new content. The model demonstrates successful scaling properties similar to LLMs, while maintaining production-level latency requirements and addressing unique challenges in recommendation systems.
Netflix
Netflix developed a foundation model for personalized recommendations to address the maintenance complexity and inefficiency of operating numerous specialized recommendation models. The company built a large-scale transformer-based model inspired by LLM paradigms that processes hundreds of billions of user interactions from over 300 million users, employing autoregressive next-token prediction with modifications for recommendation-specific challenges. The foundation model enables centralized member preference learning that can be fine-tuned for specific tasks, used directly for predictions, or leveraged through embeddings, while demonstrating clear scaling law benefits as model and data size increase, ultimately improving recommendation quality across multiple downstream applications.
Netflix
Netflix developed a unified foundation model based on transformer architecture to consolidate their diverse recommendation systems, which previously consisted of many specialized models for different content types, pages, and use cases. The foundation model uses autoregressive transformers to learn user representations from interaction sequences, incorporating multi-token prediction, multi-layer representation, and long context windows. By scaling from millions to billions of parameters over 2.5 years, they demonstrated that scaling laws apply to recommendation systems, achieving notable performance improvements while creating high leverage across downstream applications through centralized learning and easier fine-tuning for new use cases.
Campfire AI
Drawing from experience building over 50 chatbots across five continents, this case study outlines four crucial lessons for successful chatbot implementation. Key insights include treating chatbot projects as AI initiatives rather than traditional IT projects, anticipating out-of-scope queries through "99-intents", organizing intents hierarchically for more natural interactions, planning for unusual user expressions, and eliminating unhelpful "I don't understand" responses. The study emphasizes that successful chatbots require continuous optimization, aiming for 90-95% recognition rates for in-scope questions, while maintaining effective fallback mechanisms for edge cases.
Scale Venture Partners
Barak Turovsky, drawing from his experience leading Google Translate and other AI initiatives, presents a framework for evaluating LLM use cases in production. The framework analyzes use cases based on two key dimensions: accuracy requirements and fluency needs, along with consideration of stakes involved. This helps organizations determine which applications are suitable for current LLM deployment versus those that need more development. The framework suggests creative and workplace productivity applications are better immediate fits for LLMs compared to high-stakes information/decision support use cases.
Uber
Uber faced significant challenges processing a high volume of invoices daily from thousands of global suppliers, with diverse formats, 25+ languages, and varying templates requiring substantial manual intervention. The company developed TextSense, a GenAI-powered document processing platform that leverages OCR, computer vision, and large language models (specifically OpenAI GPT-4 after evaluating multiple options including fine-tuned Llama 2 and Flan T5) to automate invoice data extraction. The solution achieved 90% overall accuracy, reduced manual processing by 2x, cut average handling time by 70%, and delivered 25-30% cost savings compared to manual processes, while providing a scalable, configuration-driven platform adaptable to diverse document types.
Google developed a three-generation evolution of AI-powered systems to transform 2D product images into interactive 3D visualizations for online shopping, culminating in a solution based on their Veo video generation model. The challenge was to replicate the tactile, hands-on experience of in-store shopping in digital environments while making the technology scalable and cost-effective for retailers. The latest approach uses Veo's diffusion-based architecture, fine-tuned on millions of synthetic 3D assets, to generate realistic 360-degree product spins from as few as one to three product images. This system now powers interactive 3D visualizations across multiple product categories on Google Shopping, significantly improving the online shopping experience by enabling customers to virtually inspect products from multiple angles.
NICE Actimize
NICE Actimize implemented generative AI into their financial crime detection platform "Excite" to create an automated machine learning model factory and enhance MLOps capabilities. They developed a system that converts natural language requests into analytical artifacts, helping analysts create aggregations, features, and models more efficiently. The solution includes built-in guardrails and validation pipelines to ensure safe deployment while significantly reducing time to market for analytical solutions.
Amazon
Amazon Prime Video addresses the challenge of differentiating their streaming platform in a crowded market by implementing multiple generative AI features powered by AWS services, particularly Amazon Bedrock. The solution encompasses personalized content recommendations, AI-generated episode recaps (X-Ray Recaps), real-time sports analytics insights, dialogue enhancement features, and automated video content understanding with metadata extraction. These implementations have resulted in improved content discoverability, enhanced viewer engagement through features that prevent spoilers while keeping audiences informed, deeper sports broadcast insights, increased accessibility through AI-enhanced audio, and enriched metadata for hundreds of thousands of marketing assets, collectively improving the overall streaming experience and reducing time spent searching for content.
Myriad Genetics
Myriad Genetics, a genetic testing and precision medicine provider, faced challenges processing thousands of healthcare documents daily with their existing Amazon Comprehend and Amazon Textract solution, which cost $15,000 monthly per business unit with 8.5-minute processing times and required manual information extraction involving up to 10 full-time employees. Partnering with AWS Generative AI Innovation Center, they deployed the open-source GenAI IDP Accelerator using Amazon Bedrock with Amazon Nova models, implementing advanced prompt engineering techniques including AI-driven prompt engineering, negative prompting, few-shot learning, and chain-of-thought reasoning. The solution increased classification accuracy from 94% to 98%, reduced classification costs by 77%, decreased processing time by 80% (from 8.5 to 1.5 minutes), and automated key information extraction at 90% accuracy, projected to save $132K annually while reducing prior authorization processing time by 2 minutes per submission.
Google Photos evolved from using on-device machine learning models for basic image editing features like background blur and object removal to implementing cloud-based generative AI for their Magic Editor feature. The team transitioned from small, specialized models (10MB) running locally on devices to large-scale generative models hosted in the cloud to enable more sophisticated image editing capabilities like scene reimagination, object relocation, and advanced inpainting. This shift required significant changes in infrastructure, capacity planning, evaluation methodologies, and user experience design while maintaining focus on grounded, memory-preserving edits rather than fantastical image generation.
Agoda
Agoda integrated GPT into their CI/CD pipeline to automate SQL stored procedure optimization, addressing a significant operational bottleneck where database developers were spending 366 man-days annually on manual optimization tasks. The system provides automated analysis and suggestions for query improvements, index recommendations, and performance optimizations, leading to reduced manual review time and improved merge request processing. While achieving approximately 25% accuracy, the solution demonstrates practical benefits in streamlining database development workflows despite some limitations in handling complex stored procedures.
Salesforce
Salesforce's AI Platform team addressed the challenge of inefficient GPU utilization and high costs when hosting multiple proprietary large language models (LLMs) including CodeGen on Amazon SageMaker. They implemented SageMaker AI inference components to deploy multiple foundation models on shared endpoints with granular resource allocation, enabling dynamic scaling and intelligent model packing. This solution achieved up to an eight-fold reduction in deployment and infrastructure costs while maintaining high performance standards, allowing smaller models to efficiently utilize high-performance GPUs and optimizing resource allocation across their diverse model portfolio.
Meta
Meta faced significant challenges with AI model training as checkpoint data grew from hundreds of gigabytes to tens of terabytes, causing network bottlenecks and GPU idle time. Their solution involved implementing bidirectional multi-NIC utilization through ECMP-based load balancing for egress traffic and BGP-based virtual IP injection for ingress traffic, enabling optimal use of all available network interfaces. The implementation resulted in dramatic performance improvements, reducing job read latency from 300 seconds to 1 second and checkpoint loading time from 800 seconds to 100 seconds, while achieving 4x throughput improvement through proper traffic distribution across multiple network interfaces.
Perplexity
A technical exploration of achieving high-performance GPU memory transfer speeds (up to 3200 Gbps) on AWS SageMaker Hyperpod infrastructure, demonstrating the critical importance of optimizing memory bandwidth for large language model training and inference workloads.
Salesforce
Salesforce's AI Model Serving team tackled the challenge of deploying and optimizing large language models at scale while maintaining performance and security. Using Amazon SageMaker AI and Deep Learning Containers, they developed a comprehensive hosting framework that reduced model deployment time by 50% while achieving high throughput and low latency. The solution incorporated automated testing, security measures, and continuous optimization techniques to support enterprise-grade AI applications.
Walmart
Walmart developed Ghotok, an innovative AI system that combines predictive and generative AI to improve product categorization across their digital platforms. The system addresses the challenge of accurately mapping relationships between product categories and types across 400 million SKUs. Using an ensemble approach with both predictive and generative AI models, along with sophisticated caching and deployment strategies, Ghotok successfully reduces false positives and improves the efficiency of product categorization while maintaining fast response times in production.
Bank CenterCredit (BCC)
Bank CenterCredit (BCC), a leading Kazakhstan bank with over 3 million clients, implemented a hybrid multi-cloud architecture using AWS Outpost to deploy generative AI and machine learning services while maintaining strict regulatory compliance. The bank faced requirements that all data must be encrypted with locally stored keys and customer data must be anonymized during processing. They developed two primary use cases: fine-tuning an automatic speech recognition (ASR) model for Kazakh-Russian mixed language processing that achieved 23% accuracy improvement and $4M monthly savings, and deploying an internal HR chatbot using a hybrid RAG architecture with Amazon Bedrock that now handles 70% of HR requests. Both solutions leveraged their hybrid architecture where sensitive data processing occurs on-premise on AWS Outpost while compute-intensive model training utilizes cloud GPU resources.
Rio Tinto
Rio Tinto Aluminium faced challenges in providing technical experts in refining and smelting sectors with quick and accurate access to vast amounts of specialized institutional knowledge during their internal training programs. They developed a generative AI-powered knowledge assistant using hybrid RAG (retrieval augmented generation) on Amazon Bedrock, combining both vector search and knowledge graph databases to enable more accurate, contextually rich responses. The hybrid system significantly outperformed traditional vector-only RAG across all metrics, particularly in context quality and entity recall, showing over 53% reduction in standard deviation while maintaining high mean scores, and leveraging 11-17 technical documents per query compared to 2-3 for vector-only approaches, ultimately streamlining how employees find and utilize critical business information.
GitHub
GitHub's machine learning team worked to enhance GitHub Copilot's contextual understanding of code to provide more relevant AI-powered coding suggestions. The problem was that large language models could only process limited context (approximately 6,000 characters), making it challenging to leverage all relevant information from a developer's codebase. The solution involved sophisticated prompt engineering, implementing neighboring tabs to process multiple open files, introducing a Fill-In-the-Middle (FIM) paradigm to consider code both before and after the cursor, and experimenting with vector databases and embeddings for semantic code retrieval. These improvements resulted in measurable gains: neighboring tabs provided a 5% relative increase in suggestion acceptance, FIM yielded a 10% relative boost in performance, and the overall enhancements contributed to developers coding up to 55% faster when using GitHub Copilot.
Various
This panel discussion brings together infrastructure experts from Groq, NVIDIA, Lambda, and AMD to discuss the unique challenges of deploying AI agents in production. The panelists explore how agentic AI differs from traditional AI workloads, requiring significantly higher token generation, lower latency, and more diverse infrastructure spanning edge to cloud. They discuss the evolution from training-focused to inference-focused infrastructure, emphasizing the need for efficiency at scale, specialized hardware optimization, and the importance of smaller distilled models over large monolithic models. The discussion highlights critical operational challenges including power delivery, thermal management, and the need for full-stack engineering approaches to debug and optimize agentic systems in production environments.
Netflix
Netflix developed a centralized foundation model for personalization to replace multiple specialized models powering their homepage recommendations. Rather than maintaining numerous individual models, they created one powerful transformer-based model trained on comprehensive user interaction histories and content data at scale. The challenge then became how to effectively integrate this large foundation model into existing production systems. Netflix experimented with and deployed three distinct integration approaches—embeddings via an Embedding Store, using the model as a subgraph within downstream models, and direct fine-tuning for specific applications—each with different tradeoffs in terms of latency, computational cost, freshness, and implementation complexity. These approaches are now used in production across different Netflix personalization use cases based on their specific requirements.
Mendix
Mendix, a low-code platform provider, faced the challenge of integrating advanced generative AI capabilities into their development environment while maintaining security and scalability. They implemented Amazon Bedrock to provide their customers with seamless access to various AI models, enabling features like text generation, summarization, and multimodal image generation. The solution included custom model training, robust security measures through AWS services, and cost-effective model selection capabilities.
Wix
Wix is leveraging AI technologies, including LLMs and diffusion models, to automate and enhance the website building experience. Their AI group has developed the AI Text Creator suite using LLMs for content generation, integrated DALL-E for image creation, and introduced the Diffusion Layout Transformer (DLT) for automated layout generation. This comprehensive approach combines content generation with layout design, addressing the challenge of creating professional websites without requiring extensive design expertise.
Taralli
Taralli, a calorie tracking application, demonstrates systematic LLM improvement through rigorous evaluation and prompt optimization. The developer addressed the challenge of accurate nutritional estimation by creating a 107-example evaluation dataset, testing multiple prompt optimization techniques (vanilla, few-shot bootstrapping, MIPROv2, and GEPA) across several models (Gemini 2.5 Flash, Gemini 3 Flash, and DeepSeek v3.2). Through this methodical approach, they achieved a 15% accuracy improvement by switching from Gemini 2.5 Flash to Gemini 3 Flash while using a few-shot learning approach with 16 examples, reaching 60% accuracy within a 10% calorie prediction threshold. The system was deployed with fallback model configurations and extended to support fully offline on-device inference for iOS.
BT
BT is undertaking a major transformation of their network operations, moving from traditional telecom engineering to a software-driven approach with the goal of creating an autonomous "Dark NOC" (Network Operations Center). The initiative focuses on handling massive amounts of network data, implementing AI/ML for automated analysis and decision-making, and consolidating numerous specialized tools into a comprehensive intelligent system. The project involves significant organizational change, including upskilling teams and partnering with AWS to build data foundations and AI capabilities for predictive maintenance and autonomous network management.
LinkedIn developed JUDE (Job Understanding Data Expert), a production platform that leverages fine-tuned large language models to generate high-quality embeddings for job recommendations at scale. The system addresses the computational challenges of LLM deployment through a multi-component architecture including fine-tuned representation learning, real-time embedding generation, and comprehensive serving infrastructure. JUDE replaced standardized features in job recommendation models, resulting in +2.07% qualified applications, -5.13% dismiss-to-apply ratio, and +1.91% total job applications - representing the highest metric improvement from a single model change observed by the team.
Various
A panel discussion between experienced Kubernetes and ML practitioners exploring the challenges and opportunities of running LLMs on Kubernetes. The discussion covers key aspects including GPU management, cost optimization, training vs inference workloads, and architectural considerations. The panelists share insights from real-world implementations while highlighting both benefits (like workload orchestration and vendor agnosticism) and challenges (such as container sizes and startup times) of using Kubernetes for LLM operations.
LinkedIn developed a large foundation model called "Brew XL" with 150 billion parameters to unify all personalization and recommendation tasks across their platform, addressing the limitations of task-specific models that operate in silos. The solution involved training a massive language model on user interaction data through "promptification" techniques, then distilling it down to smaller, production-ready models (3B parameters) that could serve high-QPS recommendation systems with sub-second latency. The system demonstrated zero-shot capabilities for new tasks, improved performance on cold-start users, and achieved 7x latency reduction with 30x throughput improvement through optimization techniques including distillation, pruning, quantization, and sparsification.
Pinterest tackled the challenge of improving search relevance by implementing a large language model-based system. They developed a cross-encoder LLM teacher model trained on human-annotated data, which was then distilled into a lightweight student model for production deployment. The system processes rich Pin metadata including titles, descriptions, and synthetic image captions to predict relevance scores. The implementation resulted in a 2.18% improvement in search feed relevance (nDCG@20) and over 1.5% increase in search fulfillment rates globally, while successfully generalizing across multiple languages despite being trained primarily on US data.
Various
A panel of experts from various companies and backgrounds discusses the challenges and solutions of deploying LLMs in production. They explore three main themes: latency considerations in LLM deployments, cost optimization strategies, and building trust in LLM systems. The discussion includes practical examples from Digits, which uses LLMs for financial document processing, and insights from other practitioners about model optimization, deployment strategies, and the evolution of LLM architectures.
Google / YouTube
YouTube developed Large Recommender Models (LRM) by adapting Google's Gemini LLM for video recommendations, addressing the challenge of serving personalized content to billions of users. The solution involved creating semantic IDs to tokenize videos, continuous pre-training to teach the model both English and YouTube-specific video language, and implementing generative retrieval systems. While the approach delivered significant improvements in recommendation quality, particularly for challenging cases like new users and fresh content, the team faced substantial serving cost challenges that required 95%+ cost reductions and offline inference strategies to make production deployment viable at YouTube's scale.
Apple
Apple developed and deployed a comprehensive foundation model infrastructure consisting of a 3-billion parameter on-device model and a mixture-of-experts server model to power Apple Intelligence features across iOS, iPadOS, and macOS. The implementation addresses the challenge of delivering generative AI capabilities at consumer scale while maintaining privacy, efficiency, and quality across 15 languages. The solution involved novel architectural innovations including shared KV caches, parallel track mixture-of-experts design, and extensive optimization techniques including quantization and compression, resulting in production deployment across millions of devices with measurable performance improvements in text and vision tasks.
CommBank
Commonwealth Bank of Australia (CBA), Australia's largest bank serving 17.5 million customers, faced the challenge of modernizing decades of rich data spread across hundreds of on-premise source systems that lacked interoperability and couldn't scale for AI workloads. In partnership with HCL Tech and AWS, CBA migrated 61,000 on-premise data pipelines (equivalent to 10 petabytes of data) to an AWS-based data mesh ecosystem in 9 months. The solution leveraged AI and generative AI to transform code, check for errors, and test outputs with 100% accuracy reconciliation, conducting 229,000 tests across the migration. This enabled CBA to establish a federated data architecture called CommBank.data that empowers 40 lines of business with self-service data access while maintaining strict governance, positioning the bank for AI-driven innovation at scale.
AWS GENAIC (Japan)
Japan's GENIAC program partnered with AWS to provide 12 organizations with massive compute resources (127 P5 instances and 24 Trn1 instances) for foundation model development. The challenge revealed that successful FM training required far more than raw hardware access - it demanded structured organizational support, reference architectures, cross-functional teams, and comprehensive enablement programs. Through systematic deployment guides, monitoring infrastructure, and dedicated communication channels, multiple large-scale models were successfully trained including 100B+ parameter models, demonstrating that large-scale AI development is fundamentally an organizational rather than purely technical challenge.
Exa.ai
Exa.ai built a sophisticated GPU infrastructure combining a new 144 H200 GPU cluster with their existing 80 A100 GPU cluster to support their neural web search and retrieval models. They implemented a five-layer infrastructure stack using Pulumi, Ansible/Kubespray, NVIDIA operators, Alluxio for storage, and Flyte for orchestration, enabling efficient large-scale model training and inference while maintaining reproducibility and reliability.
Pinterest developed and deployed a large-scale learned retrieval system using a two-tower architecture to improve content recommendations for over 500 million monthly active users. The system replaced traditional heuristic approaches with an embedding-based retrieval system learned from user engagement data. The implementation includes automatic retraining capabilities and careful version synchronization between model artifacts. The system achieved significant success, becoming one of the top-performing candidate generators with the highest user coverage and ranking among the top three in save rates.
Coupang
Coupang, a major e-commerce platform operating primarily in South Korea and Taiwan, faced challenges in scaling their ML infrastructure to support LLM applications across search, ads, catalog management, and recommendations. The company addressed GPU supply shortages and infrastructure limitations by building a hybrid multi-region architecture combining cloud and on-premises clusters, implementing model parallel training with DeepSpeed, and establishing GPU-based serving using Nvidia Triton and vLLM. This infrastructure enabled production applications including multilingual product understanding, weak label generation at scale, and unified product categorization, with teams using patterns ranging from in-context learning to supervised fine-tuning and continued pre-training depending on resource constraints and quality requirements.
DoorDash
DoorDash faced challenges in scaling personalization and maintaining product catalogs as they expanded beyond restaurants into new verticals like grocery, retail, and convenience stores, dealing with millions of SKUs and cold-start scenarios for new customers and products. They implemented a layered approach combining traditional machine learning with fine-tuned LLMs, RAG systems, and LLM agents to automate product knowledge graph construction, enable contextual personalization, and provide recommendations even without historical user interaction data. The solution resulted in faster, more cost-effective catalog processing, improved personalization for cold-start scenarios, and the foundation for future agentic shopping experiences that can adapt to real-time contexts like emergency situations.
Uber
Uber Eats built a production-grade semantic search platform to improve discovery across restaurants, grocery, and retail items by addressing limitations of traditional lexical search. The solution leverages LLM-based embeddings (using Qwen as the backbone), a two-tower architecture with Matryoshka Representation Learning, and Apache Lucene Plus for indexing. Through careful optimization of ANN parameters, quantization strategies, and embedding dimensions, the team achieved significant cost reductions (34% latency reduction, 17% CPU savings, 50% storage reduction) while maintaining high recall (>0.95). The system features automated biweekly model updates with blue/green deployment, comprehensive validation gates, and serving-time reliability checks to ensure production stability at global scale.
ByteDance
ByteDance implemented multimodal LLMs for video understanding at massive scale, processing billions of videos daily for content moderation and understanding. By deploying their models on AWS Inferentia2 chips across multiple regions, they achieved 50% cost reduction compared to standard EC2 instances while maintaining high performance. The solution combined tensor parallelism, static batching, and model quantization techniques to optimize throughput and latency.
NICE Actimize
NICE Actimize, a leader in financial fraud prevention, implemented a scalable approach using vector embeddings to enhance their fraud detection capabilities. They developed a pipeline that converts tabular transaction data into meaningful text representations, then transforms them into vector embeddings using RoBERTa variants. This approach allows them to capture semantic similarities between transactions while maintaining high performance requirements for real-time fraud detection.
QuantumBlack
QuantumBlack presented two distinct LLM applications: molecular discovery for pharmaceutical research and call center analytics for banking. The molecular discovery system used chemical language models and RAG to analyze scientific literature and predict molecular properties. The call center analytics solution processed audio files through a pipeline of diarization, transcription, and LLM analysis to extract insights from customer calls, achieving 60x performance improvement through domain-specific optimizations and efficient resource utilization.
Airbnb
Airbnb implemented AI text generation models across three key customer support areas: content recommendation, real-time agent assistance, and chatbot paraphrasing. They leveraged large language models with prompt engineering to encode domain knowledge from historical support data, resulting in significant improvements in content relevance, agent efficiency, and user engagement. The implementation included innovative approaches to data preparation, model training with DeepSpeed, and careful prompt design to overcome common challenges like generic responses.
Various
Leaders from three major EdTech companies share their experiences implementing LLMs in production for language learning, coding education, and homework help. They discuss challenges around cost-effective scaling, fact generation accuracy, and content personalization, while highlighting successful approaches like retrieval-augmented generation, pre-generation of options, and using LLMs to create simpler production rules. The companies focus on using AI not just for content generation but for improving the actual teaching and learning experience.
Instacart
Instacart's search and machine learning team implemented LLMs to transform their search and discovery capabilities in grocery e-commerce, addressing challenges with tail queries and product discovery. They used LLMs to enhance query understanding models, including query-to-category classification and query rewrites, by combining LLM world knowledge with Instacart-specific domain knowledge and user behavior data. The hybrid approach involved batch pre-computing results for head/torso queries while using real-time inference for tail queries, resulting in significant improvements: 18 percentage point increase in precision and 70 percentage point increase in recall for tail queries, along with substantial reductions in zero-result queries and enhanced user engagement with discovery-oriented content.
DoorDash
DoorDash evolved from traditional numerical embeddings to LLM-generated natural language profiles for representing consumers, merchants, and food items to improve personalization and explainability. The company built an automated system that generates detailed, human-readable profiles by feeding structured data (order history, reviews, menu metadata) through carefully engineered prompts to LLMs, enabling transparent recommendations, editable user preferences, and richer input for downstream ML models. While the approach offers scalability and interpretability advantages over traditional embeddings, the implementation requires careful evaluation frameworks, robust serving infrastructure, and continuous iteration cycles to maintain profile quality in production.
Spotify
Spotify implemented LLMs to enhance their recommendation system by providing contextualized explanations for music recommendations and powering their AI DJ feature. They adapted Meta's Llama models through careful domain adaptation, human-in-the-loop training, and multi-task fine-tuning. The implementation resulted in up to 4x higher user engagement for recommendations with explanations, and a 14% improvement in Spotify-specific tasks compared to baseline Llama performance. The system was deployed at scale using vLLM for efficient serving and inference.
Pinterest Search faced significant limitations in measuring search relevance due to the high cost and low availability of human annotations, which resulted in large minimum detectable effects (MDEs) that could only identify significant topline metric movements. To address this, they fine-tuned open-source multilingual LLMs on human-annotated data to predict relevance scores on a 5-level scale, then deployed these models to evaluate ranking results across A/B experiments. This approach reduced labeling costs dramatically, enabled stratified query sampling designs, and achieved an order of magnitude reduction in MDEs (from 1.3-1.5% down to ≤0.25%), while maintaining strong alignment with human labels (73.7% exact match, 91.7% within 1 point deviation) and enabling rapid evaluation of 150,000 rows within 30 minutes on a single GPU.
LeBonCoin
leboncoin, France's largest second-hand marketplace, implemented a neural re-ranking system using large language models to improve search relevance across their 60 million classified ads. The system uses a two-tower architecture with separate Ad and Query encoders based on fine-tuned LLMs, achieving up to 5% improvement in click and contact rates and 10% improvement in user experience KPIs while maintaining strict latency requirements for their high-throughput search system.
Gerdau
Gerdau, a major steel manufacturer, implemented an LLM-based assistant to support employee re/upskilling as part of their broader digital transformation initiative. This development came after transitioning to the Databricks Data Intelligence Platform to solve data infrastructure challenges, which enabled them to explore advanced AI applications. The platform consolidation resulted in a 40% cost reduction in data processing and allowed them to onboard 300 new global data users while creating an environment conducive to AI innovation.
Cambrium
Cambrium is using LLMs and AI to design and generate novel proteins for sustainable materials, starting with vegan human collagen for cosmetics. They've developed a protein programming language and leveraged LLMs to transform protein design into a mathematical optimization problem, enabling them to efficiently search through massive protein sequence spaces. Their approach combines traditional protein engineering with modern LLM techniques, resulting in successfully bringing a biotech product to market in under two years.
Amazon (Alexa)
At Amazon Alexa, researchers tackled two key challenges in production NLP models: preventing performance degradation on common utterances during model updates and improving model robustness to input variations. They implemented positive congruent training to minimize negative prediction flips between model versions and used T5 models to generate synthetic training data variations, making the system more resilient to slight changes in user commands while maintaining consistent performance.
Oracle
A comparative study evaluating different LLM models (Claude, GPT-4, LLaMA, and Pi 3.1) for medical transcript summarization aimed at reducing administrative burden in healthcare. The study processed over 5,000 medical transcripts, comparing model performance using ROUGE scores and cosine similarity metrics. GPT-4 emerged as the top performer, followed by Pi 3.1, with results showing potential to reduce care coordinator preparation time by over 50%.
Meta
Meta addresses the critical challenge of hardware reliability in large-scale AI infrastructure, where hardware faults significantly impact training and inference workloads. The company developed comprehensive detection mechanisms including Fleetscanner, Ripple, and Hardware Sentinel to identify silent data corruptions (SDCs) that can cause training divergence and inference errors without obvious symptoms. Their multi-layered approach combines infrastructure strategies like reductive triage and hyper-checkpointing with stack-level solutions such as gradient clipping and algorithmic fault tolerance, achieving industry-leading reliability for AI operations across thousands of accelerators and globally distributed data centers.
Adept.ai
Adept.ai, building an AI model for computer interaction, faced challenges with complex fine-tuning pipelines running on Slurm. They implemented a migration strategy to Kubernetes using Metaflow and Argo for workflow orchestration, while maintaining existing Slurm workloads through a hybrid approach. This allowed them to improve pipeline management, enable self-service capabilities for data scientists, and establish robust monitoring infrastructure, though complete migration to Kubernetes remains a work in progress.
Baseten
Baseten has built a production-grade LLM inference platform focusing on three key pillars: model-level performance optimization, horizontal scaling across regions and clouds, and enabling complex multi-model workflows. The platform supports various frameworks including SGLang and TensorRT-LLM, and has been successfully deployed by foundation model companies and enterprises requiring strict latency, compliance, and reliability requirements. A key differentiator is their ability to handle mission-critical inference workloads with sub-400ms latency for complex use cases like AI phone calls.
Atlassian
Atlassian developed a machine learning-based comment ranker to improve the quality of their LLM-powered code review agent by filtering out noisy, incorrect, or unhelpful comments. The system uses a fine-tuned ModernBERT model trained on proprietary data from over 53K code review comments to predict which LLM-generated comments will lead to actual code changes. The solution improved code resolution rates from ~33% to 40-45%, approaching human reviewer performance of 45%, while maintaining robustness across different underlying LLMs and user bases, ultimately reducing PR cycle times by 30% and serving over 10K monthly active users reviewing 43K+ pull requests.
MLflow
MLflow addresses the challenges of moving LLM agents from demo to production by introducing comprehensive tooling for tracing, evaluation, and experiment tracking. The solution includes LLM tracing capabilities to debug black-box agent systems, evaluation tools for retrieval relevance and prompt engineering, and integrations with popular agent frameworks like Autogen and LlamaIndex. This enables organizations to effectively monitor, debug, and improve their LLM-based applications in production environments.
Various
The case study explores MLOps maturity levels (0-2) in enterprise settings, discussing how organizations progress from manual ML deployments to fully automated systems. It covers the challenges of implementing MLOps across different team personas (data scientists, ML engineers, DevOps), highlighting key considerations around automation, monitoring, compliance, and business value metrics. The study particularly emphasizes the differences between traditional ML and LLM deployments, and how organizations need to adapt their MLOps practices for each.
LATAM Airlines
LATAM Airlines developed Cosmos, a vendor-agnostic MLOps framework that enables both traditional ML and LLM deployments across their business operations. The framework reduced model deployment time from 3-4 months to less than a week, supporting use cases from fuel efficiency optimization to personalized travel recommendations. The platform demonstrates how a traditional airline can transform into a data-driven organization through effective MLOps practices and careful integration of AI technologies.
Meta
This case study presents a sophisticated multi-agent LLM system designed to identify, correct, and find the root causes of misinformation on social media platforms at scale. The solution addresses the limitations of pre-LLM era approaches (content-only features, no real-time information, low precision/recall) by deploying specialized agents including an Indexer (for sourcing authentic data), Extractor (adaptive retrieval and reranking), Classifier (discriminative misinformation categorization), Corrector (reasoning and correction generation), and Verifier (final validation). The system achieves high precision and recall by orchestrating these agents through a centralized coordinator, implementing comprehensive logging, evaluation at both individual agent and system levels, and optimization strategies including model distillation, semantic caching, and adaptive retrieval. The approach prioritizes accuracy over cost and latency given the high stakes of misinformation propagation on platforms.
Various (Thinking Machines, Yutori, Evolutionaryscale, Perplexity, Axiom)
This panel discussion features experts from multiple AI companies discussing the current state and future of agentic frameworks, reinforcement learning applications, and production LLM deployment challenges. The panelists from Thinking Machines, Perplexity, Evolutionary Scale AI, and Axiom share insights on framework proliferation, the role of RL in post-training, domain-specific applications in mathematics and biology, and infrastructure bottlenecks when scaling models to hundreds of GPUs, highlighting the gap between research capabilities and production deployment tools.
Meta / AWS / NVIDIA / ConverseNow
This panel discussion features leaders from Meta, AWS, NVIDIA, and ConverseNow discussing real-world challenges and solutions for deploying LLMs in production environments. The conversation covers the trade-offs between small and large language models, with ConverseNow sharing their experience building voice AI systems for restaurants that require high accuracy and low latency. Key themes include the importance of fine-tuning small models for production use cases, the convergence of training and inference systems, optimization techniques like quantization and alternative architectures, and the challenges of building reliable, cost-effective inference stacks for mission-critical applications.
AMD / Somite AI / Upstage / Rambler AI
This panel discussion at AWS re:Invent features three companies deploying AI models in production across different industries: Somite AI using machine learning for computational biology and cellular control, Upstage developing sovereign AI with proprietary LLMs and OCR for document extraction in enterprises, and Rambler AI building vision language models for industrial task verification. All three leverage AMD GPU infrastructure (MI300 series) for training and inference, emphasizing the importance of hardware choice, open ecosystems, seamless deployment, and cost-effective scaling. The discussion highlights how smaller, domain-specific models can achieve enterprise ROI where massive frontier models failed, and explores emerging areas like physical AI, world models, and data collection for robotics.
Caylent
Caylent, a development consultancy, shares their extensive experience building production LLM systems across multiple industries including environmental management, sports media, healthcare, and logistics. The presentation outlines their comprehensive approach to LLMOps, emphasizing the importance of proper evaluation frameworks, prompt engineering over fine-tuning, understanding user context, and managing inference economics. Through various client projects ranging from multimodal video search to intelligent document processing, they demonstrate key lessons learned about deploying reliable AI systems at scale, highlighting that generative AI is not a "magical pill" but requires careful engineering around inputs, outputs, evaluation, and user experience.
Addverb
Addverb developed an AI-powered voice control system for AGV (Automated Guided Vehicle) maintenance that enables warehouse workers to communicate with robots in their native language. The system uses a combination of edge-deployed Llama 3 and cloud-based ChatGPT to translate natural language commands from 98 different languages into AGV instructions, significantly reducing maintenance downtime and improving operational efficiency.
Mercado Libre
Mercado Libre tackled the classic e-commerce product-matching challenge where sellers create listings with inconsistent titles, attributes, and identifiers, making it difficult to identify identical products across the platform. The team developed a sophisticated multi-LLM orchestration system that evolved from a simple 2-node architecture to a complex 7-node pipeline, incorporating adaptive prompts, context-aware decision-making, and collaborative consensus mechanisms. Through systematic iteration and careful orchestration alongside existing ML models and embedding systems, they achieved human-level performance with 95% precision and over 50% recall at a cost-effective rate of less than $0.001 per request, enabling scalable autonomous product matching across millions of items for critical use cases including pricing, personalization, and inventory optimization.
Convirza
Convirza, facing challenges with their customer service agent evaluation system, transitioned from Longformer models to fine-tuned Llama-3-8b using Predibase's multi-LoRA serving infrastructure. This shift enabled them to process millions of call hours while reducing operational costs by 10x compared to OpenAI, achieving an 8% improvement in F1 scores, and increasing throughput by 80%. The solution allowed them to efficiently serve over 60 performance indicators across thousands of customer interactions daily while maintaining sub-second inference times.
Upwork
Upwork, a global freelance talent marketplace, developed Uma (Upwork's Mindful AI) to streamline the hiring and matching processes between clients and freelancers. The company faced the challenge of serving a large, diverse customer base with AI solutions that needed both broad applicability and precision for specific marketplace use cases like discovery, search, and matching. Their solution involved a dual approach: leveraging pretrained models like GPT-4 for rapid deployment of features such as job post generation and chat assistance, while simultaneously developing custom, use case-specific smaller language models fine-tuned on proprietary platform data, synthetic data, and human-generated content from talented writers. This strategy resulted in significant improvements, including an 80% reduction in job post creation time and more accurate, contextually relevant assistance for both freelancers and clients across the platform.
Rufus
Amazon's Rufus team faced the challenge of deploying increasingly large custom language models for their generative AI shopping assistant serving millions of customers. As model complexity grew beyond single-node memory capacity, they developed a multi-node inference solution using AWS Trainium chips, vLLM, and Amazon ECS. Their solution implements a leader/follower architecture with hybrid parallelism strategies (tensor and data parallelism), network topology-aware placement, and containerized multi-node inference units. This enabled them to successfully deploy across tens of thousands of Trainium chips, supporting Prime Day traffic while delivering the performance and reliability required for production-scale conversational AI.
Intercom
YouTube, a Google company, implements a comprehensive multilingual navigation and localization system for its global platform. The source text appears to be in Dutch, demonstrating the platform's localization capabilities, though insufficient details are provided about the specific LLMOps implementation.
Runway
Runway, a leader in generative AI for creative tools, developed a novel approach to managing multimodal training data through what they call a "multimodal feature store". This system enables efficient storage and retrieval of diverse data types (video, images, text) along with their computed features and embeddings, facilitating large-scale distributed training while maintaining researcher productivity. The solution addresses challenges in data management, feature computation, and the research-to-production pipeline, while fostering better collaboration between researchers and engineers.
Vodafone
Vodafone implemented a comprehensive AI and GenAI strategy to transform their network operations, focusing on improving customer experience through better network management. They migrated from legacy OSS systems to a cloud-based infrastructure on Google Cloud Platform, integrating over 2 petabytes of network data with commercial and IT data. The initiative includes AI-powered network investment planning, automated incident management, and device analytics, resulting in significant operational efficiency improvements and a planned 50% reduction in OSS tools.
New Relic
New Relic, a major observability platform processing 7 petabytes of data daily, implemented GenAI both internally for developer productivity and externally in their product offerings. They achieved a 15% increase in developer productivity through targeted GenAI implementations, while also developing sophisticated AI monitoring capabilities and natural language interfaces for their customers. Their approach balanced cost, accuracy, and performance through a mix of RAG, multi-model routing, and classical ML techniques.
Google Research developed an on-device grammar correction system for Gboard on Pixel 6 that detects and suggests corrections for grammatical errors as users type. The solution addresses the challenge of implementing neural grammar correction within the constraints of mobile devices (limited memory, computational power, and latency requirements) while preserving user privacy by keeping all processing local. The team built a 20MB hybrid Transformer-LSTM model using hard distillation from a cloud-based system, achieving inference on 60 characters in under 22ms on the Pixel 6 CPU, enabling real-time grammar correction for both complete sentences and partial sentence prefixes across English text in nearly any app using Gboard.
Grammarly
Grammarly developed an on-device machine learning model for their iOS keyboard that learns users' personal vocabulary and provides personalized autocorrection suggestions without sending data to the cloud. The challenge was to build a model that could distinguish between valid personal vocabulary and typos while operating within severe mobile constraints (under 5 MB RAM, minimal latency). The solution involved memory-mapped storage, time-based decay functions for vocabulary management, noisy input filtering, and edit-distance-based frequency thresholding to verify new words. Deployed to over 5 million devices, the model demonstrated measurable improvements with decreased rates of reverted suggestions and increased acceptance rates, while maintaining minimal memory footprint and responsive performance.
Grammarly
Grammarly developed a compact 1B-parameter on-device LLM to provide offline spelling and grammar correction capabilities, addressing the challenge of maintaining writing assistance functionality without internet connectivity. The team selected Llama as the base model, created comprehensive synthetic training data covering diverse writing styles and error types, and applied extensive optimizations including Grouped Query Attention, MLX framework integration for Apple silicon, and 4-bit quantization. The resulting model achieves 210 tokens/second on M2 Mac hardware while maintaining correction quality, demonstrating that multiple specialized models can be consolidated into a single efficient on-device solution that preserves user voice and delivers real-time feedback.
Cursor
Cursor developed a production LLM system called Cursor Tab that predicts developer actions and suggests code completions across codebases, handling over 400 million requests per day. To address the challenge of noisy suggestions that disrupt developer flow, they implemented an online reinforcement learning approach using policy gradient methods that directly optimizes the model to show suggestions only when acceptance probability exceeds a target threshold. This approach required building infrastructure for rapid model deployment and on-policy data collection with a 1.5-2 hour turnaround cycle. The resulting model achieved a 21% reduction in suggestions shown while simultaneously increasing the accept rate by 28%, demonstrating effective LLMOps practices for continuously improving production models using real-time user feedback.
Various (Alation, GrottoAI, Nvidia, OLX)
This panel discussion brings together experts from Nvidia, OLX, Alation, and GrottoAI to discuss practical considerations for deploying agentic AI systems in production. The conversation explores when to choose open source versus closed source tooling, the challenges of standardizing agent frameworks across enterprise organizations, and the tradeoffs between abstraction levels in agent orchestration platforms. Key themes include starting with closed source models for rapid prototyping before transitioning to open source for compliance and cost reasons, the importance of observability across heterogeneous agent frameworks, the difficulty of enabling non-technical users to build agents, and the critical difference between internal tooling with lower precision requirements versus customer-facing systems demanding 95%+ accuracy.
Boltz
Boltz, founded by Gabriele Corso and Jeremy Wohlwend, developed an open-source suite of AI models (Boltz-1, Boltz-2, and BoltzGen) for structural biology and protein design, democratizing access to capabilities previously held by proprietary systems like AlphaFold 3. The company addresses the challenge of predicting complex molecular interactions (protein-ligand, protein-protein) and designing novel therapeutic proteins by combining generative diffusion models with specialized equivariant architectures. Their approach achieved validated nanomolar binders for two-thirds of nine previously unseen protein targets, demonstrating genuine generalization beyond training data. The newly launched Boltz Lab platform provides a production-ready infrastructure with optimized GPU kernels running 10x faster than open-source versions, offering agents for protein and small molecule design with collaborative interfaces for medicinal chemists and researchers.
Convirza
Convirza transformed their call center analytics platform from using traditional large language models to implementing small language models (specifically Llama 3B) with adapter-based fine-tuning. By partnering with Predibase, they achieved a 10x cost reduction compared to OpenAI while improving accuracy by 8% and throughput by 80%. The system analyzes millions of calls monthly, extracting hundreds of custom indicators for agent performance and caller behavior, with sub-0.1 second inference times using efficient multi-adapter serving on single GPUs.
H2O.ai
H2O.ai, an enterprise AI platform provider delivering both generative and predictive AI solutions, faced significant challenges with their AWS EBS storage infrastructure that supports model training and AI workloads running on Kubernetes. The company was managing over 2 petabytes of storage with poor utilization rates (around 25%), leading to substantial cloud costs and limited ability to scale efficiently. They implemented Datafi, an autonomous storage management solution that dynamically scales EBS volumes up and down based on actual usage without downtime. The solution integrated seamlessly with their existing Kubernetes, Terraform, and GitOps workflows, ultimately improving storage utilization to 80% and reducing their storage footprint from 2 petabytes to less than 1 petabyte while simultaneously improving performance for customers.
Moveworks
Moveworks addressed latency challenges in their enterprise Copilot by implementing NVIDIA's TensorRT-LLM optimization engine. The integration resulted in significant performance improvements, including a 2.3x increase in token processing speed (from 19 to 44 tokens per second), a reduction in average request latency from 3.4 to 1.5 seconds, and nearly 3x faster time to first token. These optimizations enabled more natural conversations and improved resource utilization in production.
Rolls-Royce
Rolls-Royce collaborated with Databricks to enhance their design space exploration capabilities using conditional Generative Adversarial Networks (cGANs). The project aimed to leverage legacy simulation data to identify and assess innovative design concepts without requiring traditional geometry modeling and simulation processes. By implementing cGANs on the Databricks platform, they successfully developed a system that could handle multi-objective constraints and optimize design processes while maintaining compliance with aerospace industry requirements.
LinkedIn developed Liger-Kernel, a library to optimize GPU performance during LLM training by addressing memory access and per-operation bottlenecks. Using techniques like FlashAttention and operator fusion implemented in Triton, the library achieved a 60% reduction in memory usage, 20% improvement in multi-GPU training throughput, and a 3x reduction in end-to-end training time.
Replit
Replit faced challenges with running LLM inference on expensive GPU infrastructure and implemented a solution using preemptable cloud GPUs to reduce costs by two-thirds. The key challenge was reducing server startup time from 18 minutes to under 2 minutes to handle preemption events, which they achieved through container optimization, GKE image streaming, and improved model loading processes.
LinkedIn developed and open-sourced LIER (LinkedIn Efficient and Reusable) kernels to address the fundamental challenge of memory consumption in LLM training. By optimizing core operations like layer normalization, rotary position encoding, and activation functions, they achieved up to 3-4x reduction in memory allocation and 20% throughput improvements for large models. The solution, implemented using Python and Triton, focuses on minimizing data movement between GPU memory and compute units, making LLM training faster and more cost-effective.
LinkedIn introduced Liger-Kernel, an open-source library addressing GPU efficiency challenges in LLM training. The solution combines efficient Triton kernels with a flexible API design, integrated into a comprehensive training infrastructure stack. The implementation achieved significant improvements, including 20% better training throughput and 60% reduced memory usage for popular models like Llama, Gemma, and Qwen, while maintaining compatibility with mainstream training frameworks and distributed training systems.
Prem AI
At Prem AI, they tackled the challenge of generating realistic ethereal planet images at scale with specific constraints like aspect ratio and controllable parameters. The solution involved fine-tuning Stable Diffusion XL with a curated high-quality dataset, implementing custom upscaling pipelines, and optimizing performance through various techniques including LoRA fusion, model quantization, and efficient serving frameworks like Ray Serve.
IDInsight
Ask-a-Metric developed a WhatsApp-based AI data analyst that converts natural language questions to SQL queries. They evolved from a simple sequential pipeline to testing an agent-based approach using CrewAI, ultimately creating a hybrid "pseudo-agent" pipeline that combined the best aspects of both approaches. While the agent-based system achieved high accuracy, its high costs and slow response times led to the development of an optimized pipeline that maintained accuracy while reducing query response time to under 15 seconds and costs to less than $0.02 per query.
Snowflake
Snowflake faced performance bottlenecks when scaling embedding models for their Cortex AI platform, which processes trillions of tokens monthly. Through profiling vLLM, they identified CPU-bound inefficiencies in tokenization and serialization that left GPUs underutilized. They implemented three key optimizations: encoding embedding vectors as little-endian bytes for faster serialization, disaggregating tokenization and inference into a pipeline, and running multiple model replicas on single GPUs. These improvements delivered 16x throughput gains for short sequences and 4.2x for long sequences, while reducing costs by 16x and achieving 3x throughput improvement in production.
Various
A panel discussion featuring experts from Various companies discussing key aspects of building production LLM applications. The discussion covers critical topics including hallucination management, prompt engineering, evaluation frameworks, cost considerations, and model selection. Panelists share practical experiences and insights on deploying LLMs in production, highlighting the importance of continuous feedback loops, evaluation metrics, and the trade-offs between open source and commercial LLMs.
Google, Databricks,
A panel discussion featuring leaders from various AI companies discussing the challenges and solutions in deploying LLMs in production. Key topics included model selection criteria, cost optimization, ethical considerations, and architectural decisions. The discussion highlighted practical experiences from companies like Interact.ai's healthcare deployment, Inflection AI's emotionally intelligent models, and insights from Google and Databricks on responsible AI deployment and tooling.
Various
A panel of industry experts from companies including Titan ML, YLabs, and Outer Bounds discuss best practices for deploying LLMs in production. They cover key challenges including prototyping, evaluation, observability, hardware constraints, and the importance of iteration. The discussion emphasizes practical advice for teams moving from prototype to production, highlighting the need for proper evaluation metrics, user feedback, and robust infrastructure.
Various
A panel discussion featuring leaders from Google Cloud AI, Symbol AI, Chain ML, and Deloitte discussing the adoption, scaling, and implementation challenges of generative AI across different industries. The panel explores key considerations around model selection, evaluation frameworks, infrastructure requirements, and organizational readiness while highlighting practical approaches to successful GenAI deployment in production.
OpenAI
This case study explores OpenAI's approach to post-training and deploying large language models in production environments, featuring insights from a post-training researcher working on reasoning models. The discussion covers the operational complexities of reinforcement learning from human feedback at scale, the evolution from non-thinking to thinking models, and production challenges including model routing, context window optimization, token efficiency improvements, and interruptability features. Key developments include the shopping model release, improvements from GPT-4.1 to GPT-5.1, and the operational realities of managing complex RL training runs with multiple grading setups and infrastructure components that require constant monitoring and debugging.
Prolego
A detailed technical discussion between Prolego engineers about the practical challenges of implementing Retrieval Augmented Generation (RAG) systems in production. The conversation covers key challenges including document processing, chunking strategies, embedding techniques, and evaluation methods. The team shares real-world experiences about how RAG implementations differ from tutorial examples, particularly in handling complex document structures and different data formats.
Parlance Labs
A comprehensive discussion of LLM deployment challenges and solutions across multiple industries, focusing on practical aspects like evaluation, fine-tuning, and production deployment. The case study covers experiences from GitHub's Copilot development, real estate CRM implementation, and consulting work at Parlance Labs, highlighting the importance of rigorous evaluation, data inspection, and iterative development in LLM deployments.
Pan Cha, Senior Product Manager at LinkedIn, shares insights on integrating LLMs into products effectively. He advocates for a pragmatic approach: starting with simple implementations using existing LLM APIs to validate use cases, then iteratively improving through robust prompt engineering and evaluation. The focus is on solving real user problems rather than adding AI for its own sake, with particular attention to managing user trust and implementing proper evaluation frameworks.
Zoro UK
Zoro UK, an e-commerce subsidiary of Grainger with 3.5 million products from 300+ suppliers, faced challenges normalizing and sorting product attributes across 75,000 different attribute types. Using DSPy (a framework for optimizing LLM prompts programmatically), they built a production system that automatically determines whether attributes require alpha-numeric sorting or semantic sorting. The solution employs a two-tier architecture: Mistral 8B for initial classification and GPT-4 for complex semantic sorting tasks. The DSPy approach eliminated manual prompt engineering, provided LLM-agnostic compatibility, and enabled automated prompt optimization using genetic algorithm-like iterations, resulting in improved product discoverability and search experience for their 1 million monthly active users.
Tinder
Tinder implemented two production GenAI applications to enhance user safety and experience: a username detection system using fine-tuned Mistral 7B to identify social media handles in user bios with near-perfect recall, and a personalized match explanation feature using fine-tuned Llama 3.1 8B to help users understand why recommended profiles are relevant. Both systems required sophisticated LLMOps infrastructure including multi-model serving with LoRA adapters, GPU optimization, extensive monitoring, and iterative fine-tuning processes to achieve production-ready performance at scale.
Nubank, Harvey AI, Galileo and Convirza
A panel discussion featuring leaders from Nubank, Harvey AI, Galileo, and Convirza discussing their experiences implementing LLMs in production. The discussion covered key challenges and solutions around model evaluation, cost optimization, latency requirements, and the transition from large proprietary models to smaller fine-tuned models. Participants shared insights on modularizing LLM applications, implementing human feedback loops, and balancing the tradeoffs between model size, cost, and performance in production environments.
Various
A comprehensive webinar featuring two case studies of LLM systems in production. First, Docugami shared their experience building a document processing pipeline that leverages hierarchical chunking and semantic understanding, using custom LLMs and extensive testing infrastructure. Second, Reet presented their development of Lucy, a real estate agent co-pilot, highlighting their journey with OpenAI function calling, testing frameworks, and preparing for fine-tuning while maintaining production quality.
Superlinked
SuperLinked, a company focused on vector search infrastructure, shares production insights from deploying information retrieval systems for e-commerce and enterprise knowledge management with indexes up to 2 terabytes. The presentation addresses challenges in relevance, latency, and cost optimization when deploying vector search systems at scale. Key solutions include avoiding vector pooling/averaging, implementing late interaction models, fine-tuning embeddings for domain-specific needs, combining sparse and dense representations, leveraging graph embeddings, and using template-based query generation instead of unconstrained text-to-SQL. Results demonstrate 5%+ precision improvements through targeted fine-tuning, significant latency reductions through proper database selection and query optimization, and improved relevance through multi-encoder architectures that combine text, graph, and metadata signals.
Digits
Digits implemented a production system for generating contextual questions for accountants using fine-tuned T5 models. The system helps accountants interact with clients by automatically generating relevant questions about transactions. They addressed key challenges like hallucination and privacy through multiple validation checks, in-house fine-tuning, and comprehensive evaluation metrics. The solution successfully deployed using TensorFlow Extended on Google Cloud Vertex AI with careful attention to training-serving skew and model performance monitoring.
Reducto
Reducto has built a production document parsing system that processes over 1 billion documents by combining specialized vision-language models, traditional OCR, and layout detection models in a hybrid pipeline. The system addresses critical challenges in document parsing including hallucinations from frontier models, dense tables, handwritten forms, and complex charts. Their approach uses a divide-and-conquer strategy where different models are routed to different document regions based on complexity, achieving higher accuracy than AWS Textract, Microsoft Azure Document Intelligence, and Google Cloud OCR on their internal benchmarks. The company has expanded beyond parsing to offer extraction with pixel-level citations and an edit endpoint for automated form filling.
Playtika
Playtika, a gaming company, built an internal generative AI platform to accelerate art production for their game studios with the goal of reducing art production time by 50%. The solution involved creating a comprehensive infrastructure for fine-tuning and deploying diffusion models (Stable Diffusion 1.5, then SDXL) at scale, supporting text-to-image, image-to-image, and inpainting capabilities. The platform evolved from using DreamBooth fine-tuning with separate model deployments to LoRA adapters with SDXL, enabling efficient model switching and GPU utilization. Through optimization techniques including OneFlow acceleration framework (achieving 40% latency reduction), FP16 quantization, NVIDIA MIG partitioning, and careful infrastructure design, they built a cost-efficient system serving multiple game studios while maintaining quality and minimizing inference latency.
A LinkedIn product manager shares insights on bringing LLMs to production, focusing on their implementation of various generative AI features across the platform. The case study covers the complete lifecycle from idea exploration to production deployment, highlighting key considerations in prompt engineering, GPU resource management, and evaluation frameworks. The presentation emphasizes practical approaches to building trust-worthy AI products while maintaining scalability and user focus.
Hassan El Mghari
Hassan El Mghari, a developer relations leader at Together AI, demonstrates how to build and scale AI applications to millions of users using open source models and a simplified architecture. Through building approximately 40 AI apps over four years (averaging one per month), he developed a streamlined approach that emphasizes simplicity, rapid iteration, and leveraging the latest open source models. His applications, including commit message generators, text-to-app builders, and real-time image generators, have collectively served millions of users and generated tens of millions of outputs, proving that simple architectures with single API calls can achieve significant scale when combined with good UI design and viral sharing mechanics.
Roblox
Roblox deployed a unified transformer-based translation LLM to enable real-time chat translation across all combinations of 16 supported languages for over 70 million daily active users. The company built a custom ~1 billion parameter model using pretraining on open source and proprietary data, then distilled it down to fewer than 650 million parameters to achieve approximately 100 millisecond latency while handling over 5,000 chats per second. The solution leverages a mixture-of-experts architecture, custom translation quality estimation models, back translation techniques for low-resource language pairs, and comprehensive integration with trust and safety systems to deliver contextually appropriate translations that understand Roblox-specific slang and terminology.
Instacart
Instacart transformed their query understanding (QU) system from multiple independent traditional ML models to a unified LLM-based approach to better handle long-tail, specific, and creatively-phrased search queries. The solution employed a layered strategy combining retrieval-augmented generation (RAG) for context engineering, post-processing guardrails, and fine-tuning of smaller models (Llama-3-8B) on proprietary data. The production system achieved significant improvements including 95%+ query rewrite coverage with 90%+ precision, 6% reduction in scroll depth for tail queries, 50% reduction in complaints for poor tail query results, and sub-300ms latency through optimizations like adapter merging, H100 GPU upgrades, and autoscaling.
Square
Square developed and deployed a RoBERTa-based merchant classification system to accurately categorize millions of merchants across their platform. The system replaced unreliable self-selection methods with an ML approach that combines business names, self-selected information, and transaction data to achieve a 30% improvement in accuracy. The solution runs daily predictions at scale using distributed GPU infrastructure and has become central to Square's business metrics and strategic decision-making.
Nvidia
ServiceNow and SLB (formerly Schlumberger) leveraged Nvidia DGX Cloud on AWS to develop and deploy foundation models for their respective industries. ServiceNow focused on building efficient small language models (5B-15B parameters) for enterprise process automation and agentic systems that match frontier model performance at a fraction of the cost and size, achieving nearly 100% GPU utilization through Run AI orchestration. SLB developed domain-specific multi-modal foundation models for seismic and petrophysical data to assist geoscientists and engineers in the energy sector, accelerating time-to-market for two major product releases over two years. Both organizations benefited from the fully optimized, turnkey infrastructure stack combining high-performance GPUs, networking, Lustre storage, EKS optimization, and enterprise-grade support, enabling them to focus on model development rather than infrastructure management while achieving zero or near-zero downtime.
Meta
Meta developed and deployed an AI-powered image animation feature that needed to serve billions of users efficiently. They tackled this challenge through a comprehensive optimization strategy including floating-point precision reduction, temporal-attention improvements, DPM-Solver implementation, and innovative distillation techniques. The system was further enhanced with sophisticated traffic management and load balancing solutions, resulting in a highly efficient, globally scalable service with minimal latency and failure rates.
Meta
Meta shares their journey in scaling AI infrastructure to support massive LLM training and inference operations. The company faced challenges in scaling from 256 GPUs to over 100,000 GPUs in just two years, with plans to reach over a million GPUs by year-end. They developed solutions for distributed training, efficient inference, and infrastructure optimization, including new approaches to data center design, power management, and GPU resource utilization. Key innovations include the development of a virtual machine service for secure code execution, improvements in distributed inference, and novel approaches to reducing model hallucinations through RAG.
Meta
Microsoft's AI infrastructure team tackled the challenges of scaling large language models across massive GPU clusters by optimizing network topology, routing, and communication libraries. They developed innovative approaches including rail-optimized cluster designs, smart communication libraries like TAL and MSL, and intelligent validation frameworks like SuperBench, enabling reliable training across hundreds of thousands of GPUs while achieving top rankings in ML performance benchmarks.
Meta
Meta's network engineers Rohit Puri and Henny present the evolution of Meta's AI network infrastructure designed to support large-scale generative AI training, specifically for LLaMA models. The case study covers the journey from a 24K GPU cluster used for LLaMA 3 training to a 100K+ GPU multi-building cluster for LLaMA 4, highlighting the architectural decisions, networking challenges, and operational solutions needed to maintain performance and reliability at unprecedented scale. The presentation details technical challenges including network congestion, priority flow control issues, buffer management, and firmware inconsistencies that emerged during production deployment, along with the engineering solutions implemented to resolve these issues while maintaining model training performance.
CoActive AI
CoActive AI addresses the challenge of processing unstructured data at scale through AI systems. They identified two key lessons: the importance of logical data models in bridging the gap between data storage and AI processing, and the strategic use of embeddings for cost-effective AI operations. Their solution involves creating data+AI hybrid teams to resolve impedance mismatches and optimizing embedding computations to reduce redundant processing, ultimately enabling more efficient and scalable AI operations.
Meta
Meta tackled the challenge of deploying an AI-powered image animation feature at massive scale, requiring optimization of both model performance and infrastructure. Through a combination of model optimizations including halving floating-point precision, improving temporal-attention expansion, and leveraging DPM-Solver, along with sophisticated traffic management and deployment strategies, they successfully deployed a system capable of serving billions of users while maintaining low latency and high reliability.
Perplexity AI
Perplexity AI evolved from an internal tool for answering SQL and enterprise questions to a full-fledged AI-powered search and research assistant. The company iteratively developed their product through various stages - from Slack and Discord bots to a web interface - while tackling challenges in search relevance, model selection, latency optimization, and cost management. They successfully implemented a hybrid approach using fine-tuned GPT models and their own LLaMA-based models, achieving superior performance metrics in both citation accuracy and perceived utility compared to competitors.
Anthropic
This case study examines Anthropic's journey in scaling and operating large language models, focusing on their transition from GPT-3 era training to current state-of-the-art systems like Claude. The company successfully tackled challenges in distributed computing, model safety, and operational reliability while growing 10x in revenue. Key innovations include their approach to constitutional AI, advanced evaluation frameworks, and sophisticated MLOps practices that enable running massive training operations with hundreds of team members.
Various
A tech company needed to improve their developer documentation accessibility and understanding. They implemented a self-hosted LLM solution using retrieval augmented generation (RAG), with guard rails for content safety. The team optimized performance using vLLM for faster inference and Ray Serve for horizontal scaling, achieving significant improvements in latency and throughput while maintaining cost efficiency. The solution helped developers better understand and adopt the company's products while keeping proprietary information secure.
Propel Holdings / Xanterra Travel Collection
Propel Holdings (fintech) and Xanterra Travel Collection (travel/hospitality) implemented Cresta's AI agent solutions to address scaling challenges and operational efficiency in their contact centers. Both organizations started with agent assist capabilities before deploying conversational AI agents for chat and voice channels. Propel Holdings needed to support 40% year-over-year growth without proportionally scaling human agents, while Xanterra sought to reduce call volume for routine inquiries and provide 24/7 coverage. Starting with FAQ-based use cases and later integrating APIs for transactional capabilities, both companies achieved significant results: Propel Holdings reached 58% chat containment after API integration, while Xanterra achieved 60-90% containment on chat and 20-30% on voice channels. Within five months, Xanterra deployed 12 AI agents across different properties and channels, demonstrating rapid scaling capability while maintaining customer satisfaction and redeploying human agents to higher-value interactions.
Articul8
Articul8, a generative AI company focused on domain-specific models (DSMs), faced challenges in training and deploying specialized LLMs across semiconductor, energy, and supply chain industries due to infrastructure complexity and computational requirements. They implemented Amazon SageMaker HyperPod to manage distributed training clusters with automated fault tolerance, achieving over 95% cluster utilization and 35% productivity improvements. The solution enabled them to reduce AI deployment time by 4x and total cost of ownership by 5x while successfully developing high-performing DSMs that outperform general-purpose LLMs by 2-3x in domain-specific tasks, with their A8-Semicon model achieving twice the accuracy of GPT-4o and Claude in Verilog code generation at 50-100x smaller model sizes.
Yahoo
Yahoo Mail faced challenges with their existing ML-based email content extraction system, hitting a coverage ceiling of 80% for major senders while struggling with long-tail senders and slow time-to-market for model updates. They implemented a new solution using Google Cloud's Vertex AI and LLMs, achieving 94% coverage for standard domains and 99% for tail domains, with 51% increase in extraction richness and 16% reduction in tracking API errors. The implementation required careful consideration of hybrid infrastructure, cost management, and privacy compliance while processing billions of daily messages.
Rogo
Rogo developed an enterprise-grade AI finance platform that leverages multiple OpenAI models to automate and enhance financial research and analysis for investment banks and private equity firms. Through a layered model architecture combining GPT-4 and other models, along with fine-tuning and integration with financial datasets, they created a system that saves analysts over 10 hours per week on tasks like meeting prep and market research, while serving over 5,000 bankers across major financial institutions.
Nubank
Nubank integrated foundation models into their AI platform to enhance predictive modeling across critical banking decisions, moving beyond traditional tabular machine learning approaches. Through their acquisition of Hyperplane in July 2024, they developed billion-parameter transformer models that process sequential transaction data to better understand customer behavior. Over eight months, they achieved significant performance improvements (1.20% average AUC lift across benchmark tasks) while maintaining existing data governance and model deployment infrastructure, successfully deploying these models to production decision engines serving over 100 million customers.
LinkedIn adopted vLLM, an open-source LLM inference framework, to power over 50 GenAI use cases including LinkedIn Hiring Assistant and AI Job Search, running on thousands of hosts across their platform. The company faced challenges in deploying LLMs at scale with low latency and high throughput requirements, particularly for applications requiring complex reasoning and structured outputs. By leveraging vLLM's PagedAttention technology and implementing a five-phase evolution strategy—from offline mode to a modular, OpenAI-compatible architecture—LinkedIn achieved significant performance improvements including ~10% TPS gains and GPU savings of over 60 units for certain workloads, while maintaining sub-600ms p95 latency for thousands of QPS in production applications.
Slack
Slack faced significant challenges in scaling their generative AI features (Slack AI) to millions of daily active users while maintaining security, cost efficiency, and quality. The company needed to move from a limited, provisioned infrastructure to a more flexible system that could handle massive scale (1-5 billion messages weekly) while meeting strict compliance requirements. By migrating from SageMaker to Amazon Bedrock and implementing sophisticated experimentation frameworks with LLM judges and automated metrics, Slack achieved over 90% reduction in infrastructure costs (exceeding $20 million in savings), 90% reduction in cost-to-serve per monthly active user, 5x increase in scale, and 15-30% improvements in user satisfaction across features—all while maintaining quality and enabling experimentation with over 15 different LLMs in production.
Meta
Meta's AI infrastructure team developed a comprehensive LLM serving platform to support Meta AI, smart glasses, and internal ML workflows including RLHF processing hundreds of millions of examples. The team addressed the fundamental challenges of LLM inference through a four-stage approach: building efficient model runners with continuous batching and KV caching, optimizing hardware utilization through distributed inference techniques like tensor and pipeline parallelism, implementing production-grade features including disaggregated prefill/decode services and hierarchical caching systems, and scaling to handle multiple deployments with sophisticated allocation and cost optimization. The solution demonstrates the complexity of productionizing LLMs, requiring deep integration across modeling, systems, and product teams to achieve acceptable latency and cost efficiency at scale.
Perplexity
Perplexity AI scaled their LLM-powered search engine to handle over 435 million queries monthly by implementing a sophisticated inference architecture using NVIDIA H100 GPUs, Triton Inference Server, and TensorRT-LLM. Their solution involved serving 20+ AI models simultaneously, implementing intelligent load balancing, and using tensor parallelism across GPU pods. This resulted in significant cost savings - approximately $1 million annually compared to using third-party LLM APIs - while maintaining strict service-level agreements for latency and performance.
Meta
Meta faced the challenge of scaling their AI infrastructure from training smaller recommendation models to massive LLM training jobs like LLaMA 3. They built two 24K GPU clusters (one with RoCE, another with InfiniBand) to handle the unprecedented scale of computation required for training models with thousands of GPUs running for months. Through full-stack optimizations across hardware, networking, and software layers, they achieved 95% training efficiency for the LLaMA 3 70B model, while dealing with challenges in hardware reliability, thermal management, network topology, and collective communication operations.
DeepL
DeepL needed to scale their Language AI capabilities while maintaining low latency for production inference and handling increasing request volumes. The company transitioned from BFloat16 (BF16) to 8-bit floating point (FP8) precision for both training and inference of their large language models, leveraging NVIDIA H100 GPUs' native FP8 support through Transformer Engine for training and TensorRT-LLM for inference. This approach accelerated model training by 50% (achieving 67% Model FLOPS utilization), enabled training of larger models with more parameters, doubled inference throughput at equivalent latency levels, and delivered translation quality improvements of 1.4x for European languages and 1.7x for complex language pairs like English-Japanese, all while maintaining comparable training quality to BF16 precision.
LinkedIn faced significant performance challenges when deploying LLM-based ranking systems for AI Job Search and AI People Search, where models needed to score hundreds of items per query within strict latency SLAs (sub-500ms P99). The ranking workload differs fundamentally from text generation—it requires only the prefill phase to score candidates, not iterative token generation. LinkedIn optimized SGLang, an open-source LLM serving system, through four optimization stages: implementing comprehensive batching (tokenization and batch preservation), creating a scoring-only fast path that eliminates unnecessary decode loops and CPU-GPU synchronization, introducing in-batch prefix caching to reuse shared query context, and addressing Python runtime bottlenecks through multi-process architecture. These optimizations delivered 2-3x throughput improvements on H100 GPUs while maintaining P99 latency under 500ms, enabling production-scale LLM ranking for millions of members.
Doordash
Doordash leverages LLMs to enhance their product knowledge graph and search capabilities as they expand into new verticals beyond food delivery. They employ LLM-assisted annotations for attribute extraction, use RAG for generating training data, and implement LLM-based systems for detecting catalog inaccuracies and understanding search intent. The solution includes distributed computing frameworks, model optimization techniques, and careful consideration of latency and throughput requirements for production deployment.
Cursor
Cursor experimented with running hundreds of concurrent LLM-based coding agents autonomously for weeks on large-scale software projects. The problem was that single agents work well for focused tasks but struggle with complex projects requiring months of work. Their solution evolved from flat peer-to-peer coordination (which failed due to locking bottlenecks and risk-averse behavior) to a hierarchical planner-worker architecture where planner agents create tasks and worker agents execute them independently. Results included agents successfully building a web browser from scratch (1M+ lines of code over a week), completing a 3-week React migration (266K additions/193K deletions), optimizing video rendering by 25x, and running multiple other ambitious projects with thousands of commits and millions of lines of code.
Meta
Meta's network engineering team faced an unprecedented challenge when AI workload demands required accelerating their backbone network scaling plans from 2028 to 2024-2025, necessitating a 10x capacity increase. They addressed this through three key techniques: pre-building scalable data center metro architectures with ring topologies, platform scaling through both vendor-dependent improvements (larger chassis, faster interfaces) and internal innovations (adding backbone planes, multiple devices per plane), and IP-optical integration using coherent transceiver technology that reduced power consumption by 80-90% while dramatically improving space efficiency. Additionally, they developed specialized AI backbone solutions for connecting geographically distributed clusters within 3-100km ranges using different fiber and optical technologies based on distance requirements.
Fuzzy Labs
Fuzzy Labs helped a tech company improve their developer documentation and tooling experience by implementing a self-hosted LLM system using Mistral-7B. They tackled performance challenges through systematic load testing with Locust, optimized inference latency using vLLM's paged attention, and achieved horizontal scaling with Ray Serve. The solution improved response times from 11 seconds to 3 seconds and enabled handling of concurrent users while efficiently managing GPU resources.
Tinder
Tinder implemented a comprehensive LLM-based trust and safety system to combat various forms of harmful content at scale. The solution involves fine-tuning open-source LLMs using LoRA (Low-Rank Adaptation) for different types of violation detection, from spam to hate speech. Using the Lorax framework, they can efficiently serve multiple fine-tuned models on a single GPU, achieving real-time inference with high precision and recall while maintaining cost-effectiveness. The system demonstrates superior generalization capabilities against adversarial behavior compared to traditional ML approaches.
Notion
Notion scaled their vector search infrastructure supporting Notion AI Q&A from launch in November 2023 through early 2026, achieving a 10x increase in capacity while reducing costs by 90%. The problem involved onboarding millions of workspaces to their AI-powered semantic search feature while managing rapidly growing infrastructure costs. Their solution involved migrating from dedicated pod-based vector databases to serverless architectures, switching to turbopuffer as their vector database provider, implementing intelligent page state caching to avoid redundant embeddings, and transitioning to Ray on Anyscale for both embeddings generation and serving. The results included clearing a multi-million workspace waitlist, reducing vector database costs by 60%, cutting embeddings infrastructure costs by over 90%, and improving query latency from 70-100ms to 50-70ms while supporting 15x growth in active workspaces.
Zilliz
Zilliz, the company behind the open-source Milvus vector database, shares their approach to scaling vector search to handle billions of vectors. They employ a multi-tier storage architecture spanning from GPU memory to object storage, enabling flexible trade-offs between performance, cost, and data freshness. The system uses GPU acceleration for both index building and search, implements real-time search through a buffer strategy, and handles distributed consistency challenges at scale.
ElevenLabs
ElevenLabs developed a high-performance voice AI platform for voice cloning and multilingual speech synthesis, leveraging Google Cloud's GKE and NVIDIA GPUs for scalable deployment. They implemented GPU optimization strategies including multi-instance GPUs and time-sharing to improve utilization and reduce costs, while successfully serving 600 hours of generated audio for every hour of real time across 29 languages.
LiftOff
LiftOff LLC explored deploying open-source DeepSeek-R1 models (1.5B, 7B, 8B, 16B parameters) on AWS EC2 GPU instances to evaluate their viability as alternatives to paid AI services like ChatGPT. While technically successful in deployment using Docker, Ollama, and OpenWeb UI, the operational costs significantly exceeded expectations, with a single g5g.2xlarge instance costing $414/month compared to ChatGPT Plus at $20/user/month. The experiment revealed that smaller models lacked production-quality responses, while larger models faced memory limitations, performance degradation with longer contexts, and stability issues, concluding that self-hosting isn't cost-effective at startup scale.
Relevance AI
Relevance AI implemented DSPy-powered self-improving AI agents for outbound sales email composition, addressing the challenge of building truly adaptive AI systems that evolve with real-world usage. The solution integrates DSPy's optimization framework with a human-in-the-loop feedback mechanism, where agents pause for approval at critical checkpoints and incorporate corrections into their training data. Through this approach, the system achieved emails matching human-written quality 80% of the time and exceeded human performance in 6% of cases, while reducing agent development time by 50% through elimination of manual prompt tuning. The system demonstrates continuous improvement through automated collection of human-approved examples that feed back into DSPy's optimization algorithms.
DocETL
Shreyaa Shankar presents DocETL, an open-source system for semantic data processing that addresses the challenges of running LLM-powered operators at scale over unstructured data. The system tackles two major problems: how to make semantic operator pipelines scalable and cost-effective through novel query optimization techniques, and how to make them steerable through specialized user interfaces. DocETL introduces rewrite directives that decompose complex tasks and data to improve accuracy and reduce costs, achieving up to 86% cost reduction while maintaining target accuracy. The companion tool Doc Wrangler provides an interactive interface for iteratively authoring and debugging these pipelines. Real-world applications include public defenders analyzing court transcripts for racial bias and medical analysts extracting information from doctor-patient conversations, demonstrating significant accuracy improvements (2x in some cases) compared to baseline approaches.
Etsy
Etsy's Search Relevance team developed a comprehensive Semantic Relevance Evaluation and Enhancement Framework to address the limitations of engagement-based search models that favored popular listings over semantically relevant ones. The solution employs a three-tier cascaded distillation approach: starting with human-curated "golden" labels, scaling with an LLM annotator (o3 model) to generate training data, fine-tuning a teacher model (Qwen 3 VL 4B) for efficient large-scale evaluation, and distilling to a lightweight BERT-based student model for real-time production inference. The framework integrates semantic relevance signals into search through filtering, feature enrichment, loss weighting, and relevance boosting. Between August and October 2025, the percentage of fully relevant listings increased from 58% to 62%, demonstrating measurable improvements in aligning search results with buyer intent while addressing the cold-start problem for smaller sellers.
Flipkart
Flipkart faced the challenge of accurately extracting product attributes (like color, pattern, and material) from millions of product listings at scale. Manual labeling was expensive and error-prone, while using large Vision Language Model APIs was cost-prohibitive. The company developed a semi-supervised approach using compact VLMs (2-3 billion parameters) that combines Parameter-Efficient Fine-Tuning (PEFT) with Direct Preference Optimization (DPO) to leverage unlabeled data. The method starts with a small labeled dataset, generates multiple reasoning chains for unlabeled products using self-consistency, and then fine-tunes the model using DPO to favor preferred outputs. Results showed accuracy improvements from 75.1% to 85.7% on the Qwen2.5-VL-3B-Instruct model across twelve e-commerce verticals, demonstrating that compact models can effectively learn from unlabeled data to achieve production-grade performance.
Grammarly
Grammarly developed GECToR, a novel grammatical error correction (GEC) system that treats error correction as a sequence-tagging problem rather than the traditional neural machine translation approach. Instead of rewriting entire sentences through encoder-decoder models, GECToR tags individual tokens with custom transformations (like $DELETE, $APPEND, $REPLACE) using a BERT-like encoder with linear layers. This approach achieved state-of-the-art F0.5 scores (65.3 on CoNLL-2014, 72.4 on BEA-2019) while running up to 10 times faster than NMT-based systems, with inference speeds of 0.20-0.40 seconds compared to 0.71-4.35 seconds for transformer-NMT approaches. The system uses iterative correction over multiple passes and custom g-transformations for complex operations like verb conjugation and noun number changes, making it more suitable for real-world production deployment in Grammarly's writing assistant.
Tokyo Electron
Tokyo Electron is addressing complex semiconductor manufacturing challenges by implementing Small Specialist Agents (SSAs) powered by LLMs. These agents combine domain expertise with LLM capabilities to optimize manufacturing processes. The solution includes both public and private SSAs managed by a General Management Agent (GMA), with plans to utilize domain-specific smaller models to overcome computational and security challenges in production environments. The approach aims to replicate expert decision-making in semiconductor processing while maintaining scalability and data security.
Adyen
Adyen, a global financial technology platform, implemented LLM-powered solutions to improve their support team's efficiency. They developed a smart ticket routing system and a support agent copilot using LangChain, deployed in a Kubernetes environment. The solution resulted in more accurate ticket routing and faster response times through automated document retrieval and answer suggestions, while maintaining flexibility to switch between different LLM models.
Accenture
Accenture partnered with Databricks to transform a client's customer contact center by implementing specialized language models (SLMs) that go beyond simple prompt engineering. The client faced challenges with high call volumes, impersonal service, and missed revenue opportunities. Using Databricks' MLOps platform and GPU infrastructure, they developed and deployed fine-tuned language models that understand industry-specific context, cultural nuances, and brand styles, resulting in improved customer experience and operational efficiency. The solution includes real-time monitoring and multimodal capabilities, setting a new standard for AI-driven customer service operations.
Grammarly
Grammarly developed CoEdIT, a specialized text editing LLM that outperforms larger models while being up to 60 times smaller. Through targeted instruction tuning on a carefully curated dataset of text editing tasks, they created models ranging from 770M to 11B parameters that achieved state-of-the-art performance on multiple editing benchmarks, outperforming models like GPT-3-Edit (175B parameters) and ChatGPT in both automated and human evaluations.
Zalando
A comprehensive overview of the current state and challenges of production machine learning and LLMOps, covering key areas including motivations, industry trends, technological developments, and organizational changes. The presentation highlights the evolution from model-centric to data-centric approaches, the importance of metadata management, and the growing focus on security and monitoring in ML systems.
Doordash
DoorDash outlines a comprehensive strategy for implementing Generative AI across five key areas: customer assistance, interactive discovery, personalized content generation, information extraction, and employee productivity enhancement. The company aims to revolutionize its delivery platform while maintaining strong considerations for data privacy and security, focusing on practical applications ranging from automated cart building to SQL query generation.
Checkr
Checkr tackled the challenge of classifying complex background check records by implementing a fine-tuned small language model (SLM) solution. They moved from using GPT-4 to fine-tuning Llama-2 models on Predibase, achieving 90% accuracy for their most challenging cases while reducing costs by 5x and improving response times to 0.15 seconds. This solution helped automate their background check adjudication process, particularly for the 2% of complex cases that required classification into 230 distinct categories.
Hapag-Lloyd
Hapag-Lloyd faced challenges with time-consuming manual corporate audit processes. They implemented a GenAI solution using Databricks Mosaic AI to automate audit finding generation and executive summary creation. By fine-tuning the DBRX model and implementing a RAG-based chatbot, they achieved a 66% decrease in time spent creating new findings and a 77% reduction in executive summary review time, significantly improving their audit efficiency.
Salesforce
Salesforce's AI platform team faced operational challenges deploying customized large language models (fine-tuned versions of Llama, Qwen, and Mistral) for their Agentforce agentic AI applications. The deployment process was time-consuming, requiring months of optimization for instance families, serving engines, and configurations, while also proving expensive due to GPU capacity reservations for peak usage. By adopting Amazon Bedrock Custom Model Import, Salesforce integrated a unified API for model deployment that minimized infrastructure management while maintaining backward compatibility with existing endpoints. The results included a 30% reduction in deployment time, up to 40% cost savings through pay-per-use pricing, and maintained scalability without sacrificing performance.
FiscalNote
FiscalNote, facing challenges in deploying and updating their legislative analysis ML models efficiently, transformed their MLOps pipeline using Databricks' MLflow and Model Serving. This shift enabled them to reduce deployment time and increase model deployment frequency by 3x, while improving their ability to provide timely legislative insights to clients through better model management and deployment practices.
Booking.com
Booking.com built an AI Trip Planner to handle unstructured, natural language queries from travelers seeking personalized recommendations. The challenge was combining LLMs' ability to understand conversational requests with years of structured behavioral data (searches, clicks, bookings). Instead of relying solely on prompt engineering with external APIs, they used supervised fine-tuning on open-source LLMs with parameter-efficient methods. This approach delivered superior recommendation metrics while achieving 3x faster inference compared to prompt-based solutions, while maintaining data privacy and security by keeping all processing internal.
Altana
Altana, a global supply chain intelligence company, faced challenges in efficiently deploying and managing multiple GenAI models for diverse customer use cases. By implementing Databricks Mosaic AI platform, they transformed their ML lifecycle management, combining custom deep learning models with fine-tuned LLMs and RAG workflows. This led to 20x faster model deployment times and 20-50% performance improvements, while maintaining data privacy and governance requirements across their global operations.
ZURU
ZURU Tech, a construction technology company, collaborated with AWS to develop a text-to-floor plan generator that allows users to create building designs using natural language descriptions. The project aimed to improve upon existing GPT-2 baseline results by implementing both prompt engineering with Claude 3.5 Sonnet on Amazon Bedrock and fine-tuning approaches with Llama models on Amazon SageMaker. Through careful dataset preparation, dynamic few-shot prompting, and comprehensive evaluation frameworks, the team achieved a 109% improvement in instruction adherence accuracy compared to their baseline model, with fine-tuning also delivering a 54% improvement in mathematical correctness for spatial relationships and dimensions.
Thinking Machines
Thinking Machines, a new AI company founded by former OpenAI researcher John Schulman, has developed Tinker, a low-level fine-tuning API designed to enable sophisticated post-training of language models without requiring teams to manage GPU infrastructure or distributed systems complexity. The product aims to abstract away infrastructure concerns while providing low-level primitives for expressing nearly all post-training algorithms, allowing researchers and companies to build custom models without developing their own training infrastructure. The company plans to release their own models and expand Tinker's capabilities to include multimodal functionality and larger-scale training jobs, while making the platform more accessible to non-experts through higher-level tooling.
Institute of Science Tokyo
The Institute of Science Tokyo successfully developed Llama 3.3 Swallow, a 70-billion-parameter large language model with enhanced Japanese capabilities, using Amazon SageMaker HyperPod infrastructure. The project involved continual pre-training from Meta's Llama 3.3 70B model using 314 billion tokens of primarily Japanese training data over 16 days across 256 H100 GPUs. The resulting model demonstrates superior performance compared to GPT-4o-mini and other leading models on Japanese language benchmarks, showcasing effective distributed training techniques including 4D parallelism, asynchronous checkpointing, and comprehensive monitoring systems that enabled efficient large-scale model training in production.
Patronus AI
Patronus AI addressed the critical challenge of LLM hallucination detection by developing Lynx, a state-of-the-art model trained on their HaluBench dataset. Using Databricks' Mosaic AI infrastructure and LLM Foundry tools, they fine-tuned Llama-3-70B-Instruct to create a model that outperformed both closed and open-source LLMs in hallucination detection tasks, achieving nearly 1% better accuracy than GPT-4 across various evaluation scenarios.
Dynamo
Dynamo, an AI company focused on secure and compliant AI solutions, developed an 8-billion parameter multilingual LLM using Databricks Mosaic AI Training platform. They successfully trained the model in just 10 days, achieving a 20% speedup in training compared to competitors. The model was designed to support enterprise-grade AI systems with built-in security guardrails, compliance checks, and multilingual capabilities for various industry applications.
OpenAI
OpenAI's development and training of GPT-4.5 represents a significant milestone in large-scale LLM deployment, featuring a two-year development cycle and unprecedented infrastructure scaling challenges. The team aimed to create a model 10x smarter than GPT-4, requiring intensive collaboration between ML and systems teams, sophisticated planning, and novel solutions to handle training across massive GPU clusters. The project succeeded in achieving its goals while revealing important insights about data efficiency, system design, and the relationship between model scale and intelligence.
MosaicML
MosaicML developed and open-sourced MPT, a family of large language models including 7B and 30B parameter versions, demonstrating that high-quality LLMs could be trained for significantly lower costs than commonly believed (under $250,000 for 7B model). They built a complete training platform handling data processing, distributed training, and model deployment at scale, while documenting key lessons around planning, experimentation, data quality, and operational best practices for production LLM development.
AWS (Alexa)
AWS (Alexa) faced the challenge of evolving their voice assistant from scripted, command-based interactions to natural, generative AI-powered conversations while serving over 600 million devices and maintaining complete backward compatibility with existing integrations. The team completely rearchitected Alexa using large language models (LLMs) to create Alexa Plus, which supports conversational interactions, complex multi-step planning, and real-world action execution. Through extensive experimentation with prompt engineering, multi-model architectures, speculative execution, prompt caching, API refactoring, and fine-tuning, they achieved the necessary balance between accuracy, latency (sub-2-second responses), determinism, and model flexibility required for a production voice assistant serving hundreds of millions of users daily.
Swiggy
Swiggy, a major food delivery platform in India, implemented a novel two-stage fine-tuning approach for language models to improve search relevance in their hyperlocal food delivery service. They first performed unsupervised fine-tuning using historical search queries and order data, followed by supervised fine-tuning with manually curated query-item pairs. The solution leverages TSDAE and Multiple Negatives Ranking Loss approaches, achieving superior search relevance metrics compared to baseline models while meeting strict latency requirements of 100ms.
Doctolib
Doctolib is transforming their healthcare data platform from a reporting-focused system to an AI-enabled unified platform. The company is implementing a comprehensive LLMOps infrastructure as part of their new architecture, including features for model training, inference, and GenAI assistance for data exploration. The platform aims to support both traditional analytics and advanced AI capabilities while ensuring security, governance, and scalability for healthcare data.
Grab
Grab developed a custom foundation model to generate user embeddings that power personalization across its Southeast Asian superapp ecosystem. Traditional approaches relied on hundreds of manually engineered features that were task-specific and siloed, struggling to capture sequential user behavior effectively. Grab's solution involved building a transformer-based foundation model that jointly learns from both tabular data (user attributes, transaction history) and time-series clickstream data (user interactions and sequences). This model processes diverse data modalities including text, numerical values, IDs, and location data through specialized adapters, using unsupervised pre-training with masked language modeling and next-action prediction. The resulting embeddings serve as powerful, generalizable features for downstream applications including ad optimization, fraud detection, churn prediction, and recommendations across mobility, food delivery, and financial services, significantly improving personalization while reducing feature engineering effort.
Modal
Modal's engineering team tackled the challenge of generating aesthetically pleasing QR codes that consistently scan by implementing comprehensive evaluation systems and inference-time compute scaling. The team developed automated evaluation pipelines that measured both scan rate and aesthetic quality, using human judgment alignment to validate their metrics. They applied inference-time compute scaling by generating multiple QR codes in parallel and selecting the best candidates, achieving a 95% scan rate service-level objective while maintaining aesthetic quality and returning results in under 20 seconds.
Windsurf
Windsurf developed Tab v2, an AI-powered code autocomplete system that addresses the challenge of balancing prediction frequency, accuracy, and code length in developer tooling. The team reimagined their LLM-based autocomplete by focusing on total keystrokes saved rather than just acceptance rate, implementing extensive context engineering to reduce prompt length by 76%, and using reinforcement learning to train models with different "aggression" levels. The result was a 54% average increase in characters per prediction and 25-75% more accepted code, with user-selectable aggression parameters allowing developers to customize behavior based on personal preferences.
Paramount+
Paramount+ partnered with Google Cloud Consulting to develop two key AI use cases: video summarization and metadata extraction for their streaming platform containing over 50,000 videos. The project used Gen AI jumpstarts to prototype solutions, implementing prompt chaining, embedding generation, and fine-tuning approaches. The system was designed to enhance content discoverability and personalization while reducing manual labor and third-party costs. The implementation included a three-component architecture handling transcription creation, content generation, and personalization integration.
Meta
Meta's Media Foundation team deployed AI-powered video super-resolution (VSR) models at massive scale to enhance video quality across their ecosystem, processing over 1 billion daily video uploads. The problem addressed was the prevalence of low-quality videos from poor camera quality, cross-platform uploads, and legacy content that degraded user experience. The solution involved deploying multiple VSR models—both CPU-based (using Intel's RVSR SDK) and GPU-based—to upscale and enhance video quality for ads and generative AI features like Meta Restyle. Through extensive subjective evaluation with thousands of human raters, Meta identified effective quality metrics (VMAF-UQ), determined which videos would benefit most from VSR, and successfully deployed the technology while managing GPU resource constraints and ensuring quality improvements aligned with user preferences.
Shopify
Shopify evolved their product classification system from basic categorization to an advanced AI-driven framework using Vision Language Models (VLMs) integrated with a comprehensive product taxonomy. The system processes over 30 million predictions daily, combining VLMs with structured taxonomy to provide accurate product categorization, attribute extraction, and metadata generation. This has resulted in an 85% merchant acceptance rate of predicted categories and doubled the hierarchical precision and recall compared to previous approaches.
