193 tools with this tag
← Back to LLMOps DatabaseGoogle deployed an abstractive summarization system to automatically generate conversation summaries in Google Chat Spaces to address information overload from unread messages, particularly in hybrid work environments. The solution leveraged the Pegasus transformer model fine-tuned on a custom ForumSum dataset of forum conversations, then distilled into a hybrid transformer-encoder/RNN-decoder architecture for lower latency. The system surfaces summaries through cards when users enter Spaces with unread messages, with quality controls including heuristics for triggering, detection of low-quality summaries, and ephemeral caching of pre-generated summaries to reduce latency, ultimately delivering production value to premium Google Workspace business customers.
Pinterest enhanced their homefeed recommendation system through several advancements in embedding-based retrieval. They implemented sophisticated feature crossing techniques using MaskNet and DHEN frameworks, adopted pre-trained ID embeddings with careful overfitting mitigation, upgraded their serving corpus with time-decay mechanisms, and introduced multi-embedding retrieval and conditional retrieval approaches. These improvements led to significant gains in user engagement metrics, with increases ranging from 0.1% to 1.2% across various metrics including engaged sessions, saves, and clicks.
Amazon
Amazon teams faced challenges in deploying high-stakes LLM applications across healthcare, engineering, and e-commerce domains where basic prompt engineering and RAG approaches proved insufficient. Through systematic application of advanced fine-tuning techniques including Supervised Fine-Tuning (SFT), Proximal Policy Optimization (PPO), Direct Preference Optimization (DPO), and cutting-edge reasoning optimizations like Group-based Reinforcement Learning from Policy Optimization (GRPO) and Direct Advantage Policy Optimization (DAPO), three Amazon business units achieved production-grade results: Amazon Pharmacy reduced dangerous medication errors by 33%, Amazon Global Engineering Services achieved 80% human effort reduction in inspection reviews, and Amazon A+ Content improved quality assessment accuracy from 77% to 96%. These outcomes demonstrate that approximately one in four high-stakes enterprise applications require advanced fine-tuning beyond standard techniques to achieve necessary performance levels in production environments.
Grammarly
Grammarly, a leading AI-powered writing assistant, tackled the challenge of improving grammatical error correction (GEC) by moving beyond traditional neural machine translation approaches that optimize n-gram metrics but sometimes produce semantically inconsistent corrections. The team developed a novel generative adversarial network (GAN) framework where a sequence-to-sequence generator produces grammatical corrections, and a sentence-pair discriminator evaluates whether the generated correction is the most appropriate rewrite for the given input sentence. Through adversarial training with policy gradients, the discriminator provides task-specific rewards to the generator, enabling better distributional alignment between generated and human corrections. Experiments showed that adversarially trained models (both RNN-based and transformer-based) consistently outperformed their standard counterparts on GEC benchmarks, striking a better balance between grammatical correctness, semantic preservation, and natural phrasing while serving millions of users in production.
Snorkel
Snorkel developed a specialized benchmark dataset for evaluating AI agents in insurance underwriting, leveraging their expert network of Chartered Property and Casualty Underwriters (CPCUs). The benchmark simulates an AI copilot that assists junior underwriters by reasoning over proprietary knowledge, using multiple tools including databases and underwriting guidelines, and engaging in multi-turn conversations. The evaluation revealed significant performance variations across frontier models (single digits to ~80% accuracy), with notable error modes including tool use failures (36% of conversations) and hallucinations from pretrained domain knowledge, particularly from OpenAI models which hallucinated non-existent insurance products 15-45% of the time.
Moveworks
Moveworks developed "Brief Me," an AI-powered productivity tool that enables employees to upload documents (PDF, Word, PPT) and interact with them conversationally through their Copilot assistant. The system addresses the time-consuming challenge of manually processing lengthy documents for tasks like summarization, Q&A, comparisons, and insight extraction. By implementing a sophisticated two-stage agentic architecture with online content ingestion and generation capabilities, including hybrid search with custom-trained embeddings, multi-turn conversation support, operation planning, and a novel map-reduce approach for long context handling, the system achieves high accuracy metrics (97.24% correct actions, 89.21% groundedness, 97.98% completeness) with P90 latency under 10 seconds for ingestion, significantly reducing the hours typically required for document analysis tasks.
Goodfire
Goodfire, an AI interpretability research company, deployed AI agents extensively for conducting experiments in their research workflow over several months. They distinguish between "developer agents" (for software development) and "experimenter agents" (for research and discovery), identifying key architectural differences needed for the latter. Their solution, code-named Scribe, leverages Jupyter notebooks with interactive, stateful access via MCP (Model Context Protocol), enabling agents to iteratively run experiments across domains like genomics, vision transformers, and diffusion models. Results showed agents successfully discovering features in genomics models, performing circuit analysis, and executing complex interpretability experiments, though validation, context engineering, and preventing reward hacking remain significant challenges that require human oversight and critic systems.
Meta
Meta developed AI Lab, a pre-production framework for continuously testing and optimizing machine learning workflows, with a focus on minimizing Time to First Batch (TTFB). The system enables both proactive improvements and automatic regression prevention for ML infrastructure changes. Using AI Lab, Meta was able to achieve up to 40% reduction in TTFB through the implementation of the Python Cinder runtime, while ensuring no regressions occurred during the rollout process.
Swedish Tax Authority
The Swedish Tax Authority (Skatteverket) has been on a multi-decade digitalization journey, progressively incorporating AI and large language models into production systems to automate and enhance tax services. The organization has developed various NLP applications including text categorization, transcription, OCR pipelines, and question-answering systems using RAG architectures. They have tested both open-source models (Llama 3.1, Mixtral 7B, Cohere) and commercial solutions (GPT-3.5), finding that open-source models perform comparably for simpler queries while commercial models excel at complex questions. The Authority operates within a regulated environment requiring on-premise deployment for sensitive data, adopting Agile/SAFe methodologies and building reusable AI infrastructure components that can serve multiple business domains across different public sector silos.
Amazon
Amazon developed Dialogue Boost, an AI-powered audio processing technology that enhances dialogue clarity in TV shows, movies, and podcasts by suppressing background music and sound effects. The system uses deep neural networks for sound source separation and runs directly on-device (Echo smart speakers and Fire TV devices) thanks to breakthroughs in model compression and knowledge distillation. Originally launched on Prime Video in 2022 using cloud-based processing, the technology was compressed to less than 1% of its original size while maintaining nearly identical performance, enabling real-time processing across multiple streaming platforms including Netflix, YouTube, and Disney+. Research shows over 86% of participants preferred Dialogue-Boost-enhanced audio, with 100% approval among users with hearing loss, significantly reducing listening effort and improving accessibility for millions of viewers globally.
Cresta / OpenAI
Cresta, founded in 2017 by Stanford PhD students with OpenAI research experience, developed an AI copilot system for contact center agents that provides real-time suggestions during customer conversations. The company tackled the challenge of transforming academic NLP and reinforcement learning research into production-grade enterprise software by building domain-specific models fine-tuned on customer conversation data. Starting with Intuit as their first customer through an unconventional internship arrangement, they demonstrated measurable ROI through A/B testing, showing improved conversion rates and agent productivity. The solution evolved from custom LSTM and transformer models to leveraging pre-trained foundation models like GPT-3/4 with fine-tuning, ultimately serving Fortune 500 customers across telecommunications, airlines, and banking with demonstrated value including a pilot generating $100 million in incremental revenue.
Roblox
Roblox moderates billions of pieces of user-generated content daily across 28 languages using a sophisticated AI-driven system that combines large transformer-based models with human oversight. The platform processes an average of 6.1 billion chat messages and 1.1 million hours of voice communication per day, requiring ML models that can make moderation decisions in milliseconds. The system achieves over 750,000 requests per second for text filtering, with specialized models for different violation types (PII, profanity, hate speech). The solution integrates GPU-based serving infrastructure, model quantization and distillation for efficiency, real-time feedback mechanisms that reduce violations by 5-6%, and continuous model improvement through diverse data sampling strategies including synthetic data generation via LLMs, uncertainty sampling, and AI-assisted red teaming.
Faire
Faire, a wholesale marketplace connecting brands and retailers, implemented multiple AI initiatives across their engineering organization to enhance both internal developer productivity and external customer-facing features. The company deployed agentic development workflows using GitHub Copilot and custom orchestration systems to automate repetitive coding tasks, introduced natural-language and image-based search capabilities for retailers seeking products, and built a hybrid Python-Kotlin architecture to support multi-step AI agents that compose purchasing recommendations. These efforts aimed to reduce manual workflows, accelerate product discovery, and deliver more personalized experiences for their wholesale marketplace customers.
Delivery Hero
Delivery Hero built a comprehensive AI-powered image generation system to address the problem that 86% of food products lacked images, which significantly impacted conversion rates. The solution involved implementing both text-to-image generation and image inpainting workflows using Stable Diffusion models, with extensive optimization for cost efficiency and quality assurance. The system successfully generated over 100,000 production images, achieved 6-8% conversion rate improvements, and reduced costs to under $0.003 per image through infrastructure optimization and model fine-tuning.
Volkswagen
Volkswagen Group Services partnered with AWS to build a production-scale generative AI platform for automotive marketing content generation and compliance evaluation. The problem was a slow, manual content supply chain that took weeks to months, created confidentiality risks with pre-production vehicles, and faced massive compliance bottlenecks across 10 brands and 200+ countries. The solution involved fine-tuning diffusion models on proprietary vehicle imagery (including digital twins from CAD), automated prompt enhancement using LLMs, and multi-stage image evaluation using vision-language models for both component-level accuracy and brand guideline compliance. Results included massive time savings (weeks to minutes), automated compliance checks across legal and brand requirements, and a reusable shared platform supporting multiple use cases across the organization.
Cedars Sinai
Cedars Sinai and various academic institutions have implemented AI and machine learning solutions to improve neurosurgical outcomes across multiple areas. The applications include brain tumor classification using CNNs achieving 95% accuracy (surpassing traditional radiologists), hematoma prediction and management using graph neural networks with 80%+ accuracy, and AI-assisted surgical planning and intraoperative guidance. The implementations demonstrate significant improvements in patient outcomes while highlighting the importance of balanced innovation with appropriate regulatory oversight.
Fitbit
Fitbit developed an AI-powered personal health coach to address the fragmented and generic nature of traditional health and fitness guidance. Using Gemini models within a multi-agent framework, the system provides proactive, personalized, and adaptive coaching grounded in behavioral science and individual health metrics such as sleep and activity data. The solution employs a conversational agent for orchestration, a data science agent for numerical reasoning on physiological time series, and domain expert agents for specialized guidance. The system underwent extensive validation through the SHARP evaluation framework, involving over 1 million human annotations and 100k hours of expert evaluation across multiple health disciplines. The health coach entered public preview for eligible US-based Fitbit Premium users, providing personalized insights, goal setting, and adaptive plans to build sustainable health habits.
LinkedIn transformed their traditional keyword-based job search into an AI-powered semantic search system to serve 1.2 billion members. The company addressed limitations of exact keyword matching by implementing a multi-stage LLM architecture combining retrieval and ranking models, supported by synthetic data generation, GPU-optimized embedding-based retrieval, and cross-encoder ranking models. The solution enables natural language job queries like "Find software engineer jobs that are mostly remote with above median pay" while maintaining low latency and high relevance at massive scale through techniques like model distillation, KV caching, and exhaustive GPU-based nearest neighbor search.
LinkedIn deployed a sophisticated machine learning pipeline to extract and map skills from unstructured content across their platform (job postings, profiles, resumes, learning courses) to power their Skills Graph. The solution combines token-based and semantic skill tagging using BERT-based models, multitask learning frameworks for domain-specific scoring, and knowledge distillation to serve models at scale while meeting strict latency requirements (100ms for 200 profile edits/second). Product-driven feedback loops from recruiters and job seekers continuously improve model performance, resulting in measurable business impact including 0.46% increase in predicted confirmed hires for job recommendations and 0.76% increase in PPC revenue for job search.
eSpark
eSpark, an adaptive learning platform for K-5 students, developed an LLM-powered teacher assistant to address a critical post-COVID challenge: school administrators were emphasizing expensive core curricula investments while relegating supplemental programs like eSpark to secondary status. The team built a RAG-based recommendation system that matches eSpark's 15 years of curated content with hundreds of different core curricula, enabling teachers to seamlessly integrate eSpark activities with their mandated lesson plans. Through continuous teacher interviews and iterative development, they evolved from a conversational chatbot interface (which teachers found overwhelming) to a streamlined dropdown-based system with AI-generated follow-up questions. The solution leverages embeddings databases, tool-calling agents, and a sophisticated eval framework using Brain Trust for testing across hundreds of curricula, ultimately helping teachers work more efficiently while keeping eSpark relevant in a changing educational landscape.
Google Docs implemented automatic document summary generation to help users manage the volume of documents they receive daily. The challenge was to create concise, high-quality summaries that capture document essence while maintaining writer control over the final output. Google developed a solution based on Pegasus, a Transformer-based abstractive summarization model with custom pre-training, combined with careful data curation focusing on quality over quantity, knowledge distillation to optimize serving efficiency (distilling to a Transformer encoder + RNN decoder hybrid), and TPU-based serving infrastructure. The feature was launched for Google Workspace business customers, providing 1-2 sentence suggestions that writers can accept, edit, or ignore, helping both document creators and readers navigate content more efficiently.
NVIDIA
NVIDIA engineers developed a novel approach to automatically generate optimized GPU attention kernels using the DeepSeek-R1 language model combined with inference-time scaling. They implemented a closed-loop system where the model generates code that is verified and refined through multiple iterations, achieving 100% accuracy for Level-1 problems and 96% for Level-2 problems in Stanford's KernelBench benchmark. This approach demonstrates how additional compute resources during inference can improve code generation capabilities of LLMs.
JetBlue
JetBlue faced challenges in manually tuning prompts across complex, multi-stage LLM pipelines for applications like customer feedback classification and RAG-powered predictive maintenance chatbots. The airline adopted DSPy, a framework for building self-optimizing LLM pipelines, integrated with Databricks infrastructure including Model Serving and Vector Search. By leveraging DSPy's automatic optimization capabilities and modular architecture, JetBlue achieved 2x faster RAG chatbot deployment compared to their previous Langchain implementation, eliminated manual prompt engineering, and enabled automatic optimization of pipeline quality metrics using LLM-as-a-judge evaluations, resulting in more reliable and efficient LLM applications at scale.
Heidelberg University
Researchers at Heidelberg University developed a novel approach to address the growing workload of radiologists by automating the generation of detailed radiology reports from medical images. They implemented a system using Vision Transformers for image analysis combined with a fine-tuned Llama 3 model for report generation. The solution achieved promising results with a training loss of 0.72 and validation loss of 1.36, demonstrating the potential for efficient, high-quality report generation while running on a single GPU through careful optimization techniques.
Samsung
Samsung is implementing a comprehensive LLMOps system for autonomous semiconductor fabrication, using multi-modal LLMs and reinforcement learning to transform manufacturing processes. The system combines sensor data analysis, knowledge graphs, and LLMs to automate equipment control, defect detection, and process optimization. Early results show significant improvements in areas like RF matching efficiency and anomaly detection, though challenges remain in real-time processing and time series prediction accuracy.
Bismuth
Bismuth, a startup focused on software agents, developed SM-100, a comprehensive benchmark to evaluate AI agents' capabilities in software maintenance tasks, particularly bug detection and fixing. The benchmark revealed significant limitations in existing popular agents, with most achieving only 7% accuracy in finding complex bugs and exhibiting high false positive rates (90%+). While agents perform well on feature development benchmarks like SWE-bench, they struggle with real-world maintenance tasks that require deep system understanding, cross-file reasoning, and holistic code evaluation. Bismuth's own agent achieved better performance (10 out of 100 bugs found vs. 7 for the next best), demonstrating that targeted improvements in model architecture, prompting strategies, and navigation techniques can enhance bug detection capabilities in production software maintenance scenarios.
Instacart
Instacart built a centralized contextual retrieval system powered by BERT-like transformer models to provide real-time product recommendations across multiple shopping surfaces including search, cart, and item detail pages. The system replaced disparate legacy retrieval systems that relied on ad-hoc combinations of co-occurrence, similarity, and popularity signals with a unified approach that predicts next-product probabilities based on in-session user interaction sequences. The solution achieved a 30% lift in user cart additions for cart recommendations, 10-40% improvement in Recall@K metrics over randomized sequence baselines, and enabled deprecation of multiple legacy ad-hoc retrieval systems while serving both ads and organic recommendation surfaces.
Doordash
DoorDash addressed the challenge of behavioral silos in their multi-vertical marketplace, where customers have deep interaction history in some categories (like restaurants) but sparse data in others (like grocery or retail). They built an LLM-powered framework using hierarchical RAG to translate restaurant orders and search queries into cross-vertical affinity features aligned with their product taxonomy. These semantic features were integrated into their production multi-task ranking models. The approach delivered consistent improvements both offline and online: approximately 4.4% improvement in AUC-ROC and 4.8% in MRR offline, with similar gains in production (+4.3% AUC-ROC, +3.2% MRR). The solution proved particularly effective for cold-start scenarios while maintaining practical inference costs through prompt optimization, caching strategies, and use of smaller language models like GPT-4o-mini.
Grab
Grab developed a custom lightweight vision LLM to address the challenges of extracting information from diverse user-submitted documents like ID cards and driver's licenses across Southeast Asia. Traditional OCR systems struggled with the variety of document templates and languages, while proprietary LLMs had high latency and poor SEA language support. The team fine-tuned and ultimately built a custom ~1B parameter vision LLM from scratch, achieving performance comparable to larger 2B models while significantly reducing latency. The solution involved a four-stage training process using synthetic OCR datasets, an auto-labeling framework called Documint, and full-parameter fine-tuning, resulting in dramatic accuracy improvements (+70pp for Thai, +40pp for Vietnamese) and establishing a unified model to replace traditional OCR pipelines.
Shopify
Shopify addressed the challenge of fragmented product data across millions of merchants by building a Global Catalogue using multimodal LLMs to standardize and enrich billions of product listings. The system processes over 10 million product updates daily through a four-layer architecture involving product data foundation, understanding, matching, and reconciliation. By fine-tuning open-source vision language models and implementing selective field extraction, they achieve 40 million LLM inferences daily with 500ms median latency while reducing GPU usage by 40%. The solution enables improved search, recommendations, and conversational commerce experiences across Shopify's ecosystem.
Doordash
Doordash developed a system to automatically transcribe restaurant menu photos using LLMs, addressing the challenge of maintaining accurate menu information on their delivery platform. Instead of relying solely on LLMs, they created an innovative guardrail framework using traditional machine learning to evaluate transcription quality and determine whether AI or human processing should be used. This hybrid approach allowed them to achieve high accuracy while maintaining efficiency and adaptability to new AI models.
Cursor
Cursor developed Composer, a specialized coding agent model designed to balance speed and intelligence for real-world software engineering tasks. The challenge was creating a model that could perform at near-frontier levels while being four times more efficient at token generation than comparable models, moving away from the "airplane Wi-Fi" problem where agents were either too slow for synchronous work or required long async waits. The solution involved extensive reinforcement learning (RL) training in an environment that closely mimicked production, using custom kernels for low-precision training, parallel tool calling capabilities, semantic search with custom embeddings, and a fleet of cloud VMs to simulate the real Cursor IDE environment. The result was a model that performs close to frontier models like GPT-4.5 and Claude Sonnet 3.5 on coding benchmarks while maintaining significantly faster token generation, enabling developers to stay in flow state rather than context-switching during long agent runs.
Autodesk
Autodesk built a machine learning platform from scratch using Metaflow as the foundation for their managed training infrastructure. The platform enables data scientists to construct end-to-end ML pipelines, with particular focus on distributed training of large language models. They successfully integrated AWS services, implemented security measures, and created a user-friendly interface that supported both experimental and production workflows. The platform has been rolled out to 50 users and demonstrated successful fine-tuning of large language models, including a 6B parameter model in 50 minutes using 16 A10 GPUs.
Exa.ai
Exa.ai has built the first search engine specifically designed for AI agents rather than human users, addressing the fundamental problem that existing search engines like Google are optimized for consumer clicks and keyword-based queries rather than semantic understanding and agent workflows. The company trained its own models, built its own index, and invested heavily in compute infrastructure (including purchasing their own GPU cluster) to enable meaning-based search that returns raw, primary data sources rather than listicles or summaries. Their solution includes both an API for developers building AI applications and an agentic search tool called Websites that can find and enrich complex, multi-criteria queries. The results include serving hundreds of millions of queries across use cases like sales intelligence, recruiting, market research, and research paper discovery, with 95% inbound growth and expanding from 7 to 28+ employees within a year.
Weights & Biases
A developer built a custom voice assistant similar to Alexa using open-source LLMs, demonstrating the journey from prototype to production-ready system. The project used Whisper for speech recognition and various LLM models (Llama 2, Mistral) running on consumer hardware, with systematic improvements through prompt engineering and fine-tuning to achieve 98% accuracy in command interpretation, showing how iterative improvement and proper evaluation frameworks are crucial for LLM applications.
Cursor
Cursor, an AI-powered code editor startup, entered an extremely competitive market dominated by Microsoft's GitHub Copilot and well-funded competitors like Poolside, Augment, and Magic.dev. Despite initial skepticism from advisors about competing against Microsoft's vast resources and distribution, Cursor succeeded by focusing on the right short-term product decisions—specifically deep IDE integration through forking VS Code and delivering immediate value through "Cursor Tab" code completion. The company differentiated itself through rapid iteration, concentrated talent, bottom-up adoption among developers, and eventually building their own fast agent models. Cursor demonstrated that startups can compete against tech giants by moving quickly, dog-fooding their own product, and correctly identifying what developers need in the near term rather than betting solely on long-term agent capabilities.
Cursor
Cursor, an AI-powered IDE built by Anysphere, faced the challenge of scaling from zero to serving billions of code completions daily while handling 1M+ queries per second and 100x growth in load within 12 months. The solution involved building a sophisticated architecture using TypeScript and Rust, implementing a low-latency sync engine for autocomplete suggestions, utilizing Merkle trees and embeddings for semantic code search without storing source code on servers, and developing Anyrun, a Rust-based orchestrator service. The results include reaching $500M+ in annual revenue, serving more than half of the Fortune 500's largest tech companies, and processing hundreds of millions of lines of enterprise code written daily, all while maintaining privacy through encryption and secure indexing practices.
Product Talk
Teresa Torres, founder of Product Talk, describes her journey building an AI interview coach over four months to help students in her Continuous Discovery course practice customer interviewing skills. Starting from a position of limited AI engineering experience, she developed a production system that analyzes interview transcripts and provides detailed feedback across four dimensions of interviewing technique. The case study focuses extensively on her implementation of a comprehensive evaluation (eval) framework, including human annotation, code-based assertions, and LLM-as-judge evaluations, to ensure quality and reliability of the AI coach's feedback before deploying it to real students.
Devin
Cognition, the company behind Devon (an AI software engineer), addresses the challenge of enabling AI agents to work effectively within large, existing codebases where traditional LLMs struggle with limited context windows and complex dependencies. Their solution involves creating DeepWiki, a continuously-updated interactive knowledge graph and wiki system that indexes codebases using both code and metadata (pull requests, git history, team discussions), combined with Devon Search for deep codebase research, and custom post-training using multi-turn reinforcement learning to optimize models for specific narrow domains. Results include Devon being used by teams worldwide to autonomously go from ticket to pull request, the release of Kevin 32B (an open-source model achieving 91% correctness on CUDA kernel generation, outperforming frontier models like GPT-4), and thousands of open-source projects incorporating DeepWiki into their official documentation.
LinkedIn developed Hiring Assistant, an AI agent designed to transform the recruiting workflow by automating repetitive tasks like candidate sourcing, evaluation, and engagement across 1.2+ billion profiles. The system addresses the challenge of recruiters spending excessive time on pattern-recognition tasks rather than high-value decision-making and relationship building. Using a plan-and-execute agent architecture with specialized sub-agents for intake, sourcing, evaluation, outreach, screening, and learning, Hiring Assistant combines real-time conversational interfaces with large-scale asynchronous execution. The solution leverages LinkedIn's Economic Graph for talent insights, custom fine-tuned LLMs for candidate evaluation, and cognitive memory systems that learn from recruiter behavior over time. The result is a globally available agentic product that enables recruiters to work with greater speed, scale, and intelligence while maintaining human-in-the-loop control for critical decisions.
Replit
Replit, a software development platform, aimed to democratize coding by developing their own code completion LLM. Using Databricks' Mosaic AI Training infrastructure, they successfully built and deployed a multi-billion parameter model in just three weeks, enabling them to launch their code completion feature on time with a small team. The solution allowed them to abstract away infrastructure complexity and focus on model development, resulting in a production-ready code generation system that serves their 25 million users.
Mistral
Mistral, a European AI company, evolved from developing academic LLMs to building and deploying enterprise-grade language models. They started with the successful launch of Mistral-7B in September 2023, which became one of the top 10 most downloaded models on Hugging Face. The company focuses not just on model development but on providing comprehensive solutions for enterprise deployment, including custom fine-tuning, on-premise deployment infrastructure, and efficient inference optimization. Their approach demonstrates the challenges and solutions in bringing LLMs from research to production at scale.
Google Deepmind
This case study explores the evolution of LLM-based systems in production through discussions with Raven Kumar from Google DeepMind about building products like Notebook LM, Project Mariner, and working with the Gemini and Gemma model families. The conversation covers the rapid progression from simple function calling to complex agentic systems capable of multi-step reasoning, the critical importance of evaluation harnesses as competitive advantages, and practical considerations around context engineering, tool orchestration, and model selection. Key insights include how model improvements are causing teams to repeatedly rebuild agent architectures, the importance of shipping products quickly to learn from real users, and strategies for evaluating increasingly complex multi-modal agentic systems across different scales from edge devices to cloud-based deployments.
LinkedIn's journey in developing their GenAI application tech stack, transitioning from simple prompt-based solutions to complex conversational agents. The company evolved from Java-based services to a Python-first approach using LangChain, implemented comprehensive prompt management, developed a skill-based task automation framework, and built robust conversational memory infrastructure. This transformation included migrating existing applications while maintaining production stability and enabling both commercial and fine-tuned open-source LLM deployments.
Microsoft
Microsoft's Skilling organization built "Ask Learn," a retrieval-augmented generation (RAG) system that powers AI-driven question-answering capabilities for Microsoft Q&A and serves as ground truth for Microsoft Copilot for Azure. Starting from a 2023 hackathon project, the team evolved a naïve RAG implementation into an advanced RAG system featuring sophisticated pre- and post-processing pipelines, continuous content ingestion from Microsoft Learn documentation, vector database management, and comprehensive evaluation frameworks. The system handles massive scale, provides accurate and verifiable answers, and serves multiple use cases including direct question answering, grounding data for other chat handlers, and fallback functionality when the Copilot cannot complete requested tasks.
Anthropic
Anthropic's Boris Churnney, creator of Claude Code, describes the journey from an accidental terminal prototype in September 2024 to a production coding tool used by 70% of startups and responsible for 4% of all public commits globally. Starting as a simple API testing tool, Claude Code evolved through continuous user feedback and rapid iteration, with the entire codebase rewritten every few months to adapt to improving model capabilities. The tool achieved remarkable productivity gains at Anthropic itself, with engineers seeing 70% productivity increases per capita despite team doubling, and total productivity improvements of 150% since launch. The development philosophy centered on building for future model capabilities rather than current ones, anticipating improvements 6 months ahead, and minimizing scaffolding that would become obsolete with each new model release.
Cursor
Cursor's AI research team built Composer, an agent-based LLM designed for coding that combines frontier-level intelligence with four times faster token generation than comparable models. The problem they addressed was creating an agentic coding assistant that feels fast enough for interactive use while maintaining high intelligence for realistic software engineering tasks. Their solution involved training a large mixture-of-experts model using reinforcement learning (RL) at scale, developing custom low-precision training kernels, and building infrastructure that integrates their production environment directly into the training loop. The result is a model that performs nearly as well as the best frontier models on their internal benchmarks while delivering edits and tool calls in seconds rather than minutes, fundamentally changing how developers interact with AI coding assistants.
Tzafon
Tzafon, a research lab focused on training foundation models for computer use agents, tackled the challenge of enabling LLMs to autonomously interact with computers through visual understanding and action execution. The company identified fundamental limitations in existing models' ability to ground visual information and coordinate actions, leading them to develop custom infrastructure (Waypoint) for data generation at scale, fine-tune vision encoders on screenshot data, and ultimately pre-train models from scratch with specialized computer interaction capabilities. While initial approaches using supervised fine-tuning and reinforcement learning on successful trajectories showed limited generalization, their focus on solving the grounding problem through improved vision-language integration and domain-specific pre-training has positioned them to release models and desktop applications for autonomous computer use, though performance on benchmarks like OS World remains a challenge across the industry.
eBay
eBay developed a hybrid system for pricing recommendations and similar item search in their marketplace, specifically focusing on sports trading cards. They combined semantic similarity models with direct price prediction approaches, using transformer-based architectures to create embeddings that balance both price accuracy and item similarity. The system helps sellers price their items accurately by finding similar items that have sold recently, while maintaining semantic relevance.
AlixPartners
A technical consultant presents a comprehensive workshop on using DSPy, a declarative framework for building modular LLM-powered applications in production. The presenter demonstrates how DSPy enables rapid iteration on LLM applications by treating LLMs as first-class citizens in Python programs, with built-in support for structured outputs, type guarantees, tool calling, and automatic prompt optimization. Through multiple real-world use cases including document classification, contract analysis, time entry correction, and multi-modal processing, the workshop shows how DSPy's core primitives—signatures, modules, tools, adapters, optimizers, and metrics—allow teams to build production-ready systems that are transferable across models, optimizable without fine-tuning, and maintainable at scale.
Upwork
Upwork developed Uma, their "mindful AI" assistant, by rejecting off-the-shelf LLM solutions in favor of building custom-trained models using proprietary platform data and in-house AI research. The company hired expert freelancers to create high-quality training datasets, generated synthetic data anchored in real platform interactions, and fine-tuned open-source LLMs specifically for hiring workflows. This approach enabled Uma to handle complex, business-critical tasks including crafting job posts, matching freelancers to opportunities, autonomously coordinating interviews, and evaluating candidates. The strategy resulted in models that substantially outperform generic alternatives on domain-specific tasks while reducing costs by up to 10x and improving reliability in production environments. Uma now operates as an increasingly agentic system that takes meaningful actions across the full hiring lifecycle.
Various
Climate tech startups are leveraging Amazon SageMaker HyperPod to build specialized foundation models that address critical environmental challenges including weather prediction, sustainable material discovery, ecosystem monitoring, and geological modeling. Companies like Orbital Materials and Hum.AI are training custom models from scratch on massive environmental datasets, achieving significant breakthroughs such as tenfold performance improvements in carbon capture materials and the ability to see underwater from satellite imagery. These startups are moving beyond traditional LLM fine-tuning to create domain-specific models with billions of parameters that process multimodal environmental data including satellite imagery, sensor networks, and atmospheric measurements at scale.
Rolls-Royce
Rolls-Royce implemented a cloud-based generative AI approach using GANs (Generative Adversarial Networks) to support preliminary engineering design tasks. The system combines geometric parameters and simulation data to generate and validate new design concepts, with a particular focus on aerospace applications. By leveraging Databricks' cloud infrastructure, they reduced training time from one week to 4-6 hours while maintaining data security through careful governance and transfer learning approaches.
Contextual
Contextual has developed an end-to-end context engineering platform designed to address the challenges of building production-ready RAG and agentic systems across multiple domains including e-commerce, code generation, and device testing. The platform combines multimodal ingestion, hierarchical document processing, hybrid search with reranking, and dynamic agents to enable effective reasoning over large document collections. In a recent context engineering hackathon, Contextual's dynamic agent achieved competitive results on a retail dataset of nearly 100,000 documents, demonstrating the value of constrained sub-agents, turn limits, and intelligent tool selection including MCP server management.
ChromaDB
ChromaDB's technical report examines how large language models (LLMs) experience performance degradation as input context length increases, challenging the assumption that models process context uniformly. Through evaluation of 18 state-of-the-art models including GPT-4.1, Claude 4, Gemini 2.5, and Qwen3 across controlled experiments, the research reveals that model reliability decreases significantly with longer inputs, even on simple tasks like retrieval and text replication. The study demonstrates that factors like needle-question similarity, presence of distractors, haystack structure, and semantic relationships all impact performance non-uniformly as context length grows, suggesting that current long-context benchmarks may not adequately reflect real-world performance challenges.
Windsurf
Windsurf, an AI coding toolkit company, addresses the challenge of generating contextually relevant code for individual developers and organizations. While generating generic code has become straightforward, the real challenge lies in producing code that fits into existing large codebases, adheres to organizational standards, and aligns with personal coding preferences. Windsurf's solution centers on a sophisticated context management system that combines user behavioral heuristics (cursor position, open files, clipboard content, terminal activity) with hard evidence from the codebase (code, documentation, rules, memories). Their approach optimizes for relevant context selection rather than simply expanding context windows, leveraging their background in GPU optimization to efficiently find and process relevant context at scale.
LinkedIn faced the challenge that while AI coding agents were powerful, they lacked organizational context about the company's thousands of microservices, internal frameworks, data infrastructure, and specialized systems. To address this, they built CAPT (Contextual Agent Playbooks & Tools), a unified framework built on the Model Context Protocol (MCP) that provides AI agents with access to internal tools and executable playbooks encoding institutional workflows. The system enables over 1,000 engineers to perform complex tasks like experiment cleanup, data analysis, incident debugging, and code review with significant productivity gains: 70% reduction in issue triage time, 3× faster data analysis workflows, and automated debugging that cuts time spent by more than half in many cases.
Lmsys
Intel PyTorch Team collaborated with the SGLang project to develop a cost-effective CPU-only deployment solution for large Mixture of Experts (MoE) models like DeepSeek R1, addressing the challenge of high memory requirements that typically necessitate multiple expensive AI accelerators. Their solution leverages Intel Xeon 6 processors with Advanced Matrix Extensions (AMX) and implements highly optimized kernels for attention mechanisms and MoE computations, achieving 6-14x speedup in time-to-first-token (TTFT) and 2-4x speedup in time-per-output-token (TPOT) compared to llama.cpp, while supporting multiple quantization formats including BF16, INT8, and FP8.
Nvidia
NVIDIA implemented a data flywheel approach to optimize their internal employee support AI agent, addressing the challenge of maintaining accuracy while reducing inference costs. The system continuously collects user feedback and production data to fine-tune smaller, more efficient models that can replace larger, expensive foundational models. Through this approach, they achieved comparable accuracy (94-96%) with significantly smaller models (1B-8B parameters instead of 70B), resulting in 98% cost savings and 70x lower latency while maintaining the agent's effectiveness in routing employee queries across HR, IT, and product documentation domains.
Bayezian Limited
Bayezian Limited deployed a multi-agent AI system to monitor protocol deviations in clinical trials, where traditional manual review processes were time-consuming and error-prone. The system used specialized LLM agents, each responsible for checking specific protocol rules (visit timing, medication use, inclusion criteria, etc.), working on top of a pipeline that processed clinical documents and used FAISS for semantic retrieval of protocol requirements. While the system successfully identified patterns early and improved reviewer efficiency by shifting focus from manual checking to intelligent triage, it encountered significant challenges including handover failures between agents, memory lapses causing coordination breakdowns, and difficulties handling real-world data ambiguities like time windows and exceptions. The team improved performance through structured memory snapshots, flexible prompt engineering, stronger handoff signals, and process tracking, ultimately creating a useful but imperfect system that highlighted the gap between agentic AI theory and production reality.
Nvidia
Financial institutions including Capital One, Royal Bank of Canada (RBC), and Visa are deploying agentic AI systems in production to handle real-time financial transactions and complex workflows. These multi-agent systems go beyond simple generative AI by reasoning through problems and taking action autonomously, requiring 100-200x more computational resources than traditional single-shot inference. The implementations focus on use cases like automotive purchasing assistance, investment research automation, and fraud detection, with organizations building proprietary models using open-source foundations (like Llama or Mistral) combined with bank-specific data to achieve 60-70% accuracy improvements. The results include 60% cycle time improvements in report generation, 10x more data analysis capacity, and enhanced fraud detection capabilities, though these gains require substantial investment in AI infrastructure and talent development.
Trigent Software
Trigent Software attempted to develop IRGPT, a fine-tuned LLM for multilingual Ayurvedic medical consultations. The project aimed to combine traditional Ayurvedic medicine with modern AI capabilities, targeting multiple South Indian languages. Despite assembling a substantial dataset and implementing a fine-tuning pipeline using GPT-2 medium, the team faced significant challenges with multilingual data quality and cultural context. While the English-only version showed promise, the full multilingual implementation remains a work in progress.
eBay
eBay tackled the challenge of incorporating LLMs into their e-commerce platform by developing e-Llama, a domain-adapted version of Llama 3.1. Through continued pre-training on a mix of e-commerce and general domain data, they created 8B and 70B parameter models that achieved 25% improvement in e-commerce tasks while maintaining strong general performance. The training was completed efficiently using 480 NVIDIA H100 GPUs and resulted in production-ready models aligned with human feedback and safety requirements.
Wix
Wix developed a customized LLM for their enterprise needs by applying multi-task supervised fine-tuning (SFT) and domain adaptation using full weights fine-tuning (DAPT). Despite having limited data and tokens, their smaller customized model outperformed GPT-3.5 on various Wix-specific tasks. The project focused on three key components: comprehensive evaluation benchmarks, extensive data collection methods, and advanced modeling processes to achieve full domain adaptation capabilities.
Ebay
eBay developed customized large language models by adapting Meta's Llama 3.1 models (8B and 70B parameters) to the e-commerce domain through continued pretraining on a mixture of proprietary eBay data and general domain data. This hybrid approach allowed them to infuse domain-specific knowledge while avoiding the resource intensity of training from scratch. Using 480 NVIDIA H100 GPUs and advanced distributed training techniques, they trained the models on 1 trillion tokens, achieving approximately 25% improvement on e-commerce benchmarks for English (30% for non-English) with only 1% degradation on general domain tasks. The resulting "e-Llama" models were further instruction-tuned and aligned with human feedback to power various AI initiatives across the company in a cost-effective, scalable manner.
Meta / Ray Ban
Meta Reality Labs developed a production AI system for Ray-Ban Meta smart glasses that brings AI capabilities directly to wearable devices through a four-part architecture combining on-device processing, smartphone connectivity, and cloud-based AI services. The system addresses unique challenges of wearable AI including power constraints, thermal management, connectivity limitations, and real-time performance requirements while enabling features like visual question answering, photo capture, and voice commands with sub-second response times for on-device operations and under 3-second response times for cloud-based AI interactions.
Wayve
Wayve is developing self-driving technology that works across multiple vehicle types and global markets by leveraging end-to-end foundation models trained on driving data rather than traditional rule-based systems. The company moved away from intermediate representations like object detection to a more holistic approach where a single neural network learns to drive from examples, similar to how large language models learn language. This architecture enabled rapid global expansion from primarily driving in London to operating across 500 cities in Japan, Europe, the UK, and the US within a year. The system uses foundation models for multiple tasks including driving, simulation, scenario classification, and even natural language explanations of driving decisions, with all components compressed into a single 75-watt model deployable in production vehicles.
Pinterest improved their ads engagement modeling by implementing a Multi-gate Mixture-of-Experts (MMoE) architecture combined with knowledge distillation techniques. The system faced challenges with short data retention periods and computational efficiency, which they addressed through mixed precision inference and lightweight gate layers. The solution resulted in significant improvements in both offline accuracy and online metrics while achieving a 40% reduction in inference latency.
Rubrik
Predibase, a fine-tuning and model serving platform, announced its acquisition by Rubrik, a data security and governance company, with the goal of combining Predibase's generative AI capabilities with Rubrik's secure data infrastructure. The integration aims to address the critical challenge that over 50% of AI pilots never reach production due to issues with security, model quality, latency, and cost. By combining Predibase's post-training and inference capabilities with Rubrik's data security posture management, the merged platform seeks to provide an end-to-end solution that enables enterprises to deploy generative AI applications securely and efficiently at scale.
DeepL
DeepL, a translation company founded in 2017, has built a successful enterprise-focused business using neural machine translation models to tackle the language barrier problem at scale. The company handles hundreds of thousands of customers by developing specialized neural translation models that balance accuracy and fluency, training them on curated parallel and monolingual corpora while leveraging context injection rather than per-customer fine-tuning for scalability. By building their own GPU infrastructure early on and developing custom frameworks for inference optimization, DeepL maintains a competitive edge over general-purpose LLMs and established players like Google Translate, demonstrating strong product-market fit in high-stakes enterprise use cases where translation quality directly impacts legal compliance, customer experience, and business operations.
Smartling
Smartling operates an enterprise-scale AI-first agentic translation delivery platform serving major corporations like Disney and IBM. The company addresses challenges around automation, centralization, compliance, brand consistency, and handling diverse content types across global markets. Their solution employs multi-step agentic workflows where different model functions validate each other's outputs, combining neural machine translation with large language models, RAG for accessing validated linguistic assets, sophisticated prompting, and automated post-editing for hyper-localization. The platform demonstrates measurable improvements in throughput (from 2,000 to 6,000-7,000 words per day), cost reduction (4-10x cheaper than human translation), and quality approaching 70% human parity for certain language pairs and content types, while maintaining enterprise requirements for repeatability, compliance, and brand voice consistency.
Wesco
Wesco, a B2B supply chain and industrial distribution company, presents a comprehensive case study on deploying enterprise-grade AI applications at scale, moving from POC to production. The company faced challenges in transitioning from traditional predictive analytics to cognitive intelligence using generative AI and agentic systems. Their solution involved building a composable AI platform with proper governance, MLOps/LLMOps pipelines, and multi-agent architectures for use cases ranging from document processing and knowledge retrieval to fraud detection and inventory management. Results include deployment of 50+ use cases, significant improvements in employee productivity through "everyday AI" applications, and quantifiable ROI through transformational AI initiatives in supply chain optimization, with emphasis on proper observability, compliance, and change management to drive adoption.
Factory AI
Factory AI developed an evaluation framework to assess context compression strategies for AI agents working on extended software development tasks that generate millions of tokens across hundreds of messages. The company compared three approaches—their structured summarization method, OpenAI's compact endpoint, and Anthropic's built-in compression—using probe-based evaluation that tests factual retention, file tracking, task planning, and reasoning chains. Testing on over 36,000 production messages from debugging, code review, and feature implementation sessions, Factory's structured summarization approach scored 3.70 overall compared to 3.44 for Anthropic and 3.35 for OpenAI, demonstrating superior retention of technical details like file paths and error messages while maintaining comparable compression ratios.
Microsoft
Microsoft worked with an advertising customer to enable 1:1 ad personalization while ensuring product image integrity in AI-generated content. They developed a comprehensive evaluation system combining template matching, Mean Squared Error (MSE), Peak Signal to Noise Ratio (PSNR), and Cosine Similarity to verify that AI-generated backgrounds didn't alter the original product images. The solution successfully enabled automatic verification of product image fidelity in AI-generated advertising materials.
Writer
Writer, an enterprise AI platform company, evolved their retrieval-augmented generation (RAG) system from traditional vector search to a sophisticated graph-based approach to address limitations in handling dense, specialized enterprise data. Starting with keyword search and progressing through vector embeddings, they encountered accuracy issues with chunking and struggled with concentrated enterprise data where documents shared similar terminology. Their solution combined knowledge graphs with fusion-in-decoder techniques, using specialized models for graph structure conversion and storing graph data as JSON in Lucene-based search engines. This approach resulted in improved accuracy, reduced hallucinations, and better performance compared to seven different vector search systems in benchmarking tests.
NVIDA / Lepton
This lecture transcript from Yangqing Jia, VP at NVIDIA and founder of Lepton AI (acquired by NVIDIA), explores the evolution of AI system design from an engineer's perspective. The talk covers the progression from research frameworks (Caffe, TensorFlow, PyTorch) to production AI infrastructure, examining how LLM applications are built and deployed at scale. Jia discusses the emergence of "neocloud" infrastructure designed specifically for AI workloads, the challenges of GPU cluster management, and practical considerations for building consumer and enterprise LLM applications. Key insights include the trade-offs between open-source and closed-source models, the importance of RAG and agentic AI patterns, infrastructure design differences between conventional cloud and AI-specific platforms, and the practical challenges of operating LLMs in production, including supply chain management for GPUs and cost optimization strategies.
Cursor
This research presentation details four years of work developing evaluation methodologies for coding LLMs across varying time horizons, from second-level code completions to hour-long codebase translations. The speaker addresses critical challenges in evaluating production coding AI systems including data contamination, insufficient test suites, and difficulty calibration. Key solutions include LiveCodeBench's dynamic evaluation approach with periodically updated problem sets, automated test generation using LLM-driven approaches, and novel reward hacking detection systems for complex optimization tasks. The work demonstrates how evaluation infrastructure must evolve alongside model capabilities, incorporating intermediate grading signals, latency-aware metrics, and LLM-as-judge approaches to detect non-idiomatic coding patterns that pass traditional tests but fail real-world quality standards.
Hitachi
Hitachi's journey in implementing AI across industrial applications showcases the evolution from traditional machine learning to advanced generative AI solutions. The case study highlights how they transformed from focused applications in maintenance, repair, and operations to a more comprehensive approach integrating LLMs, focusing particularly on reliability, small data scenarios, and domain expertise. Key implementations include repair recommendation systems for fleet management and fault tree extraction from manuals, demonstrating the practical challenges and solutions in industrial AI deployment.
Various
A detailed case study of implementing LLMs in a supplier discovery product at Scoutbee, evolving from simple API integration to a sophisticated LLMOps architecture. The team tackled challenges of hallucinations, domain adaptation, and data quality through multiple stages: initial API integration, open-source LLM deployment, RAG implementation, and finally a comprehensive data expansion phase. The result was a production-ready system combining knowledge graphs, Chain of Thought prompting, and custom guardrails to provide reliable supplier discovery capabilities.
Mercado Libre
Mercado Libre (MELI) faced the challenge of categorizing millions of financial transactions across Latin America in multiple languages and formats as Open Finance unlocked access to customer financial data. Starting with a brittle regex-based system in 2021 that achieved only 60% accuracy and was difficult to maintain, they evolved through three generations: first implementing GPT-3.5 Turbo in 2023 to achieve 80% accuracy with 75% cost reduction, then transitioning to GPT-4o-mini in 2024, and finally developing custom BERT-based semantic embeddings trained on regional financial text to reach 90% accuracy with an additional 30% cost reduction. This evolution enabled them to scale from processing tens of millions of transactions per quarter to tens of millions per week, while enabling near real-time categorization that powers personalized financial insights across their ecosystem.
Impel
Impel, an automotive retail AI company, migrated from a third-party LLM to a fine-tuned Meta Llama model deployed on Amazon SageMaker to power their Sales AI product, which provides 24/7 personalized customer engagement for dealerships. The transition addressed cost predictability concerns and customization limitations, resulting in 20% improved accuracy across core features including response personalization, conversation summarization, and follow-up generation, while achieving better security and operational control.
Roots
Roots, an insurance AI company, developed and deployed fine-tuned 7B Mistral models in production using the vLLM framework to process insurance documents for entity extraction, classification, and summarization. The company evaluated multiple inference frameworks and selected vLLM for its performance advantages, achieving up to 130 tokens per second throughput on A100 GPUs with the ability to handle 32 concurrent requests. Their fine-tuned models outperformed GPT-4 on specialized insurance tasks while providing cost-effective processing at $30,000 annually for handling 20-30 million documents, demonstrating the practical benefits of self-hosting specialized models over relying on third-party APIs.
Robinhood Markets
Robinhood Markets developed a sophisticated LLMOps platform to deploy AI agents serving millions of users across multiple use cases including customer support, content generation (Cortex Digest), and code generation (custom indicators and scans). To address the "generative AI trilemma" of balancing cost, quality, and latency in production, they implemented a hierarchical tuning approach starting with prompt optimization, progressing to trajectory tuning with dynamic few-shot examples, and culminating in LoRA-based fine-tuning. Their CX AI agent achieved over 50% latency reduction (from 3-6 seconds to under 1 second) while maintaining quality parity with frontier models, supported by a comprehensive three-layer evaluation system combining LLM-as-judge, human feedback, and task-specific metrics.
Cosine
Cosine, a company building enterprise coding agents, faced the challenge of deploying high-performance AI systems in highly constrained environments including on-premise and air-gapped deployments where large frontier models were not viable. They developed a multi-agent architecture using specialized orchestrator and worker models, leveraging model distillation, supervised fine-tuning, preference optimization, and reinforcement fine-tuning to create smaller models that could match or exceed the performance of much larger models. The result was a 31% performance increase on the SWE-bench Freelancer benchmark, 3X latency improvement, 60% reduction in GPU footprint, and 20% fewer errors in generated code, all while operating on as few as 4 H100 GPUs and maintaining full deployment flexibility across cloud, VPC, and on-premise environments.
Large Gaming Company
AWS Professional Services helped a major gaming company build an automated toxic speech detection system by fine-tuning Large Language Models. Starting with only 100 labeled samples, they experimented with different BERT-based models and data augmentation techniques, ultimately moving from a two-stage to a single-stage classification approach. The final solution achieved 88% precision and 83% recall while reducing operational complexity and costs compared to the initial proof of concept.
Vannevar Labs
Vannevar Labs needed to improve their sentiment analysis capabilities for defense intelligence across multiple languages, finding that GPT-4 provided insufficient accuracy (64%) and high costs. Using Databricks Mosaic AI, they successfully fine-tuned a Mistral 7B model on domain-specific data, achieving 76% accuracy while reducing latency by 75%. The entire process from development to deployment took only two weeks, enabling efficient processing of multilingual content for defense-related applications.
Apoidea Group
Apoidea Group tackled the challenge of efficiently processing banking documents by developing a solution using multimodal large language models. They fine-tuned the Qwen2-VL-7B-Instruct model using LLaMA-Factory on Amazon SageMaker HyperPod to enhance visual information extraction from complex banking documents. The solution significantly improved table structure recognition accuracy from 23.4% to 81.1% TEDS score, approaching the performance of more advanced models while maintaining computational efficiency. This enabled reduction of financial spreading process time from 4-6 hours to just 10 minutes.
Nubank
Nubank developed a sophisticated approach to customer behavior modeling by combining transformer-based transaction embeddings with tabular data through supervised fine-tuning and joint fusion training. Starting with self-supervised pre-trained foundation models for transaction data, they implemented a DCNv2-based architecture that incorporates numerical and categorical feature embeddings to blend sequential transaction data with traditional tabular features. This joint fusion approach, which simultaneously optimizes the transformer and blending model during fine-tuning, outperforms both late fusion methods and standalone LightGBM models, achieving measurable improvements in AUC across multiple benchmark tasks while eliminating the need for manual feature engineering from sequential transaction data.
Meta
Meta developed GEM (Generative Ads Recommendation Model), an LLM-scale foundation model trained on thousands of GPUs to enhance ads recommendation across Facebook and Instagram. The model addresses challenges of sparse signals in billions of daily user-ad interactions, diverse multimodal data, and efficient large-scale training. GEM achieves 4x efficiency improvement over previous models through novel architecture innovations including stackable factorization machines, pyramid-parallel sequence processing, and cross-feature learning. The system employs sophisticated post-training knowledge transfer techniques achieving 2x the effectiveness of standard distillation, propagating learnings across hundreds of vertical models. Since launch in early 2025, GEM delivered a 5% increase in ad conversions on Instagram and 3% on Facebook Feed in Q2, with Q3 architectural improvements doubling performance gains from additional compute and data.
Netflix
Netflix developed a foundation model approach to centralize and scale their recommendation system, transitioning from multiple specialized models to a unified architecture. The system processes hundreds of billions of user interactions, employing sophisticated tokenization, sparse attention mechanisms, and incremental training to handle cold-start problems and new content. The model demonstrates successful scaling properties similar to LLMs, while maintaining production-level latency requirements and addressing unique challenges in recommendation systems.
Netflix
Netflix developed a foundation model for personalized recommendations to address the maintenance complexity and inefficiency of operating numerous specialized recommendation models. The company built a large-scale transformer-based model inspired by LLM paradigms that processes hundreds of billions of user interactions from over 300 million users, employing autoregressive next-token prediction with modifications for recommendation-specific challenges. The foundation model enables centralized member preference learning that can be fine-tuned for specific tasks, used directly for predictions, or leveraged through embeddings, while demonstrating clear scaling law benefits as model and data size increase, ultimately improving recommendation quality across multiple downstream applications.
Netflix
Netflix developed a unified foundation model based on transformer architecture to consolidate their diverse recommendation systems, which previously consisted of many specialized models for different content types, pages, and use cases. The foundation model uses autoregressive transformers to learn user representations from interaction sequences, incorporating multi-token prediction, multi-layer representation, and long context windows. By scaling from millions to billions of parameters over 2.5 years, they demonstrated that scaling laws apply to recommendation systems, achieving notable performance improvements while creating high leverage across downstream applications through centralized learning and easier fine-tuning for new use cases.
Uber
Uber faced significant challenges processing a high volume of invoices daily from thousands of global suppliers, with diverse formats, 25+ languages, and varying templates requiring substantial manual intervention. The company developed TextSense, a GenAI-powered document processing platform that leverages OCR, computer vision, and large language models (specifically OpenAI GPT-4 after evaluating multiple options including fine-tuned Llama 2 and Flan T5) to automate invoice data extraction. The solution achieved 90% overall accuracy, reduced manual processing by 2x, cut average handling time by 70%, and delivered 25-30% cost savings compared to manual processes, while providing a scalable, configuration-driven platform adaptable to diverse document types.
Doordash
DoorDash developed a GenAI-powered system to create personalized store carousels on their homepage, addressing limitations in their previous heuristic-based content system that featured only 300 curated carousels with insufficient diversity and overly broad categories. The new system leverages LLMs to analyze comprehensive consumer profiles and generate unique carousel titles with metadata for each user, then uses embedding-based retrieval to populate carousels with relevant stores and dishes. Early A/B tests in San Francisco and Manhattan showed double-digit improvements in click rates, improved conversion rates and homepage relevance metrics, and increased merchant discovery, particularly benefiting small and mid-sized businesses.
Google developed a three-generation evolution of AI-powered systems to transform 2D product images into interactive 3D visualizations for online shopping, culminating in a solution based on their Veo video generation model. The challenge was to replicate the tactile, hands-on experience of in-store shopping in digital environments while making the technology scalable and cost-effective for retailers. The latest approach uses Veo's diffusion-based architecture, fine-tuned on millions of synthetic 3D assets, to generate realistic 360-degree product spins from as few as one to three product images. This system now powers interactive 3D visualizations across multiple product categories on Google Shopping, significantly improving the online shopping experience by enabling customers to virtually inspect products from multiple angles.
Amazon
Amazon Prime Video addresses the challenge of differentiating their streaming platform in a crowded market by implementing multiple generative AI features powered by AWS services, particularly Amazon Bedrock. The solution encompasses personalized content recommendations, AI-generated episode recaps (X-Ray Recaps), real-time sports analytics insights, dialogue enhancement features, and automated video content understanding with metadata extraction. These implementations have resulted in improved content discoverability, enhanced viewer engagement through features that prevent spoilers while keeping audiences informed, deeper sports broadcast insights, increased accessibility through AI-enhanced audio, and enriched metadata for hundreds of thousands of marketing assets, collectively improving the overall streaming experience and reducing time spent searching for content.
Prosus / Microsoft / Inworld AI / IUD
This panel discussion features experts from Microsoft, Google Cloud, InWorld AI, and Brazilian e-commerce company IUD (Prosus partner) discussing the challenges of deploying reliable AI agents for e-commerce at scale. The panelists share production experiences ranging from Google Cloud's support ticket routing agent that improved policy adherence from 45% to 90% using DPO adapters, to Microsoft's shift away from prompt engineering toward post-training methods for all Copilot models, to InWorld AI's voice agent architecture optimization through cascading models, and IUD's struggles with personalization balance in their multi-channel shopping agent. Key challenges identified include model localization for UI elements, cost efficiency, real-time voice adaptation, and finding the right balance between automation and user control in commerce experiences.
Amazon Health Services
Amazon Health Services faced the challenge of integrating healthcare services into Amazon's e-commerce search experience, where traditional product search algorithms weren't designed to handle complex relationships between symptoms, conditions, treatments, and healthcare services. They developed a comprehensive solution combining machine learning for query understanding, vector search for product matching, and large language models for relevance optimization. The solution uses AWS services including Amazon SageMaker for ML models, Amazon Bedrock for LLM capabilities, and Amazon EMR for data processing, implementing a three-component architecture: query understanding pipeline to classify health searches, LLM-enhanced product knowledge base for semantic search, and hybrid relevance optimization using both human labeling and LLM-based classification. This system now serves daily health-related search queries, helping customers find everything from prescription medications to primary care services through improved discovery pathways.
Netflix
Netflix developed FM-Intent, a novel recommendation model that enhances their existing foundation model by incorporating hierarchical multi-task learning to predict user session intent alongside next-item recommendations. The problem addressed was that while their foundation model successfully predicted what users might watch next, it lacked understanding of underlying user intents (such as discovering new content versus continuing existing viewing, genre preferences, and content type preferences). FM-Intent solves this by establishing a hierarchical relationship where intent predictions inform item recommendations, using Transformer encoders to process interaction metadata and attention-based aggregation to combine multiple intent signals. The solution demonstrated a statistically significant 7.4% improvement in next-item prediction accuracy compared to the previous state-of-the-art baseline (TransAct) in offline experiments, and has been successfully integrated into Netflix's production recommendation ecosystem for applications including personalized UI optimization, analytics, and enhanced recommendation signals.
Rio Tinto
Rio Tinto Aluminium faced challenges in providing technical experts in refining and smelting sectors with quick and accurate access to vast amounts of specialized institutional knowledge during their internal training programs. They developed a generative AI-powered knowledge assistant using hybrid RAG (retrieval augmented generation) on Amazon Bedrock, combining both vector search and knowledge graph databases to enable more accurate, contextually rich responses. The hybrid system significantly outperformed traditional vector-only RAG across all metrics, particularly in context quality and entity recall, showing over 53% reduction in standard deviation while maintaining high mean scores, and leveraging 11-17 technical documents per query compared to 2-3 for vector-only approaches, ultimately streamlining how employees find and utilize critical business information.
GitHub
GitHub's machine learning team worked to enhance GitHub Copilot's contextual understanding of code to provide more relevant AI-powered coding suggestions. The problem was that large language models could only process limited context (approximately 6,000 characters), making it challenging to leverage all relevant information from a developer's codebase. The solution involved sophisticated prompt engineering, implementing neighboring tabs to process multiple open files, introducing a Fill-In-the-Middle (FIM) paradigm to consider code both before and after the cursor, and experimenting with vector databases and embeddings for semantic code retrieval. These improvements resulted in measurable gains: neighboring tabs provided a 5% relative increase in suggestion acceptance, FIM yielded a 10% relative boost in performance, and the overall enhancements contributed to developers coding up to 55% faster when using GitHub Copilot.
Various
This panel discussion brings together infrastructure experts from Groq, NVIDIA, Lambda, and AMD to discuss the unique challenges of deploying AI agents in production. The panelists explore how agentic AI differs from traditional AI workloads, requiring significantly higher token generation, lower latency, and more diverse infrastructure spanning edge to cloud. They discuss the evolution from training-focused to inference-focused infrastructure, emphasizing the need for efficiency at scale, specialized hardware optimization, and the importance of smaller distilled models over large monolithic models. The discussion highlights critical operational challenges including power delivery, thermal management, and the need for full-stack engineering approaches to debug and optimize agentic systems in production environments.
Netflix
Netflix developed a centralized foundation model for personalization to replace multiple specialized models powering their homepage recommendations. Rather than maintaining numerous individual models, they created one powerful transformer-based model trained on comprehensive user interaction histories and content data at scale. The challenge then became how to effectively integrate this large foundation model into existing production systems. Netflix experimented with and deployed three distinct integration approaches—embeddings via an Embedding Store, using the model as a subgraph within downstream models, and direct fine-tuning for specific applications—each with different tradeoffs in terms of latency, computational cost, freshness, and implementation complexity. These approaches are now used in production across different Netflix personalization use cases based on their specific requirements.
Ericsson
Ericsson's System Comprehension Lab is exploring the integration of symbolic reasoning capabilities into telecom-oriented large language models to address critical limitations in current LLM architectures for telecommunications infrastructure management. The problem centers on LLMs' inability to provide deterministic, explainable reasoning required for telecom network optimization, security, and anomaly detection—domains where hallucinations, lack of logical consistency, and black-box behavior are unacceptable. The proposed solution involves hybrid neural-symbolic AI architectures that combine the pattern recognition strengths of transformer-based LLMs with rule-based reasoning engines, connected through techniques like symbolic chain-of-thought prompting, program-aided reasoning, and external solver integration. This approach aims to enable AI-native wireless systems for 6G infrastructure that can perform cross-layer optimization, real-time decision-making, and intent-driven network management while maintaining the explainability and logical rigor demanded by production telecom environments.
Taralli
Taralli, a calorie tracking application, demonstrates systematic LLM improvement through rigorous evaluation and prompt optimization. The developer addressed the challenge of accurate nutritional estimation by creating a 107-example evaluation dataset, testing multiple prompt optimization techniques (vanilla, few-shot bootstrapping, MIPROv2, and GEPA) across several models (Gemini 2.5 Flash, Gemini 3 Flash, and DeepSeek v3.2). Through this methodical approach, they achieved a 15% accuracy improvement by switching from Gemini 2.5 Flash to Gemini 3 Flash while using a few-shot learning approach with 16 examples, reaching 60% accuracy within a 10% calorie prediction threshold. The system was deployed with fallback model configurations and extended to support fully offline on-device inference for iOS.
LinkedIn developed JUDE (Job Understanding Data Expert), a production platform that leverages fine-tuned large language models to generate high-quality embeddings for job recommendations at scale. The system addresses the computational challenges of LLM deployment through a multi-component architecture including fine-tuned representation learning, real-time embedding generation, and comprehensive serving infrastructure. JUDE replaced standardized features in job recommendation models, resulting in +2.07% qualified applications, -5.13% dismiss-to-apply ratio, and +1.91% total job applications - representing the highest metric improvement from a single model change observed by the team.
LinkedIn's customer service team faced challenges with retrieving relevant past issue tickets to resolve customer inquiries efficiently. Traditional text-based retrieval-augmented generation (RAG) approaches treated historical tickets as plain text, losing crucial structural information and inter-issue relationships. LinkedIn developed a novel system that integrates RAG with knowledge graphs, constructing tree-structured representations of issue tickets while maintaining explicit and implicit connections between issues. The system uses GPT-4 for parsing and answer generation, E5 embeddings for semantic retrieval, and converts user queries into graph database queries for precise subgraph extraction. Deployed across multiple product lines, the system achieved a 77.6% improvement in MRR, a 0.32 increase in BLEU score, and reduced median issue resolution time by 28.6% over six months of production use.
LinkedIn developed a large foundation model called "Brew XL" with 150 billion parameters to unify all personalization and recommendation tasks across their platform, addressing the limitations of task-specific models that operate in silos. The solution involved training a massive language model on user interaction data through "promptification" techniques, then distilling it down to smaller, production-ready models (3B parameters) that could serve high-QPS recommendation systems with sub-second latency. The system demonstrated zero-shot capabilities for new tasks, improved performance on cold-start users, and achieved 7x latency reduction with 30x throughput improvement through optimization techniques including distillation, pruning, quantization, and sparsification.
Pinterest's search relevance team integrated large language models into their search pipeline to improve semantic relevance prediction for over 6 billion monthly searches across 45 languages and 100+ countries. They developed a cross-encoder teacher model using fine-tuned open-source LLMs that achieved 12-20% performance improvements over existing models, then used knowledge distillation to create a production-ready bi-encoder student model that could scale efficiently. The solution incorporated visual language model captions, user engagement signals, and multilingual capabilities, ultimately improving search relevance metrics internationally while producing reusable semantic embeddings for other Pinterest surfaces.
Pinterest tackled the challenge of improving search relevance by implementing a large language model-based system. They developed a cross-encoder LLM teacher model trained on human-annotated data, which was then distilled into a lightweight student model for production deployment. The system processes rich Pin metadata including titles, descriptions, and synthetic image captions to predict relevance scores. The implementation resulted in a 2.18% improvement in search feed relevance (nDCG@20) and over 1.5% increase in search fulfillment rates globally, while successfully generalizing across multiple languages despite being trained primarily on US data.
Google / YouTube
YouTube developed Large Recommender Models (LRM) by adapting Google's Gemini LLM for video recommendations, addressing the challenge of serving personalized content to billions of users. The solution involved creating semantic IDs to tokenize videos, continuous pre-training to teach the model both English and YouTube-specific video language, and implementing generative retrieval systems. While the approach delivered significant improvements in recommendation quality, particularly for challenging cases like new users and fresh content, the team faced substantial serving cost challenges that required 95%+ cost reductions and offline inference strategies to make production deployment viable at YouTube's scale.
Apple
Apple developed and deployed a comprehensive foundation model infrastructure consisting of a 3-billion parameter on-device model and a mixture-of-experts server model to power Apple Intelligence features across iOS, iPadOS, and macOS. The implementation addresses the challenge of delivering generative AI capabilities at consumer scale while maintaining privacy, efficiency, and quality across 15 languages. The solution involved novel architectural innovations including shared KV caches, parallel track mixture-of-experts design, and extensive optimization techniques including quantization and compression, resulting in production deployment across millions of devices with measurable performance improvements in text and vision tasks.
AWS GENAIC (Japan)
Japan's GENIAC program partnered with AWS to provide 12 organizations with massive compute resources (127 P5 instances and 24 Trn1 instances) for foundation model development. The challenge revealed that successful FM training required far more than raw hardware access - it demanded structured organizational support, reference architectures, cross-functional teams, and comprehensive enablement programs. Through systematic deployment guides, monitoring infrastructure, and dedicated communication channels, multiple large-scale models were successfully trained including 100B+ parameter models, demonstrating that large-scale AI development is fundamentally an organizational rather than purely technical challenge.
Exa.ai
Exa.ai built a sophisticated GPU infrastructure combining a new 144 H200 GPU cluster with their existing 80 A100 GPU cluster to support their neural web search and retrieval models. They implemented a five-layer infrastructure stack using Pulumi, Ansible/Kubespray, NVIDIA operators, Alluxio for storage, and Flyte for orchestration, enabling efficient large-scale model training and inference while maintaining reproducibility and reliability.
Coupang
Coupang, a major e-commerce platform operating primarily in South Korea and Taiwan, faced challenges in scaling their ML infrastructure to support LLM applications across search, ads, catalog management, and recommendations. The company addressed GPU supply shortages and infrastructure limitations by building a hybrid multi-region architecture combining cloud and on-premises clusters, implementing model parallel training with DeepSpeed, and establishing GPU-based serving using Nvidia Triton and vLLM. This infrastructure enabled production applications including multilingual product understanding, weak label generation at scale, and unified product categorization, with teams using patterns ranging from in-context learning to supervised fine-tuning and continued pre-training depending on resource constraints and quality requirements.
DoorDash
DoorDash faced challenges in scaling personalization and maintaining product catalogs as they expanded beyond restaurants into new verticals like grocery, retail, and convenience stores, dealing with millions of SKUs and cold-start scenarios for new customers and products. They implemented a layered approach combining traditional machine learning with fine-tuned LLMs, RAG systems, and LLM agents to automate product knowledge graph construction, enable contextual personalization, and provide recommendations even without historical user interaction data. The solution resulted in faster, more cost-effective catalog processing, improved personalization for cold-start scenarios, and the foundation for future agentic shopping experiences that can adapt to real-time contexts like emergency situations.
Uber
Uber Eats built a production-grade semantic search platform to improve discovery across restaurants, grocery, and retail items by addressing limitations of traditional lexical search. The solution leverages LLM-based embeddings (using Qwen as the backbone), a two-tower architecture with Matryoshka Representation Learning, and Apache Lucene Plus for indexing. Through careful optimization of ANN parameters, quantization strategies, and embedding dimensions, the team achieved significant cost reductions (34% latency reduction, 17% CPU savings, 50% storage reduction) while maintaining high recall (>0.95). The system features automated biweekly model updates with blue/green deployment, comprehensive validation gates, and serving-time reliability checks to ensure production stability at global scale.
ByteDance
ByteDance implemented multimodal LLMs for video understanding at massive scale, processing billions of videos daily for content moderation and understanding. By deploying their models on AWS Inferentia2 chips across multiple regions, they achieved 50% cost reduction compared to standard EC2 instances while maintaining high performance. The solution combined tensor parallelism, static batching, and model quantization techniques to optimize throughput and latency.
Airbnb
Airbnb implemented AI text generation models across three key customer support areas: content recommendation, real-time agent assistance, and chatbot paraphrasing. They leveraged large language models with prompt engineering to encode domain knowledge from historical support data, resulting in significant improvements in content relevance, agent efficiency, and user engagement. The implementation included innovative approaches to data preparation, model training with DeepSpeed, and careful prompt design to overcome common challenges like generic responses.
Zalando
Zalando's Partner Tech team faced significant challenges maintaining two distinct in-house UI component libraries across 15 B2B applications, leading to inconsistent user experiences, duplicated efforts, and increased maintenance complexity. To address this technical debt, they explored using Large Language Models (LLMs) to automate the migration from one library to another. Through an iterative experimentation process involving five iterations of prompt engineering, they developed a Python-based migration tool using GPT-4o that achieved over 90% accuracy in component transformations. The solution proved highly cost-effective at under $40 per repository and significantly reduced manual migration effort, though it still required human oversight for visual verification and handling of complex edge cases.
Meta
Meta developed the Automated Compliance Hardening (ACH) tool to address the challenge of scaling compliance adherence across its products while maintaining developer velocity. Traditional compliance processes relied on manual, error-prone approaches that couldn't keep pace with rapid technology development. By leveraging LLMs for mutation-guided test generation, ACH generates realistic, problem-specific mutants (deliberately introduced faults) and automatically creates tests to catch them through plain-text prompts. During a trial from October to December 2024 across Facebook, Instagram, WhatsApp, and Meta's wearables platforms, privacy engineers accepted 73% of generated tests, with 36% judged as privacy-relevant. The system overcomes traditional barriers to mutation testing deployment including scalability issues, unrealistic mutants, equivalent mutants, computational costs, and testing overstretch.
Pinterest Search faced significant limitations in measuring search relevance due to the high cost and low availability of human annotations, which resulted in large minimum detectable effects (MDEs) that could only identify significant topline metric movements. To address this, they fine-tuned open-source multilingual LLMs on human-annotated data to predict relevance scores on a 5-level scale, then deployed these models to evaluate ranking results across A/B experiments. This approach reduced labeling costs dramatically, enabled stratified query sampling designs, and achieved an order of magnitude reduction in MDEs (from 1.3-1.5% down to ≤0.25%), while maintaining strong alignment with human labels (73.7% exact match, 91.7% within 1 point deviation) and enabling rapid evaluation of 150,000 rows within 30 minutes on a single GPU.
eBay
eBay developed Mercury, an internal agentic framework designed to scale LLM-powered recommendation experiences across its massive marketplace of over two billion active listings. The platform addresses the challenge of transforming vast amounts of unstructured data into personalized product recommendations by integrating Retrieval-Augmented Generation (RAG) with a custom Listing Matching Engine that bridges the gap between LLM-generated text outputs and eBay's dynamic inventory. Mercury enables rapid development through reusable, plug-and-play components following object-oriented design principles, while its near-real-time distributed queue-based execution platform handles cost and latency requirements at industrial scale. The system combines multiple retrieval mechanisms, semantic search using embedding models, anomaly detection, and personalized ranking to deliver contextually relevant shopping experiences to hundreds of millions of users.
Meta
Meta addresses the critical challenge of hardware reliability in large-scale AI infrastructure, where hardware faults significantly impact training and inference workloads. The company developed comprehensive detection mechanisms including Fleetscanner, Ripple, and Hardware Sentinel to identify silent data corruptions (SDCs) that can cause training divergence and inference errors without obvious symptoms. Their multi-layered approach combines infrastructure strategies like reductive triage and hyper-checkpointing with stack-level solutions such as gradient clipping and algorithmic fault tolerance, achieving industry-leading reliability for AI operations across thousands of accelerators and globally distributed data centers.
Baseten
Baseten has built a production-grade LLM inference platform focusing on three key pillars: model-level performance optimization, horizontal scaling across regions and clouds, and enabling complex multi-model workflows. The platform supports various frameworks including SGLang and TensorRT-LLM, and has been successfully deployed by foundation model companies and enterprises requiring strict latency, compliance, and reliability requirements. A key differentiator is their ability to handle mission-critical inference workloads with sub-400ms latency for complex use cases like AI phone calls.
Atlassian
Atlassian developed a machine learning-based comment ranker to improve the quality of their LLM-powered code review agent by filtering out noisy, incorrect, or unhelpful comments. The system uses a fine-tuned ModernBERT model trained on proprietary data from over 53K code review comments to predict which LLM-generated comments will lead to actual code changes. The solution improved code resolution rates from ~33% to 40-45%, approaching human reviewer performance of 45%, while maintaining robustness across different underlying LLMs and user bases, ultimately reducing PR cycle times by 30% and serving over 10K monthly active users reviewing 43K+ pull requests.
Totogi
Totogi, an AI company serving the telecommunications industry, faced challenges with traditional Business Support Systems (BSS) that required lengthy change request processing—typically taking 7 days and involving costly, specialized engineering talent. To address this, Totogi developed BSS Magic, which combines a comprehensive telco ontology with a multi-agent AI framework powered by Anthropic Claude models on Amazon Bedrock. The solution orchestrates five specialized AI agents (Business Analyst, Technical Architect, Developer, QA, and Tester) through AWS Step Functions and Lambda, automating the entire software development lifecycle from requirements analysis to code generation and testing. In collaboration with the AWS Generative AI Innovation Center, Totogi achieved significant results: reducing change request processing time from 7 days to a few hours, achieving 76% code coverage in automated testing, and delivering production-ready telecom-grade code with minimal human intervention.
Meta
This case study presents a sophisticated multi-agent LLM system designed to identify, correct, and find the root causes of misinformation on social media platforms at scale. The solution addresses the limitations of pre-LLM era approaches (content-only features, no real-time information, low precision/recall) by deploying specialized agents including an Indexer (for sourcing authentic data), Extractor (adaptive retrieval and reranking), Classifier (discriminative misinformation categorization), Corrector (reasoning and correction generation), and Verifier (final validation). The system achieves high precision and recall by orchestrating these agents through a centralized coordinator, implementing comprehensive logging, evaluation at both individual agent and system levels, and optimization strategies including model distillation, semantic caching, and adaptive retrieval. The approach prioritizes accuracy over cost and latency given the high stakes of misinformation propagation on platforms.
Various (Thinking Machines, Yutori, Evolutionaryscale, Perplexity, Axiom)
This panel discussion features experts from multiple AI companies discussing the current state and future of agentic frameworks, reinforcement learning applications, and production LLM deployment challenges. The panelists from Thinking Machines, Perplexity, Evolutionary Scale AI, and Axiom share insights on framework proliferation, the role of RL in post-training, domain-specific applications in mathematics and biology, and infrastructure bottlenecks when scaling models to hundreds of GPUs, highlighting the gap between research capabilities and production deployment tools.
Meta / AWS / NVIDIA / ConverseNow
This panel discussion features leaders from Meta, AWS, NVIDIA, and ConverseNow discussing real-world challenges and solutions for deploying LLMs in production environments. The conversation covers the trade-offs between small and large language models, with ConverseNow sharing their experience building voice AI systems for restaurants that require high accuracy and low latency. Key themes include the importance of fine-tuning small models for production use cases, the convergence of training and inference systems, optimization techniques like quantization and alternative architectures, and the challenges of building reliable, cost-effective inference stacks for mission-critical applications.
AMD / Somite AI / Upstage / Rambler AI
This panel discussion at AWS re:Invent features three companies deploying AI models in production across different industries: Somite AI using machine learning for computational biology and cellular control, Upstage developing sovereign AI with proprietary LLMs and OCR for document extraction in enterprises, and Rambler AI building vision language models for industrial task verification. All three leverage AMD GPU infrastructure (MI300 series) for training and inference, emphasizing the importance of hardware choice, open ecosystems, seamless deployment, and cost-effective scaling. The discussion highlights how smaller, domain-specific models can achieve enterprise ROI where massive frontier models failed, and explores emerging areas like physical AI, world models, and data collection for robotics.
Upwork
Upwork, a global freelance talent marketplace, developed Uma (Upwork's Mindful AI) to streamline the hiring and matching processes between clients and freelancers. The company faced the challenge of serving a large, diverse customer base with AI solutions that needed both broad applicability and precision for specific marketplace use cases like discovery, search, and matching. Their solution involved a dual approach: leveraging pretrained models like GPT-4 for rapid deployment of features such as job post generation and chat assistance, while simultaneously developing custom, use case-specific smaller language models fine-tuned on proprietary platform data, synthetic data, and human-generated content from talented writers. This strategy resulted in significant improvements, including an 80% reduction in job post creation time and more accurate, contextually relevant assistance for both freelancers and clients across the platform.
Grammarly
Grammarly's Strategic Research team developed mEdIT, a multilingual extension of their CoEdIT text editing model, to support intelligent writing assistance across seven languages and three editing tasks (grammatical error correction, text simplification, and paraphrasing). The problem addressed was that foundational LLMs produce low-quality outputs for text editing tasks, and prior specialized models only supported either multiple tasks in one language or single tasks across multiple languages. By fine-tuning multilingual LLMs (including mT5, mT0, BLOOMZ, PolyLM, and Bactrian-X) on over 200,000 carefully curated instruction-output pairs across Arabic, Chinese, English, German, Japanese, Korean, and Spanish, mEdIT achieved strong performance across tasks and languages, even when instructions were given in a different language than the text being edited. The models demonstrated generalization to unseen languages, with causal language models performing best, and received high ratings from human evaluators, though the work has not yet been integrated into Grammarly's production systems.
Infosys
Infosys developed an advanced multimodal Retrieval-Augmented Generation (RAG) solution using Amazon Bedrock to process complex oil and gas drilling documentation containing text, images, charts, and technical diagrams. The solution addresses the challenge of extracting insights from thousands of technical documents including well completion reports, drilling logs, and lithology diagrams that traditional document processing methods struggle to handle effectively. Through iterative development exploring various chunking strategies, embedding models, and search approaches, the team ultimately implemented a hybrid search system with parent-child chunking hierarchy, achieving 92% retrieval accuracy, sub-2-second response times, and delivering significant operational efficiency gains including 40-50% reduction in manual document processing costs and 60% time savings for field engineers and geologists.
Google Research developed an on-device grammar correction system for Gboard on Pixel 6 that detects and suggests corrections for grammatical errors as users type. The solution addresses the challenge of implementing neural grammar correction within the constraints of mobile devices (limited memory, computational power, and latency requirements) while preserving user privacy by keeping all processing local. The team built a 20MB hybrid Transformer-LSTM model using hard distillation from a cloud-based system, achieving inference on 60 characters in under 22ms on the Pixel 6 CPU, enabling real-time grammar correction for both complete sentences and partial sentence prefixes across English text in nearly any app using Gboard.
Grammarly
Grammarly developed a compact 1B-parameter on-device LLM to provide offline spelling and grammar correction capabilities, addressing the challenge of maintaining writing assistance functionality without internet connectivity. The team selected Llama as the base model, created comprehensive synthetic training data covering diverse writing styles and error types, and applied extensive optimizations including Grouped Query Attention, MLX framework integration for Apple silicon, and 4-bit quantization. The resulting model achieves 210 tokens/second on M2 Mac hardware while maintaining correction quality, demonstrating that multiple specialized models can be consolidated into a single efficient on-device solution that preserves user voice and delivers real-time feedback.
Cursor
Cursor developed a production LLM system called Cursor Tab that predicts developer actions and suggests code completions across codebases, handling over 400 million requests per day. To address the challenge of noisy suggestions that disrupt developer flow, they implemented an online reinforcement learning approach using policy gradient methods that directly optimizes the model to show suggestions only when acceptance probability exceeds a target threshold. This approach required building infrastructure for rapid model deployment and on-policy data collection with a 1.5-2 hour turnaround cycle. The resulting model achieved a 21% reduction in suggestions shown while simultaneously increasing the accept rate by 28%, demonstrating effective LLMOps practices for continuously improving production models using real-time user feedback.
Meta
Meta released Code Llama, a family of specialized large language models for code generation built on top of Llama 2, aiming to assist developers with coding tasks and lower barriers to entry for new programmers. The solution includes multiple model sizes (7B, 13B, 34B, and 70B parameters) with three variants: a foundational code model, a Python-specialized version, and an instruction-tuned variant, all trained on 500B-1T tokens of code and supporting up to 100,000 token contexts. Benchmark testing showed Code Llama 34B achieved 53.7% on HumanEval and 56.2% on MBPP, matching ChatGPT performance while being released under an open license for both research and commercial use, with extensive safety evaluations and red teaming conducted to address responsible AI concerns.
Boltz
Boltz, founded by Gabriele Corso and Jeremy Wohlwend, developed an open-source suite of AI models (Boltz-1, Boltz-2, and BoltzGen) for structural biology and protein design, democratizing access to capabilities previously held by proprietary systems like AlphaFold 3. The company addresses the challenge of predicting complex molecular interactions (protein-ligand, protein-protein) and designing novel therapeutic proteins by combining generative diffusion models with specialized equivariant architectures. Their approach achieved validated nanomolar binders for two-thirds of nine previously unseen protein targets, demonstrating genuine generalization beyond training data. The newly launched Boltz Lab platform provides a production-ready infrastructure with optimized GPU kernels running 10x faster than open-source versions, offering agents for protein and small molecule design with collaborative interfaces for medicinal chemists and researchers.
Moveworks
Moveworks addressed latency challenges in their enterprise Copilot by implementing NVIDIA's TensorRT-LLM optimization engine. The integration resulted in significant performance improvements, including a 2.3x increase in token processing speed (from 19 to 44 tokens per second), a reduction in average request latency from 3.4 to 1.5 seconds, and nearly 3x faster time to first token. These optimizations enabled more natural conversations and improved resource utilization in production.
Rolls-Royce
Rolls-Royce collaborated with Databricks to enhance their design space exploration capabilities using conditional Generative Adversarial Networks (cGANs). The project aimed to leverage legacy simulation data to identify and assess innovative design concepts without requiring traditional geometry modeling and simulation processes. By implementing cGANs on the Databricks platform, they successfully developed a system that could handle multi-objective constraints and optimize design processes while maintaining compliance with aerospace industry requirements.
LinkedIn developed Liger-Kernel, a library to optimize GPU performance during LLM training by addressing memory access and per-operation bottlenecks. Using techniques like FlashAttention and operator fusion implemented in Triton, the library achieved a 60% reduction in memory usage, 20% improvement in multi-GPU training throughput, and a 3x reduction in end-to-end training time.
Replit
Replit faced challenges with running LLM inference on expensive GPU infrastructure and implemented a solution using preemptable cloud GPUs to reduce costs by two-thirds. The key challenge was reducing server startup time from 18 minutes to under 2 minutes to handle preemption events, which they achieved through container optimization, GKE image streaming, and improved model loading processes.
LinkedIn developed and open-sourced LIER (LinkedIn Efficient and Reusable) kernels to address the fundamental challenge of memory consumption in LLM training. By optimizing core operations like layer normalization, rotary position encoding, and activation functions, they achieved up to 3-4x reduction in memory allocation and 20% throughput improvements for large models. The solution, implemented using Python and Triton, focuses on minimizing data movement between GPU memory and compute units, making LLM training faster and more cost-effective.
LinkedIn introduced Liger-Kernel, an open-source library addressing GPU efficiency challenges in LLM training. The solution combines efficient Triton kernels with a flexible API design, integrated into a comprehensive training infrastructure stack. The implementation achieved significant improvements, including 20% better training throughput and 60% reduced memory usage for popular models like Llama, Gemma, and Qwen, while maintaining compatibility with mainstream training frameworks and distributed training systems.
IDIADA
IDIADA developed AIDA, an intelligent chatbot powered by Amazon Bedrock, to assist their workforce with various tasks. To optimize performance, they implemented specialized classification pipelines using different approaches including LLMs, k-NN, SVM, and ANN with embeddings from Amazon Titan and Cohere models. The optimized system achieved 95% accuracy in request routing and drove a 20% increase in team productivity, handling over 1,000 interactions daily.
Snowflake
Snowflake faced performance bottlenecks when scaling embedding models for their Cortex AI platform, which processes trillions of tokens monthly. Through profiling vLLM, they identified CPU-bound inefficiencies in tokenization and serialization that left GPUs underutilized. They implemented three key optimizations: encoding embedding vectors as little-endian bytes for faster serialization, disaggregating tokenization and inference into a pipeline, and running multiple model replicas on single GPUs. These improvements delivered 16x throughput gains for short sequences and 4.2x for long sequences, while reducing costs by 16x and achieving 3x throughput improvement in production.
OpenAI
This case study explores OpenAI's approach to post-training and deploying large language models in production environments, featuring insights from a post-training researcher working on reasoning models. The discussion covers the operational complexities of reinforcement learning from human feedback at scale, the evolution from non-thinking to thinking models, and production challenges including model routing, context window optimization, token efficiency improvements, and interruptability features. Key developments include the shopping model release, improvements from GPT-4.1 to GPT-5.1, and the operational realities of managing complex RL training runs with multiple grading setups and infrastructure components that require constant monitoring and debugging.
Bonnier News
Bonnier News, a major Swedish media publisher with over 200 brands including Expressen and local newspapers, has deployed AI and machine learning systems in production to solve content personalization and newsroom automation challenges. The company's data science team, led by product manager Hans Yell (PhD in computational linguistics) and head of architecture Magnus Engster, has built white-label personalization engines using embedding-based recommendation systems that outperform manual content curation while scaling across multiple brands. They leverage vector similarity and user reading patterns rather than traditional metadata, achieving significant engagement lifts. Additionally, they're developing LLM-powered tools for journalists including headline generation, news aggregation summaries, and trigger questions for articles. Through a WASP-funded PhD collaboration, they're working on domain-adapted Swedish language models via continued pre-training of Llama models with Bonnier's extensive text corpus, focusing on capturing brand tone and improving journalistic workflows while maintaining data sovereignty.
Tinder
Tinder implemented two production GenAI applications to enhance user safety and experience: a username detection system using fine-tuned Mistral 7B to identify social media handles in user bios with near-perfect recall, and a personalized match explanation feature using fine-tuned Llama 3.1 8B to help users understand why recommended profiles are relevant. Both systems required sophisticated LLMOps infrastructure including multi-model serving with LoRA adapters, GPU optimization, extensive monitoring, and iterative fine-tuning processes to achieve production-ready performance at scale.
jonfernandes
Independent AI engineer Jonathan Fernandez shares his experience developing a production-ready RAG (Retrieval Augmented Generation) stack through 37 failed iterations, focusing on building solutions for financial institutions. The case study demonstrates the evolution from a naive RAG implementation to a sophisticated system incorporating query processing, reranking, and monitoring components. The final architecture uses LlamaIndex for orchestration, Qdrant for vector storage, open-source embedding models, and Docker containerization for on-premises deployment, achieving significantly improved response quality for document-based question answering.
Superlinked
SuperLinked, a company focused on vector search infrastructure, shares production insights from deploying information retrieval systems for e-commerce and enterprise knowledge management with indexes up to 2 terabytes. The presentation addresses challenges in relevance, latency, and cost optimization when deploying vector search systems at scale. Key solutions include avoiding vector pooling/averaging, implementing late interaction models, fine-tuning embeddings for domain-specific needs, combining sparse and dense representations, leveraging graph embeddings, and using template-based query generation instead of unconstrained text-to-SQL. Results demonstrate 5%+ precision improvements through targeted fine-tuning, significant latency reductions through proper database selection and query optimization, and improved relevance through multi-encoder architectures that combine text, graph, and metadata signals.
Reducto
Reducto has built a production document parsing system that processes over 1 billion documents by combining specialized vision-language models, traditional OCR, and layout detection models in a hybrid pipeline. The system addresses critical challenges in document parsing including hallucinations from frontier models, dense tables, handwritten forms, and complex charts. Their approach uses a divide-and-conquer strategy where different models are routed to different document regions based on complexity, achieving higher accuracy than AWS Textract, Microsoft Azure Document Intelligence, and Google Cloud OCR on their internal benchmarks. The company has expanded beyond parsing to offer extraction with pixel-level citations and an edit endpoint for automated form filling.
Playtika
Playtika, a gaming company, built an internal generative AI platform to accelerate art production for their game studios with the goal of reducing art production time by 50%. The solution involved creating a comprehensive infrastructure for fine-tuning and deploying diffusion models (Stable Diffusion 1.5, then SDXL) at scale, supporting text-to-image, image-to-image, and inpainting capabilities. The platform evolved from using DreamBooth fine-tuning with separate model deployments to LoRA adapters with SDXL, enabling efficient model switching and GPU utilization. Through optimization techniques including OneFlow acceleration framework (achieving 40% latency reduction), FP16 quantization, NVIDIA MIG partitioning, and careful infrastructure design, they built a cost-efficient system serving multiple game studios while maintaining quality and minimizing inference latency.
Thomson Reuters
Thomson Reuters implemented a Retrieval-Augmented Generation (RAG) system to enhance customer support for their legal and tax domain products. The challenge involved customer support agents experiencing cognitive overload while navigating hundreds of thousands of knowledge base articles across complex product lines like Westlaw, Practical Law, and Checkpoint. By building a RAG architecture combining dense retrieval systems (using Milvus vector database and sentence transformers) with GPT-4, Thomson Reuters created a conversational interface that provides agents with relevant, accurate solutions from their curated knowledge base. The solution reduced resolution times and improved the accuracy of support responses by grounding GPT-4's outputs in company-specific documentation, avoiding hallucinations common in standalone LLM deployments.
University of California Los Angeles
The University of California Los Angeles (UCLA) Office of Advanced Research Computing (OARC) partnered with UCLA's Center for Research and Engineering in Media and Performance (REMAP) to build an AI-powered system for an immersive production of the musical "Xanadu." The system enabled up to 80 concurrent audience members and performers to create sketches on mobile phones, which were processed in near real-time (under 2 minutes) through AWS generative AI services to produce 2D images and 3D meshes displayed on large LED screens during live performances. Using a serverless-first architecture with Amazon SageMaker AI endpoints, Amazon Bedrock foundation models, and AWS Lambda orchestration, the system successfully supported 7 performances in May 2025 with approximately 500 total audience members, demonstrating that cloud-based generative AI can reliably power interactive live entertainment experiences.
Roblox
Roblox deployed a unified transformer-based translation LLM to enable real-time chat translation across all combinations of 16 supported languages for over 70 million daily active users. The company built a custom ~1 billion parameter model using pretraining on open source and proprietary data, then distilled it down to fewer than 650 million parameters to achieve approximately 100 millisecond latency while handling over 5,000 chats per second. The solution leverages a mixture-of-experts architecture, custom translation quality estimation models, back translation techniques for low-resource language pairs, and comprehensive integration with trust and safety systems to deliver contextually appropriate translations that understand Roblox-specific slang and terminology.
Capital One
Capital One developed enhanced input guardrails to protect LLM-powered conversational assistants from adversarial attacks and malicious inputs. The company used chain-of-thought prompting combined with supervised fine-tuning (SFT) and alignment techniques like Direct Preference Optimization (DPO) and Kahneman-Tversky Optimization (KTO) to improve the accuracy of LLM-as-a-Judge moderation systems. Testing on four open-source models (Mistral 7B, Mixtral 8x7B, Llama2 13B, and Llama3 8B) showed significant improvements in F1 scores and attack detection rates of over 50%, while maintaining low false positive rates, demonstrating that effective guardrails can be achieved with small training datasets and minimal computational resources.
Cursor
This case study examines Cursor's implementation of reinforcement learning (RL) for training coding models and agents in production environments. The team discusses the unique challenges of applying RL to code generation compared to other domains like mathematics, including handling larger action spaces, multi-step tool calling processes, and developing reward signals that capture real-world usage patterns. They explore various technical approaches including test-based rewards, process reward models, and infrastructure optimizations for handling long context windows and high-throughput inference during RL training, while working toward more human-centric evaluation metrics beyond traditional test coverage.
Instacart
Instacart transformed their query understanding (QU) system from multiple independent traditional ML models to a unified LLM-based approach to better handle long-tail, specific, and creatively-phrased search queries. The solution employed a layered strategy combining retrieval-augmented generation (RAG) for context engineering, post-processing guardrails, and fine-tuning of smaller models (Llama-3-8B) on proprietary data. The production system achieved significant improvements including 95%+ query rewrite coverage with 90%+ precision, 6% reduction in scroll depth for tail queries, 50% reduction in complaints for poor tail query results, and sub-300ms latency through optimizations like adapter merging, H100 GPU upgrades, and autoscaling.
Square
Square developed and deployed a RoBERTa-based merchant classification system to accurately categorize millions of merchants across their platform. The system replaced unreliable self-selection methods with an ML approach that combines business names, self-selected information, and transaction data to achieve a 30% improvement in accuracy. The solution runs daily predictions at scale using distributed GPU infrastructure and has become central to Square's business metrics and strategic decision-making.
Meta
Meta developed and deployed an AI-powered image animation feature that needed to serve billions of users efficiently. They tackled this challenge through a comprehensive optimization strategy including floating-point precision reduction, temporal-attention improvements, DPM-Solver implementation, and innovative distillation techniques. The system was further enhanced with sophisticated traffic management and load balancing solutions, resulting in a highly efficient, globally scalable service with minimal latency and failure rates.
Slack
Slack's Developer Experience team embarked on a multi-year journey to integrate generative AI into their internal development workflows, moving from experimental prototypes to production-grade AI assistants and agentic systems. Starting with Amazon SageMaker for initial experimentation, they transitioned to Amazon Bedrock for simplified infrastructure management, achieving a 98% cost reduction. The team rolled out AI coding assistants using Anthropic's Claude Code and Cursor integrated with Bedrock, resulting in 99% developer adoption and a 25% increase in pull request throughput. They then evolved their internal knowledge bot (Buddybot) into a sophisticated multi-agent system handling over 5,000 escalation requests monthly, using AWS Strands as an orchestration framework with Claude Code sub-agents, Temporal for workflow durability, and MCP servers for standardized tool access. The implementation demonstrates a pragmatic approach to LLMOps, prioritizing incremental deployment, security compliance (FedRAMP), observability through OpenTelemetry, and maintaining model agnosticism while scaling to millions of tokens per minute.
Meta
Meta tackled the challenge of deploying an AI-powered image animation feature at massive scale, requiring optimization of both model performance and infrastructure. Through a combination of model optimizations including halving floating-point precision, improving temporal-attention expansion, and leveraging DPM-Solver, along with sophisticated traffic management and deployment strategies, they successfully deployed a system capable of serving billions of users while maintaining low latency and high reliability.
Articul8
Articul8, a generative AI company focused on domain-specific models (DSMs), faced challenges in training and deploying specialized LLMs across semiconductor, energy, and supply chain industries due to infrastructure complexity and computational requirements. They implemented Amazon SageMaker HyperPod to manage distributed training clusters with automated fault tolerance, achieving over 95% cluster utilization and 35% productivity improvements. The solution enabled them to reduce AI deployment time by 4x and total cost of ownership by 5x while successfully developing high-performing DSMs that outperform general-purpose LLMs by 2-3x in domain-specific tasks, with their A8-Semicon model achieving twice the accuracy of GPT-4o and Claude in Verilog code generation at 50-100x smaller model sizes.
Nubank
Nubank integrated foundation models into their AI platform to enhance predictive modeling across critical banking decisions, moving beyond traditional tabular machine learning approaches. Through their acquisition of Hyperplane in July 2024, they developed billion-parameter transformer models that process sequential transaction data to better understand customer behavior. Over eight months, they achieved significant performance improvements (1.20% average AUC lift across benchmark tasks) while maintaining existing data governance and model deployment infrastructure, successfully deploying these models to production decision engines serving over 100 million customers.
LinkedIn adopted vLLM, an open-source LLM inference framework, to power over 50 GenAI use cases including LinkedIn Hiring Assistant and AI Job Search, running on thousands of hosts across their platform. The company faced challenges in deploying LLMs at scale with low latency and high throughput requirements, particularly for applications requiring complex reasoning and structured outputs. By leveraging vLLM's PagedAttention technology and implementing a five-phase evolution strategy—from offline mode to a modular, OpenAI-compatible architecture—LinkedIn achieved significant performance improvements including ~10% TPS gains and GPU savings of over 60 units for certain workloads, while maintaining sub-600ms p95 latency for thousands of QPS in production applications.
Meta
Meta's AI infrastructure team developed a comprehensive LLM serving platform to support Meta AI, smart glasses, and internal ML workflows including RLHF processing hundreds of millions of examples. The team addressed the fundamental challenges of LLM inference through a four-stage approach: building efficient model runners with continuous batching and KV caching, optimizing hardware utilization through distributed inference techniques like tensor and pipeline parallelism, implementing production-grade features including disaggregated prefill/decode services and hierarchical caching systems, and scaling to handle multiple deployments with sophisticated allocation and cost optimization. The solution demonstrates the complexity of productionizing LLMs, requiring deep integration across modeling, systems, and product teams to achieve acceptable latency and cost efficiency at scale.
Meta
Meta faced the challenge of scaling their AI infrastructure from training smaller recommendation models to massive LLM training jobs like LLaMA 3. They built two 24K GPU clusters (one with RoCE, another with InfiniBand) to handle the unprecedented scale of computation required for training models with thousands of GPUs running for months. Through full-stack optimizations across hardware, networking, and software layers, they achieved 95% training efficiency for the LLaMA 3 70B model, while dealing with challenges in hardware reliability, thermal management, network topology, and collective communication operations.
DeepL
DeepL needed to scale their Language AI capabilities while maintaining low latency for production inference and handling increasing request volumes. The company transitioned from BFloat16 (BF16) to 8-bit floating point (FP8) precision for both training and inference of their large language models, leveraging NVIDIA H100 GPUs' native FP8 support through Transformer Engine for training and TensorRT-LLM for inference. This approach accelerated model training by 50% (achieving 67% Model FLOPS utilization), enabled training of larger models with more parameters, doubled inference throughput at equivalent latency levels, and delivered translation quality improvements of 1.4x for European languages and 1.7x for complex language pairs like English-Japanese, all while maintaining comparable training quality to BF16 precision.
LinkedIn faced significant performance challenges when deploying LLM-based ranking systems for AI Job Search and AI People Search, where models needed to score hundreds of items per query within strict latency SLAs (sub-500ms P99). The ranking workload differs fundamentally from text generation—it requires only the prefill phase to score candidates, not iterative token generation. LinkedIn optimized SGLang, an open-source LLM serving system, through four optimization stages: implementing comprehensive batching (tokenization and batch preservation), creating a scoring-only fast path that eliminates unnecessary decode loops and CPU-GPU synchronization, introducing in-batch prefix caching to reuse shared query context, and addressing Python runtime bottlenecks through multi-process architecture. These optimizations delivered 2-3x throughput improvements on H100 GPUs while maintaining P99 latency under 500ms, enabling production-scale LLM ranking for millions of members.
Meta
Meta launched Feed Deep Dive as an AI-powered feature on Facebook in April 2024 to address information-seeking and context enrichment needs when users encounter posts they want to learn more about. The challenge was scaling from launch to product-market fit while maintaining high-quality responses at Meta scale, dealing with LLM hallucinations and refusals, and providing more value than users would get from simply scrolling Facebook Feed. Meta's solution involved evolving from traditional orchestration to agentic models with planning, tool calling, and reflection capabilities; implementing auto-judges for online quality evaluation; using smart caching strategies focused on high-traffic posts; and leveraging ML-based user cohort targeting to show the feature to users who derived the most value. The results included achieving product-market fit through improved quality and engagement, with the team now moving toward monetization and expanded use cases.
Notion
Notion scaled their vector search infrastructure supporting Notion AI Q&A from launch in November 2023 through early 2026, achieving a 10x increase in capacity while reducing costs by 90%. The problem involved onboarding millions of workspaces to their AI-powered semantic search feature while managing rapidly growing infrastructure costs. Their solution involved migrating from dedicated pod-based vector databases to serverless architectures, switching to turbopuffer as their vector database provider, implementing intelligent page state caching to avoid redundant embeddings, and transitioning to Ray on Anyscale for both embeddings generation and serving. The results included clearing a multi-million workspace waitlist, reducing vector database costs by 60%, cutting embeddings infrastructure costs by over 90%, and improving query latency from 70-100ms to 50-70ms while supporting 15x growth in active workspaces.
Etsy
Etsy's Search Relevance team developed a comprehensive Semantic Relevance Evaluation and Enhancement Framework to address the limitations of engagement-based search models that favored popular listings over semantically relevant ones. The solution employs a three-tier cascaded distillation approach: starting with human-curated "golden" labels, scaling with an LLM annotator (o3 model) to generate training data, fine-tuning a teacher model (Qwen 3 VL 4B) for efficient large-scale evaluation, and distilling to a lightweight BERT-based student model for real-time production inference. The framework integrates semantic relevance signals into search through filtering, feature enrichment, loss weighting, and relevance boosting. Between August and October 2025, the percentage of fully relevant listings increased from 58% to 62%, demonstrating measurable improvements in aligning search results with buyer intent while addressing the cold-start problem for smaller sellers.
Flipkart
Flipkart faced the challenge of accurately extracting product attributes (like color, pattern, and material) from millions of product listings at scale. Manual labeling was expensive and error-prone, while using large Vision Language Model APIs was cost-prohibitive. The company developed a semi-supervised approach using compact VLMs (2-3 billion parameters) that combines Parameter-Efficient Fine-Tuning (PEFT) with Direct Preference Optimization (DPO) to leverage unlabeled data. The method starts with a small labeled dataset, generates multiple reasoning chains for unlabeled products using self-consistency, and then fine-tunes the model using DPO to favor preferred outputs. Results showed accuracy improvements from 75.1% to 85.7% on the Qwen2.5-VL-3B-Instruct model across twelve e-commerce verticals, demonstrating that compact models can effectively learn from unlabeled data to achieve production-grade performance.
Grammarly
Grammarly developed GECToR, a novel grammatical error correction (GEC) system that treats error correction as a sequence-tagging problem rather than the traditional neural machine translation approach. Instead of rewriting entire sentences through encoder-decoder models, GECToR tags individual tokens with custom transformations (like $DELETE, $APPEND, $REPLACE) using a BERT-like encoder with linear layers. This approach achieved state-of-the-art F0.5 scores (65.3 on CoNLL-2014, 72.4 on BEA-2019) while running up to 10 times faster than NMT-based systems, with inference speeds of 0.20-0.40 seconds compared to 0.71-4.35 seconds for transformer-NMT approaches. The system uses iterative correction over multiple passes and custom g-transformations for complex operations like verb conjugation and noun number changes, making it more suitable for real-world production deployment in Grammarly's writing assistant.
Salesforce
Salesforce's AI platform team faced operational challenges deploying customized large language models (fine-tuned versions of Llama, Qwen, and Mistral) for their Agentforce agentic AI applications. The deployment process was time-consuming, requiring months of optimization for instance families, serving engines, and configurations, while also proving expensive due to GPU capacity reservations for peak usage. By adopting Amazon Bedrock Custom Model Import, Salesforce integrated a unified API for model deployment that minimized infrastructure management while maintaining backward compatibility with existing endpoints. The results included a 30% reduction in deployment time, up to 40% cost savings through pay-per-use pricing, and maintained scalability without sacrificing performance.
Shopify
Shopify's augmented engineering team developed ROAST, an open-source workflow orchestration tool designed to address challenges of maintaining developer productivity at massive scale (5,000+ repositories, 500,000+ PRs annually, millions of lines of code). The team recognized that while agentic AI tools like Claude Code excel at exploratory tasks, deterministic structured workflows are better suited for predictable, repeatable operations like test generation, coverage optimization, and code migrations. By interleaving Claude Code's non-deterministic agentic capabilities with ROAST's deterministic workflow orchestration, Shopify created a bidirectional system where ROAST can invoke Claude Code as a tool within workflows, and Claude Code can execute ROAST workflows for specific steps. The solution has rapidly gained adoption within Shopify, reaching 500 daily active users and 250,000 requests per second at peak, with developers praising the combination for minimizing instruction complexity at each workflow step and reducing entropy accumulation in multi-step processes.
Booking.com
Booking.com built an AI Trip Planner to handle unstructured, natural language queries from travelers seeking personalized recommendations. The challenge was combining LLMs' ability to understand conversational requests with years of structured behavioral data (searches, clicks, bookings). Instead of relying solely on prompt engineering with external APIs, they used supervised fine-tuning on open-source LLMs with parameter-efficient methods. This approach delivered superior recommendation metrics while achieving 3x faster inference compared to prompt-based solutions, while maintaining data privacy and security by keeping all processing internal.
Zed
Zed, an AI-enabled code editor built from scratch in Rust, implemented comprehensive testing and evaluation strategies to ensure reliable agentic editing capabilities. The company faced the challenge of maintaining their rigorous empirical testing approach while dealing with the non-deterministic nature of LLM outputs. They developed a multi-layered approach combining stochastic testing with deterministic unit tests, addressing issues like streaming edits, XML tag parsing, indentation handling, and escaping behaviors. Through statistical testing methods running hundreds of iterations and setting pass/fail thresholds, they successfully deployed reliable AI-powered code editing features that work effectively with frontier models like Claude 4.
ZURU
ZURU Tech, a construction technology company, collaborated with AWS to develop a text-to-floor plan generator that allows users to create building designs using natural language descriptions. The project aimed to improve upon existing GPT-2 baseline results by implementing both prompt engineering with Claude 3.5 Sonnet on Amazon Bedrock and fine-tuning approaches with Llama models on Amazon SageMaker. Through careful dataset preparation, dynamic few-shot prompting, and comprehensive evaluation frameworks, the team achieved a 109% improvement in instruction adherence accuracy compared to their baseline model, with fine-tuning also delivering a 54% improvement in mathematical correctness for spatial relationships and dimensions.
Thinking Machines
Thinking Machines, a new AI company founded by former OpenAI researcher John Schulman, has developed Tinker, a low-level fine-tuning API designed to enable sophisticated post-training of language models without requiring teams to manage GPU infrastructure or distributed systems complexity. The product aims to abstract away infrastructure concerns while providing low-level primitives for expressing nearly all post-training algorithms, allowing researchers and companies to build custom models without developing their own training infrastructure. The company plans to release their own models and expand Tinker's capabilities to include multimodal functionality and larger-scale training jobs, while making the platform more accessible to non-experts through higher-level tooling.
Institute of Science Tokyo
The Institute of Science Tokyo successfully developed Llama 3.3 Swallow, a 70-billion-parameter large language model with enhanced Japanese capabilities, using Amazon SageMaker HyperPod infrastructure. The project involved continual pre-training from Meta's Llama 3.3 70B model using 314 billion tokens of primarily Japanese training data over 16 days across 256 H100 GPUs. The resulting model demonstrates superior performance compared to GPT-4o-mini and other leading models on Japanese language benchmarks, showcasing effective distributed training techniques including 4D parallelism, asynchronous checkpointing, and comprehensive monitoring systems that enabled efficient large-scale model training in production.
Patronus AI
Patronus AI addressed the critical challenge of LLM hallucination detection by developing Lynx, a state-of-the-art model trained on their HaluBench dataset. Using Databricks' Mosaic AI infrastructure and LLM Foundry tools, they fine-tuned Llama-3-70B-Instruct to create a model that outperformed both closed and open-source LLMs in hallucination detection tasks, achieving nearly 1% better accuracy than GPT-4 across various evaluation scenarios.
OpenAI
OpenAI's development and training of GPT-4.5 represents a significant milestone in large-scale LLM deployment, featuring a two-year development cycle and unprecedented infrastructure scaling challenges. The team aimed to create a model 10x smarter than GPT-4, requiring intensive collaboration between ML and systems teams, sophisticated planning, and novel solutions to handle training across massive GPU clusters. The project succeeded in achieving its goals while revealing important insights about data efficiency, system design, and the relationship between model scale and intelligence.
Swiggy
Swiggy, a major food delivery platform in India, implemented a novel two-stage fine-tuning approach for language models to improve search relevance in their hyperlocal food delivery service. They first performed unsupervised fine-tuning using historical search queries and order data, followed by supervised fine-tuning with manually curated query-item pairs. The solution leverages TSDAE and Multiple Negatives Ranking Loss approaches, achieving superior search relevance metrics compared to baseline models while meeting strict latency requirements of 100ms.
Grab
Grab developed a custom foundation model to generate user embeddings that power personalization across its Southeast Asian superapp ecosystem. Traditional approaches relied on hundreds of manually engineered features that were task-specific and siloed, struggling to capture sequential user behavior effectively. Grab's solution involved building a transformer-based foundation model that jointly learns from both tabular data (user attributes, transaction history) and time-series clickstream data (user interactions and sequences). This model processes diverse data modalities including text, numerical values, IDs, and location data through specialized adapters, using unsupervised pre-training with masked language modeling and next-action prediction. The resulting embeddings serve as powerful, generalizable features for downstream applications including ad optimization, fraud detection, churn prediction, and recommendations across mobility, food delivery, and financial services, significantly improving personalization while reducing feature engineering effort.
Pinterest sought to evolve from a simple content recommendation platform to an inspiration-to-realization platform by understanding users' underlying, long-term goals through identifying "user journeys" - sequences of interactions centered on particular interests and intents. To address the challenge of limited training data, Pinterest built a hybrid system that dynamically extracts keywords from user activities, performs hierarchical clustering to identify journey candidates, and then applies specialized models for journey ranking, stage prediction, naming, and expansion. The team leveraged pretrained foundation models and increasingly incorporated LLMs for tasks like journey naming, expansion, and relevance evaluation. Initial experiments with journey-aware notifications demonstrated substantial improvements, including an 88% higher email click rate and 32% higher push open rate compared to interest-based notifications, along with a 23% increase in positive user feedback.
Modal
Modal's engineering team tackled the challenge of generating aesthetically pleasing QR codes that consistently scan by implementing comprehensive evaluation systems and inference-time compute scaling. The team developed automated evaluation pipelines that measured both scan rate and aesthetic quality, using human judgment alignment to validate their metrics. They applied inference-time compute scaling by generating multiple QR codes in parallel and selecting the best candidates, achieving a 95% scan rate service-level objective while maintaining aesthetic quality and returning results in under 20 seconds.
Flipkart
Flipkart faced the challenge of evaluating AI-generated opinion summaries of customer reviews, where traditional metrics like ROUGE failed to align with human judgment and couldn't comprehensively assess summary quality across multiple dimensions. The company developed OP-I-PROMPT, a novel single-prompt framework that uses LLMs as evaluators across seven critical dimensions (fluency, coherence, relevance, faithfulness, aspect coverage, sentiment consistency, and specificity), along with SUMMEVAL-OP, a new benchmark dataset with 2,912 expert annotations. The solution achieved a 0.70 Spearman correlation with human judgments, significantly outperforming previous approaches especially on open-source models like Mistral-7B, while demonstrating that high-quality summaries directly impact business metrics like conversion rates and product return rates.
Fight Health Insurance
Fight Health Insurance is an open-source project that uses fine-tuned large language models to help people appeal denied health insurance claims in the United States. The system processes denial letters, extracts relevant information, and generates appeal letters based on training data from independent medical review boards. The project addresses the widespread problem of insurance claim denials by automating the complex and time-consuming process of crafting effective appeals, making it accessible to individuals who lack the resources or knowledge to navigate the appeals process themselves. The tool is available both as an open-source Python package and as a free hosted service, though the sustainability model is still being developed.
Windsurf
Windsurf developed Tab v2, an AI-powered code autocomplete system that addresses the challenge of balancing prediction frequency, accuracy, and code length in developer tooling. The team reimagined their LLM-based autocomplete by focusing on total keystrokes saved rather than just acceptance rate, implementing extensive context engineering to reduce prompt length by 76%, and using reinforcement learning to train models with different "aggression" levels. The result was a 54% average increase in characters per prediction and 25-75% more accepted code, with user-selectable aggression parameters allowing developers to customize behavior based on personal preferences.
Meta
Meta's Media Foundation team deployed AI-powered video super-resolution (VSR) models at massive scale to enhance video quality across their ecosystem, processing over 1 billion daily video uploads. The problem addressed was the prevalence of low-quality videos from poor camera quality, cross-platform uploads, and legacy content that degraded user experience. The solution involved deploying multiple VSR models—both CPU-based (using Intel's RVSR SDK) and GPU-based—to upscale and enhance video quality for ads and generative AI features like Meta Restyle. Through extensive subjective evaluation with thousands of human raters, Meta identified effective quality metrics (VMAF-UQ), determined which videos would benefit most from VSR, and successfully deployed the technology while managing GPU resource constraints and ensuring quality improvements aligned with user preferences.
