Scaling ZenML: 200x Performance Improvement Through Database and FastAPI Optimizations in v0.83.0
A technical deep dive into the performance optimizations that improved ZenML's throughput by 200x
Jun 2, 202515 mins
12 posts with this tag
A technical deep dive into the performance optimizations that improved ZenML's throughput by 200x
ZenML 0.80.0 transforms tenant structures into workspace/project hierarchies with advanced RBAC for Pro users, while enhancing tagging, resource filtering, and dashboard design. Open-source improvements include Kubernetes security upgrades, SkyPilot integration, and significantly faster CLI operations. Both Pro and OSS users benefit from dramatic performance optimizations, GitLab improvements, and enhanced build tracking.
ZenML 0.75.0 introduces dashboard enhancements that allow users to create and update stack components directly from the dashboard, along with improvements to service connectors, model artifact handling, and documentation. This release streamlines ML workflows with better component management capabilities, enhanced SageMaker integration, and critical fixes for custom flavor components and sorting logic.
ZenML 0.74.0 introduces key cloud provider features including SageMaker pipeline scheduling, Azure Container Registry implicit authentication, and Vertex AI persistent resource support. The release adds API Tokens for secure, time-boxed API authentication while delivering comprehensive improvements to timezone handling, database performance, and Helm chart deployments.
ZenML 0.71.0 features the Modal Step Operator for fast, configurable cloud execution, dynamic artifact naming, and enhanced visualizations. It improves API token management, dashboard usability, and infrastructure stability while fixing key bugs. Expanded documentation supports advanced workflows and big data management.
ZenML 0.70.0 has launched with major improvements but requires careful handling during upgrade due to significant database schema changes. Key highlights include enhanced artifact versioning with batch processing capabilities, improved scalability through reduced server requests, unified metadata management via the new log_metadata method, and flexible filtering with the new oneof operator. The release also features expanded documentation covering finetuning and LLM/ML engineering resources. Due to the database changes, users must back up their data and test the upgrade in a non-production environment before deploying to production systems.
ZenML 0.68.0 introduces several major enhancements including the return of stack components visualization on the dashboard, powerful client-side caching for improved performance, and a streamlined onboarding process that unifies starter and production setups. The release also brings improved artifact management with the new `register_artifact` function, enhanced BentoML integration (v1.3.5), and comprehensive documentation updates, while deprecating legacy features including Python 3.8 support.
This release incorporates updates to the SageMaker Orchestrator, DAG Visualizer, and environment variable handling. It also includes Kubernetes support for Skypilot and an updated Deepchecks integration. Various other improvements and bug fixes have been implemented across different areas of the platform.
ZenML's latest release 0.66.0 adds support for Python 3.12, removes some dependencies for a slimmer Client package and adds the ability to view all your pipeline runs in the dashboard.
ZenML's latest release 0.65.0 enhances MLOps workflows with single-step pipeline execution, AzureML SDK v2 integration, and dynamic model versioning. The update also introduces a new quickstart experience, improved logging, and better artifact handling. These features aim to streamline ML development, improve cloud integration, and boost efficiency for data science teams across local and cloud environments.
ZenML's latest release 0.64.0 streamlines MLOps workflows with notebook integration for remote pipelines, optimized Docker builds, AzureML orchestrator support, and Terraform modules for cloud stack provisioning. These updates aim to speed up development, ease cloud deployments, and improve efficiency for data science teams.
Recent releases of ZenML’s Python package have included a better way to deploy machine learning infrastructure or stacks, new annotation tool integrations, an upgrade of our Pydantic dependency and lots of documentation improvements.