On this page
On July 13, 2026, Prefect announced it had agreed to acquire Dagster Labs.
If you were evaluating Dagster vs Prefect two weeks ago, you were choosing between two competitors. Today, you are choosing between two products with a single owner and a shared thesis, summarized in one line from the announcement: ‘Dagster defines outcomes, Prefect executes the work, and FastMCP governs access.’
To be clear, this is not a distress story. Both are excellent, widely deployed orchestrators, and their commitments are unusually specific. As per Dagster’s letter to customers:
- Dagster and Dagster+ keep their names, open-source licenses, and roadmaps, maintained by many of the same engineers.
- Pricing for both Prefect Cloud and Dagster+ is unchanged.
- Both open-source projects continue to receive maintenance releases, new features, and security patches.
If you are making a three-to-five-year platform bet this quarter, this Dagster vs Prefect vs ZenML comparison will help you find the right fit. We compare all three tools across outputs and assets, lineage and reproducibility, infrastructure portability, and failure recovery.
Dagster vs Prefect vs ZenML: Key Takeaways
- Dagster: Pick if your team thinks in data assets rather than tasks. Software-defined assets, partitions, backfills, and the lineage graph make it a great choice when your question begins with the origin. It’s a data platform tool first, with ML supported through integrations.
- Prefect: Pick if you want the shortest path from working Python to a scheduled run with retries and history. The
@flowand@taskdecorators impose no DAG ceremony, and keep the code close to ordinary Python. - ZenML: Pick if the thing you are orchestrating is machine learning. Every returned step output becomes a versioned artifact linked to its run, producing step, upstream inputs, and stack configuration. Exact Git commit tracking requires a configured code repository, while immutable snapshots containing the pipeline DAG, code, configuration, and container images are a ZenML Pro feature.
👀 Note: Dagster and Prefect are now one company. If orchestrator independence matters to your platform strategy, an abstraction layer above the orchestrator is the cleanest hedge available.
What Pain Point Does Each Solve for You?
All three are Python frameworks that run workflows, and that similarity hides very different centers of gravity.
Dagster
Dagster was built for data platform teams that lost track of what data exists, who owns it, and whether it is up to date. Instead of organizing the system primarily around tasks, you define assets, the computations that produce them, and their dependencies. Dagster then uses those definitions to plan materializations, track lineage, and determine what may need updating.
That makes partitioned backfills, freshness tracking, and a browsable asset catalog feel native rather than bolted on. ML teams can place training data and feature tables in one lineage graph, provided you model them as assets.
Prefect
Prefect was built for engineers who already have working Python and need it to survive schedules, retries, flaky APIs, and infrastructure.
Prefect records operational run history by default and now provides an Assets API for materializations, dependencies, and metadata. However, you must explicitly adopt @materialize, configure result persistence where needed, and manage physical data versioning and complete execution-environment reproduction separately.
ZenML
ZenML was built for ML platform teams facing the last-mile gap between experimentation and production. You need to know which data and code produced a model, then move the same pipeline onto production infrastructure.
ZenML answers it with a pipeline abstraction where reproducibility is the default and infrastructure is a swappable stack rather than rigid code. Stacks keep pipeline logic separate from execution and storage, so reproducibility becomes part of the pipeline model.
Here is a quick factual snapshot:
| Dagster | Prefect | ZenML | |
|---|---|---|---|
| First public release | Introduced July 2019 (repo since 2018) | Prefect Core, 2018 | v0.1, December 2020 |
| GitHub stars | ~15.8k | ~23.3k | ~5.5k |
| Latest version (July 2026) | 1.13.14 | 3.7.8 | 0.96.2 |
| Core abstraction | Software-defined assets | Flows and tasks, dynamic runtime | Pipelines, steps, and stacks |
Comparing Features: Dagster vs Prefect vs ZenML
Short on time? Here is the TL;DR of the four features we compare in depth:
| Feature | Dagster | Prefect | ZenML |
|---|---|---|---|
| Outputs and assets | Assets are core; I/O managers handle storage and metadata | Results are temporary by default; persistence and assets require setup | Every step output is stored and versioned as an artifact |
| Lineage and reproducibility | Asset graph, materialization history, and code/data versions; first-party column-level lineage in the Dagster+ UI is listed for the Pro plan. | Run states and logs by default; asset lineage requires @materialize | Tracks code, data, config, and images per run and model version |
| Infrastructure portability | Executors and launchers control deployment; Pipes connect external compute | Work pools route flows; infrastructure is set per deployment | Stacks swap orchestrators; steps can use separate resources and containers |
| Failure recovery | Op and run retries, plus partition backfills; no default caching | Task and flow retries, cache policies with persisted results, and transactions with user-defined commit and rollback hooks. | Step retries, default caching, and selective output reuse |
Feature 1. Output Persistence, Artifact Management, and Asset Semantics
What happens to the objects your pipeline produces? This section covers how pipeline outputs are stored and serialized. It also looks at version history and reuse.
ZenML
In ZenML, artifact management is not something you switch on later. It is the default behavior of every pipeline; any value returned from a step becomes a versioned artifact.
Materializers pick the serialization based on your output type. Built-in materializers cover primitives, NumPy, Pandas, or common ML objects, and the result lands versioned in your artifact store, like S3, GCS, Azure Blob, or a local directory.
from zenml import pipeline, step
import pandas as pd
from sklearn.base import ClassifierMixin
@step
def train_model(train_df: pd.DataFrame) -> ClassifierMixin:
model = ... # your training logic
return model # serialized, versioned, and linked to this run automatically
@pipeline
def training_pipeline():
train_df = load_data()
train_model(train_df)ZenML records metadata and can generate visualizations for supported artifact types, and works genuinely well for ML objects. However, if your workload is mostly in-place SQL transformations, an artifact-passing model is the wrong shape.
Dagster
In Dagster, an @asset function’s return value is written by an I/O manager, a pluggable component that separates compute from storage. The default FilesystemIOManager stores values to local disk with no configuration. You can replace that component with an I/O manager for S3, Snowflake, or another system without rewriting the asset logic.
Every materialization is recorded as an event with metadata. Asset checks can validate data quality, and checks configured with blocking=True can stop downstream assets when an error-severity check fails.
However, Dagster is honest about I/O managers not always being the right tool. If you have SQL that updates tables in place, or data too large for memory, they recommend skipping the abstraction entirely.
Prefect
Prefect splits this problem into three concepts: Results, Artifacts, and Assets.
Results are your task return values, and by default they are ephemeral; no reference is maintained in the API. Cross-run recovery or caching requires result persistence and a storage block.
from prefect import task
from prefect_aws.s3 import S3Bucket
@task(
persist_result=True,
result_storage=S3Bucket.load("ml-results"),
result_serializer="pickle",
)
def train_model(train_df):
...Artifacts are human-readable outputs shown in the UI. They can appear as markdown, tables, links, or progress bars.
Assets use the @materialize decorator to record that a task produced an object at a URI like s3://bucket/features. It is intent-based. As in, Prefect records the event and builds lineage from it, but your code still performs the write.
Prefect is actually strong in flexibility. It does not force a storage abstraction on the workflow, and for teams with existing data conventions, that is a feature.
Verdict: ZenML provides the strongest default lifecycle for ML outputs. Dagster has the deepest asset model. Prefect gives you the most control over what gets persisted, but that control comes with more setup.
Feature 2. Lineage, Metadata, and Run Reproducibility
Suppose a model starts behaving differently three weeks after training. How much of the original run can the platform reconstruct for you?
ZenML
A ZenML pipeline run records the whole picture by default. Steps, every input or output artifact, configuration, the stack it ran on, logs, and cache status. If you desperately need a breath there, try connecting a code repository, and each run also pins the exact Git commit.
Two pieces make this ML-shaped. The Model Control Plane groups pipelines, artifacts, and metadata under model versions with stages like staging and production. Pipeline snapshots capture the DAG and code, plus configuration and container images.
Model-management APIs are available programmatically in OSS, while the full visual Model Control Plane is a Pro feature. Running pipeline snapshots through the SDK, CLI, dashboard, or API is also a ZenML Pro-only feature.
Dagster
Dagster’s event log is its memory. Every asset builds a materialization history with metadata, and the global lineage graph shows dependencies across the deployment. Assets also carry code versions and data versions, which help identify stale downstream assets.
Dagster can carry column-lineage metadata, while first-party column-level lineage in the Dagster+ catalog is currently listed as a Pro-plan feature.
Dagster’s current FreshnessPolicy API became generally available in Dagster 1.12, replacing the legacy freshness API. The Freshness Daemon now runs by default and can be disabled through configuration.
Undoubtedly, Dagster’s asset lineage is the best of the three. But because it’s focused on asset history and not the full run environment, reconstructing an older container image or package set still depends on your deployment process.
Prefect
In Prefect, every flow and task run has a state history. It also distinguishes a code-level Failed state and an infrastructure-level Crashed state. Logs, events, and parameters add the rest of the operational record. Persisted results make the actual return values recoverable too, and keyed artifacts version themselves across runs.
Data lineage, however, depends on adopting the assets API. Where @materialize is used, Prefect infers dependencies from declared asset events and builds a workspace-wide lineage view.
Two details though: cached task runs do not emit new asset events, and the asset graph and catalog are part of Prefect Cloud features and not the open-source server.
Prefect is your buddy when the question is “what happened last night?” But just to be clear, it’s only answerable to the degree you wired him up.
Feature 3. Execution Backends and Infrastructure Portability
Can code written on a laptop run on Kubernetes next week and a managed training service after that? The answer depends on what each platform treats as configuration.
ZenML
Portability is a core ZenML idea. A stack bundles an orchestrator, artifact store, container registry, and optional components like step operators. Pipeline code does not reference those components directly, so changing infrastructure is often one CLI command, for example:
zenml stack set local && python run.py # runs locally
zenml stack set k8s-prod && python run.py # same code, now containerized on KubernetesWe ship more than a dozen orchestrator integrations including Kubernetes, Kubeflow, Airflow, SageMaker, Vertex AI, AzureML, and others. Our step operators run a single step on separate specialized infrastructures. For example, a training step could use a GPU instance while the rest of the pipeline stays on a smaller compute.
from zenml import step
from zenml.config import ResourceSettings
@step(settings={"resources": ResourceSettings(cpu_count=8, gpu_count=2, memory="16GB")})
def train_model(train_df):
...Because ZenML delegates execution to the orchestrator underneath, capabilities vary by stack, and not every orchestrator enforces resource settings.
The active orchestrator still sets the boundary. Kubernetes can enforce these settings, while some backends may ignore part of them.
Dagster
Dagster separates the concern into two layers:
- Executors: control how steps inside a run execute, like in-process, multiprocess, Docker, Celery, or one Kubernetes pod per step
- Run launchers: decide where the run worker starts, like a process, container, Kubernetes job, or ECS.
Both are deployment configuration rather than asset code. Dagster Pipes extends this model to jobs that run on external systems. Those jobs can send logs and materialization metadata back to Dagster.
A per-asset compute is possible through Kubernetes tags. Like in the example below, a dagster-k8s/config tag with container resource limits, taking effect under the k8s_job_executor:
import dagster as dg
@dg.asset(
op_tags={
"dagster-k8s/config": {
"container_config": {
"resources": {"limits": {"cpu": "2", "memory": "16Gi", "nvidia.com/gpu": "1"}}
}
}
}
)
def trained_model(features):
...Those tags only work under the Kubernetes executor. Separate pods also require shared storage, so teams usually replace the local filesystem I/O manager with S3 or another shared store.
Prefect
Prefect routes deployments through work pools and workers. A deployment targets a pool, and workers pick up runs on the configured infrastructure, while Push pools remove the worker for supported serverless targets.
Prefect Managed pools run flows on Prefect’s own compute. Each pool has a base job template that deployments override with job_variables to vary images, namespaces, or machine sizes.
The main infrastructure unit is the flow run. Tasks do not normally route to separate deployment targets. The idiomatic workarounds are subflows deployed to different pools, or a Dask or Ray task runner distributing tasks across an existing cluster.
This work pool model is flexible across a wide infrastructure menu, and moving a deployment between targets is configuration. However, per-task computation or, let’s say, the GPU-training-step problem you hit constantly will require restructuring your flows around it.
Verdict: Prefect makes it easy to move whole flows between deployment targets. Dagster separates run and step execution with clear deployment controls. ZenML is the strongest fit when one ML pipeline needs different orchestrators or per-step compute.
Feature 4. Retries, Caching, and Partial Re-Execution
When step seven of nine fails after four hours of compute, do you have to pay again?
ZenML
from zenml.config.retry_config import StepRetryConfig
@step(
retry=StepRetryConfig(
max_retries=3, # Maximum number of retry attempts
delay=10, # Initial delay in seconds before first retry
backoff=2 # Factor by which delay increases after each retry
)
)
def unreliable_step():
# This step might fail due to transient issues
...ZenML layers three mechanisms. Step retries handle temporary failures. Caching is on by default, so later runs can reuse an artifact when the step code and inputs have not changed. Parameters and configuration are part of the cache decision too.
Replays, added in ZenML 0.94, help you start from an existing run and choose which steps to skip. ZenML reuses the selected outputs even when caching was disabled for the original run. You can also override inputs or step configuration.
In practice, your four-hour feature engineering succeeds before training fails with an out-of-memory error. You can replay the run, skip the completed feature steps, and give the training step a larger instance.
Dagster
import dagster as dg
@dg.job(tags={"dagster/max_retries": 3})
def sample_job():
pass
@dg.job(tags={"dagster/max_retries": 3, "dagster/retry_strategy": "ALL_STEPS"})
def other_sample_job():
passA Dagster RetryPolicy gives ops and assets declarative retries with delay, backoff, and jitter.
Run retries can restart a failed run, defaulting to FROM_FAILURE. Successful steps are skipped when their outputs are available to the new run.
The UI offers the same recovery path used through re-execution from failure or from a selected set of steps. For partitioned assets, backfills recompute exactly the slices that need it.
However, FROM_FAILURE re-execution needs an I/O manager whose outputs are reachable across runs. On Kubernetes, the S3 pickle I/O manager works, but the default filesystem one does not, because the retry runs in a fresh pod with a fresh disk.
Dagster also has no default cross-run cache. You must rely on code versions, data versions, and storage rules to avoid unnecessary work.
Dagster’s from-failure re-execution is mature when storage is configured properly, though it might feel limiting because nothing is cached automatically. Skipping redundant recomputation is something you architect with versions and I/O managers rather than get by default.
Prefect
Prefect gives you retry controls at both the task and flow level. You can set a fixed delay, provide a different delay for each attempt, or generate an exponential backoff schedule. Jitter helps prevent many failed runs from retrying at the same time.
Prefect also supports conditional retries, so you can inspect the exception or task state before deciding whether another attempt makes sense. Defaults can be set globally and overridden for individual tasks.
Caching is composable, and uses policies that can combine task inputs and source code:
from prefect import task
from prefect.cache_policies import INPUTS, TASK_SOURCE
@task(
cache_policy=INPUTS + TASK_SOURCE,
persist_result=True, # cross-run caching requires persisted results
retries=3,
)
def featurize(raw):
...Cross-run caching requires persisted results, and persistence is off by default. The same rule affects manual recovery because Prefect can only skip completed work when it can retrieve the earlier result.
Prefect 3’s transactions API adds a genuinely distinctive tool. You can group tasks in a transaction, attach on_rollback hooks, and use idempotency keys so re-runs skip work that is already committed.
Verdict: Dagster has mature re-execution and partition backfills. Prefect offers the most detailed retry controls and adds transaction support. ZenML gives ML teams the simplest default path to cached artifacts and selective replay.
Dagster vs Prefect vs ZenML: Integrations
ZenML
ZenML organizes integrations as stack components. Each component handles one part of the ML workflow, so you can change the underlying service without rewriting the pipeline around a new API.
ZenML lists 66 integrations, including 11 agent-framework integrations.
- Orchestrators: Airflow, Kubernetes, Kubeflow, SageMaker, Vertex AI, Azure ML, and Databricks
- Artifact stores: Amazon S3, Google Cloud Storage, Azure Blob Storage, and local storage
- Experiment trackers: MLflow, Weights & Biases, and Comet
- Deployers and pipeline deployments: Docker, Kubernetes, AWS App Runner, GCP Cloud Run, Hugging Face Inference Endpoints
- Data quality and feature stores: Great Expectations, Evidently, Deepchecks, and Feast
- Agent frameworks: LangGraph, PydanticAI, CrewAI, and the OpenAI Agents SDK
The main benefit is the common interface. Moving from Kubernetes to SageMaker becomes a stack change rather than a pipeline rewrite. The same applies when replacing an experiment tracker or artifact store.
Dagster
Dagster’s catalog leans heavily toward analytics, ingestion, and data transformation. Many integrations bring external tables and jobs into Dagster as assets, so teams can view them through one lineage graph.
Its main integration groups include:
- Transformation: dbt and SQLMesh
- Data ingestion: Fivetran, Airbyte, dlt, Sling, and Meltano
- Warehouses and databases: Snowflake, BigQuery, Redshift, DuckDB, PostgreSQL, and MySQL
- Compute and cloud: Databricks, Kubernetes, AWS, Google Cloud, and Azure
- Data quality: Great Expectations, Soda, and other validation tools
- ML tooling: MLflow and Weights & Biases
Dagster Components packages many of these connections as reusable definitions through YAML and the dg CLI. Dagster Pipes covers jobs that run outside Dagster. External training or processing jobs can still report logs and materialization metadata back to the asset graph.
Prefect
Prefect maintains a smaller first-party integration catalog. Its collections mainly cover cloud infrastructure, distributed execution, and common data services.
The main packages include:
- Cloud platforms: AWS, GCP, and Azure
- Containers and clusters: Docker, Kubernetes
- Distributed execution: Dask, Ray
- Data platforms: Snowflake, Databricks, DBT
- Storage and messaging: S3, Google Cloud Storage, Azure Blob Storage, and common database services
Prefect does not provide first-party MLflow or Weights & Biases packages. You can still call their Python SDKs inside a Prefect task, but your team owns the setup and metadata flow.
Dagster vs Prefect vs ZenML: Pricing
All three tools offer a free Apache 2.0 open-source edition and a paid cloud product. Both announcements explicitly state Dagster+ and Prefect Cloud pricing is unchanged by the acquisition.
ZenML
ZenML OSS is free and self-hosted. The Scale plan is a managed SaaS offering, starting at $399/month for 500 executions, 1 project, and 1 snapshot. Higher Scale variants cost $999 and $2,499 per month.
We also offer an Enterprise plan with unlimited executions and projects for which you can talk to an engineer from our team.
Dagster
Dagster OSS is free to self-host. Dagster+ includes platform and pipeline metrics across its current plans, while custom metrics are listed as a Pro feature. Retention and availability should be described according to the selected plan rather than as one universal 30-day window.
Its self-serve tiers are:
- Solo: $10 per month plus $0.040 per credit
- Starter: $100 per month plus $0.035 per credit
- Pro and Enterprise: Custom pricing
Prefect
Prefect Open Source is free to self-host. Prefect Cloud plans are based mainly on users and deployment, with a free forever Hobby plan and three paid plans:
- Hobby: Free
- Starter: $100 per month
- Team: Starts at $400 per month
- Pro and Enterprise: Custom pricing
Final Recommendation: Which Platform Fits Your Needs?
There is no universal winner in Dagster vs Prefect vs ZenML, and the acquisition does not change the technical calculus so much as sharpen the strategic one.
- Choose Dagster if your pain is data platform pain
- Choose Prefect if your pain is workflow pain
- Choose ZenML if your pain is ML pain
Our read on the acquisition is simple. The announcement alone is no reason to move an existing Dagster or Prefect deployment. If you are choosing a new platform in 2026, remember that two visible orchestration products are set to share one owner.
If you want to see what orchestrator-independent ML pipelines feel like, try the open-source version of ZenML with your current stack. And when you are ready to standardize across teams, book a demo to see how ZenML Pro handles artifact lineage, model control, and replay at organizational scale.
Relevant reading:
