Dagster Labs Reviews

I used Dagster Labs in a previous assignment in 2024, where I was working on a POC that required DBT and Dagster Labs integration for ETL pipeline orchestration. In that project, I worked mainly on DBT, and we selected Dagster Labs for the orchestration purpose so that we could create an ETL pipeline with orchestration and scheduling jobs.

The main use case I worked on for Dagster Labs was the DBT with Dagster Labs integration, where all our ETL jobs were built into DBT. There was a need to create a lineage and also the orchestration, and for the orchestration purpose, we used Dagster Labs.

Dagster Labs helped with lineage and orchestration in my project by allowing us to see what the impacted upstream and downstream systems were, which was a limitation in DBT. It enabled us to create dependencies in DBT, and I saw the flexibility of Dagster Labs where we could directly integrate it seamlessly with the DBT tool. All the ETL jobs could be imported directly into Dagster Cloud, and we could create the dependencies and schedule the jobs based on the frequency decided by the business, and it was quite seamless.

The best features Dagster Labs offers mainly include error handling, job scheduling, and lineage visualization, with all three being quite easy to manage. After fixing the code, we do not have to rerun the pipeline again; it automatically restarts. There is also an option to put data validation and data contracts within Dagster Labs. If there is any data quality issue that happens from the source, we can put the data validation checks over there, and Dagster Labs can easily check that and based on the results or the outcome of it, Dagster Labs can run the pipeline. The data lineage part is quite visible, making it easy to see what is the source, what is the target, and what are the in-between intermediate layers we have put in place for the pipeline.

The data contract or data validation part is a great feature because it has enabled us to do our quality checks at the runtime and that has saved a lot of manual efforts from the engineering team.

Dagster Labs has impacted my organization positively by enabling faster project delivery since it has reduced a lot of manual efforts, especially the manual scheduling part. With Dagster Labs, we have scheduled all our jobs and orchestration automatically, and we did not have any data lineage tool or orchestration tool before. After onboarding Dagster Labs, we have achieved some good results.

In terms of project delivery, it has reduced the effort significantly because a lot of manual things were automated, resulting in our timelines being met on time. Earlier, it was getting delayed, but we have made our deliveries right on time, so that was the really outstanding outcome.

There are multiple products from Dagster Labs: Dagster Cloud, Dagster Labs, and sometimes it is quite confusing to choose which one for what purposes. I know there are some licensing buckets for small organizations, medium organizations, and bigger organizations, so the modeling and the cost estimation part could be more intuitive, making it easier for beginners to understand how much time and money they would save by onboarding Dagster Labs in their project.

I deducted two points because I faced some challenges working with the Git repository integration with Dagster Labs due to some security keys or some weird issue. I had to get in a call with Dagster Labs subject matter expert, and we really struggled for two to three weeks because we could not establish the secure connection through that.

I have been working in the Data and AI industry for almost 14 years now.

Dagster Labs is quite stable.

Dagster Labs is quite scalable, allowing us to scale it to enormous sizes.

The customer support was good; I had a weekly call with them when I was running into some issues, and they helped me out promptly.

I would rate the customer support nine out of 10.

We were not using any previous solution, so that was the first experience, having onboarded Dagster Labs.

The initial setup is quite good. I am really impressed.

The documentation is quite clear, and the integration is seamless as I mentioned, such as with DBT, which was good. The user experience was also good.

I do not have the numbers with me, but we have definitely reduced the manual efforts, resulting in one or two fewer employees needed and some productive tasks being accomplished. We have saved some time over there.

The pricing, setup cost, and licensing were quite intuitive but not as clear for the data architect to understand what would be the cost estimation when onboarding Dagster Labs.

We were thinking of using Airflow, but after comparing it with Dagster Labs, Dagster Labs came out as the winner, leading us to decide to go ahead with it.

One of the user experiences was that earlier DBT was a back-end tool, and Dagster Labs is something we can actually show to stakeholders with all the job run logs. It is easy to monitor all your running jobs and also what has been completed and what has errored out, and also the error status messages are quite descriptive. If there is any error that occurs, you can look into the error log details and then easily debug that and identify the error and fix it. If there is any failure that happens at one checkpoint, we do not need to restart the job from the start; from that checkpoint itself, we can resume the job after fixing the code, and it was quite seamless.

I would advise others to start using Dagster Labs and explore their functionalities and features because it is easy to onboard, and their cloud-native features are really awesome. It has a great lineage with all the ETL tools and also with GCP and other cloud platforms, so I would definitely recommend others to start using it.

From the security perspective, everything was good, and I do not have any feedback on that, to be honest.

It met all the organization's requirements. I would rate this review an overall 8 out of 10.

I perform orchestration around data by connecting raw app data using Fivetran inside BigQuery warehouse to DBT. I do this entire process using Dagster Labs.

My workflow is that Fivetran reads data from the raw data source and sends the data to BigQuery. From BigQuery, the data is read inside the BI dashboards. The entire pipeline from raw to Fivetran to BigQuery to BI dashboard is monitored inside Dagster Labs.

What I find beneficial with Dagster Labs is that it excels at pipeline monitoring. It provides good notification of schema changes and indicates whether the pipeline is stable or not.

I appreciate the visual lineage graph that Dagster Labs offers. It shows exactly which upstream table or API pull failed, timed out, or produced an anomaly.

Dagster Labs has given me better monitoring ability. I know whether data can be trusted based on what I see on the dashboards. This gives me more confidence in my data sources.

Dagster Labs has provided a good sense of the data pipeline I have been using. I have started to trust it even more because I have full confidence in whether it has been updated correctly and if it has failed. When there are any anomalies, I end up trusting my data even more. I have full confidence and control over the data pipeline. Overall, it has created a very positive impact.

One of the features I appreciate in Dagster Labs is resources, which is a cool capability to have.

There are many ways Dagster Labs can be improved. I believe the UI is very slow and it prevents loading state.

Regarding needed improvements, I think there should be very good support available for migration if someone is migrating from a legacy system to Dagster Labs. Maybe that support needs to be better in a way that there should be customer service available for it, with more documentation or GitHub documentation that details what you should do and should not do, including do's and don'ts and how it will be beneficial or not beneficial. Something approach may be helpful during migration.

I have been using Dagster Labs for around a year.

It has definitely improved my downtime by twenty percent in relation to Dagster Labs.

Based on how Dagster Labs works, I have more confidence in them. They would surely have a very good model to support all of the features they are providing.

My advice to others looking into using Dagster Labs is to take it slow and give time for things to develop. Things will get better.

I would rate this product an eight out of ten.

My main use case for Dagster Labs is data pipeline orchestration.

A specific example of how I am using Dagster Labs for data pipeline orchestration is that Dagster can be used with dbt, which is another technology that allows scheduling data modeling pipelines and integration with other tools. This integration with dbt is particularly useful, and on the monitoring side, we can add data quality monitors to run on a schedule. Additionally, Dagster Labs integrates with Slack for alerts. The specific example is running data models on a schedule.

Regarding my use case, we also use Dagster Labs to integrate with some AWS components, such as AWS Batch jobs, and we use Dagster's Dagster Pipe technology to connect with our orchestration runs within Dagster, although the process is orchestrated into AWS services. While this is not unique, it is one of our use cases. Another use case is triggering remote Spark jobs, which is connected to data pipeline orchestration, making it one of Dagster Labs' official integrations.

Dagster Labs offers excellent features, including a lot of integrations out of the box, allowing for integration with all major players in the data landscape. Another great feature is the polished UI and developer experience. Unlike some other products with clunky UIs, Dagster Labs provides a pleasant experience with a single glass panel approach. This makes it easy to quickly glance at pipeline health, asset lineage, and everything important for a data engineer is just one or two clicks away. Additionally, it is an extensible product, enabling power users to easily adapt available libraries and code to fit their use cases.

A standout feature is the dbt integration, which is probably well-known and regarded as the industry's top integration, prompting my organization and others where I worked to use Dagster Labs.

Dagster Labs has positively impacted my organization mainly with cost management, allowing us to move away from a more locked-down tool, dbt Cloud. The impact was migrating all our dbt Cloud pipelines to Dagster Labs, which allowed us to save money.

Regarding cost savings, I find that while I do not see a reduction in employee hours or improved efficiency—since users were accustomed to working with dbt Cloud and it allowed more business users to interact with the tool—with Dagster Labs, the cost savings are primarily raw savings due to the platform price, limiting access to previous pipelines.

Some points of improvement for Dagster Labs include leveraging integrations with other great tools such as SQLMesh, which many people are interested in, as shown by its traction and frequent requests on GitHub. Additionally, the billing for Dagster Labs Cloud Plus is somewhat restrictive regarding the number of seats, which often pushes users towards the enterprise plan. This raises concerns for mid-sized companies.

I have been using Dagster Labs since 2021.

Dagster Labs is stable, with a really good release cycle.

Dagster Labs is scalable based on your compute environment, allowing for serverless scaling when using Dagster Labs Cloud. While there are limitations on workload size, transitioning to your own cloud platform resolves scalability issues, particularly suitable for Kubernetes.

I only used customer support while setting things up, and although I have not needed to reach out since, my experience was good and responsive.

In previous use cases, we utilized dbt Cloud, which is primarily a tool designed for dbt and has even worse pricing. We switched to Dagster Labs, and before that, we also used open source alternatives such as Prefect and Airflow, which are acceptable tools, with Airflow being the industry standard. However, I believe Dagster Labs is moving in the right direction.

My experiences with pricing, setup cost, and licensing vary. For my current organization, we opted for Dagster Labs open source, and we are considering moving to the cloud offering. In a previous consultancy role, we selected the standard plan, and while the pricing was acceptable, it became limiting when it came to the number of seats, pushing us to choose a simpler plan. The setup cost was negligible as I had prior experience setting up Dagster Labs, and licensing was not a major concern for us.

It is difficult to quantify a return on investment, especially since I am in an individual contributor position and do not have access to metrics as I am not on the leadership side of the organization.

Before choosing Dagster Labs, we evaluated Airflow, specifically its AWS offering, and also looked at Prefect in other roles.

I would rate Dagster Labs an eight out of ten. I chose an eight out of ten mainly due to the pricing and the way it handles seats, which can be unwelcoming for smaller organizations. As a product, it is one of the best I have had the pleasure to work with.

My advice for others looking into using Dagster Labs is to be wary of the pricing model. Understanding how to best leverage Dagster Labs hinges on having a solid grasp of the tool and its extensibility, enabling users to expand use cases creatively, while also avoiding overcomplicating pipelines. Dagster Labs' documentation offers a solid foundation to begin working.

I think it is crucial for Dagster Labs to keep maintaining the open source solution, as I believe it is a good gateway for new customers and essential for the data landscape, despite the for-profit nature of the company.

A unique workflow I have developed is that I have completely automated all the pipelines by using sensors that Dagster Labs provides by default. This is my main use case and a unique flow that I have developed. Since automating with Dagster Labs, I save around 90% of manual effort. I monitor Dagster Labs jobs once a week to check whether everything is working fine, but other than that, I never open Dagster Labs to see how jobs are processing.

What makes the Python-based orchestrator in Dagster Labs stand out for me compared to others is hosting Dagster Labs on Snowflake, which is a very useful feature because due to the heavy pipelines that I run, Snowflake eases my tasks by scaling up and down. Connecting to different sources, I mostly use Spark jobs for submitting Spark jobs directly from Dagster Labs.

Dagster Labs has positively impacted my organization by increasing efficiency because I automated all my pipelines using sensors in Dagster Labs, which reduced a lot of manual effort and less manual intervention. Unless there is a major issue with data or something else, there is no need for manual intervention, and everything goes smoothly.

Smaller pain points include that after I switched from native Dagster Labs to Snowflake, some of the GraphQL APIs started failing; they load only when I load on the browser for the first time or when I have to clear the cache. Sometimes the GraphQL APIs silently fail, which shows the health of the pipelines that have been run in the last 24 hours; I haven't debugged much into it because it's not impacting much, but it could provide some useful metrics.

Dagster Labs handles scheduling and monitoring of pipelines fairly well; however, one issue while scheduling jobs is that I can't trigger another job from another job. If job A succeeds, I can't directly trigger job B from job A. The only possibility I see is using schedule sensor-based jobs, where I poll the current job, and once that current job succeeds, I trigger another job. It's an alternate way, but it would be useful if there was the functionality of triggering job B from job A.

My advice for others looking into using Dagster Labs is to first understand all the functionalities that Dagster Labs offers; take a little bit of time to try and explore all the features, and then it's better to go into implementation. That way, you can design your pipelines better.

I chose eight out of 10 because of the retry issue I mentioned earlier; I would have rated it 10 out of 10 if there was an option to retry the pipeline from the point of failure. It would be great if the feedback items could be released in upcoming versions.

The main use case for using Dagster Labs is to utilize ETL processes such as extract, transform, and load because we already have raw data, and we perform all the necessary transformations, including feature engineering and mathematical calculations, and then we load it to the front end so that customers can see how their data has evolved into something useful, allowing them to make actionable decisions.

Let me consider a scenario where a customer has raw and unreadable data. We need to identify features first. For example, a customer may have IP addresses with last seen timestamps along with raw security logs. During the initial feature engineering phase, we check for use cases such as detecting if a user on the customer side is using unauthorized resources. To accomplish this, we need to perform feature engineering on several attributes, such as the user's authorization level and the objects they are accessing. For a second example, we can detect if a user is authorizing company access after work hours by using the last seen attribute. We can perform feature engineering on such scenarios.

That covers my main use case, but we are also currently trying to use the DLT tool by using Dagster Labs so that we can extract the data as well.

The best features Dagster Labs offers are that it is very beginner-friendly, the UI is very nice, and we can backfill most of the processes. The backfill feature is the best feature I would consider.

When any new user comes in, they can really see the UI and do all sorts of things rather than going into code-level work. When it comes to backfill, if an error occurred in the past and we want to retrieve the data from that point, we can do that using backfill, which makes it good for our workflow.

Dagster Labs has positively impacted our organization by making it much easier to create data pipelines, helping us create new use cases, and allowing us to transform all of the data we have into features, which Dagster Labs has greatly assisted with.

I can say that since using Dagster Labs, we have saved time, increased productivity, and also cut down on some third-party vendor applications, which saves us money in infrastructure costs and other expenses.

Dagster Labs is currently providing a Dagster University course, but those offerings are somewhat high-level. When a user comes in and tries to learn Dagster Labs, there need to be different kinds of projects that they can understand. I do not think that Dagster University course has that.

Also, when we are doing something in Dagster Labs, we need to know that when I make a change in the UI, it needs to be reflected in the code, so as a developer, I can check on the code and understand the workflow. The UI needs to be connected to the code in real-time.

I have been using Dagster Labs for the past four to five months.

Dagster Labs is stable.

Dagster Labs's scalability is good because you can easily create different pipelines for different purposes, which is beneficial.

I would advise others looking into using Dagster Labs to utilize all of the features provided by Dagster Labs, as it has many great offerings. I would rate this product a seven out of ten.

I primarily use Dagster Labs for asset-based orchestration, scheduling and automating pipelines, and visualizing dependencies between datasets, and testing it. I test unit test cases in the pipeline.

A practical example of how I use Dagster Labs for asset-based orchestration is when I take the raw orders, clean orders, sales fact tables, daily revenue summary, and VBA dashboard. I use Dagster Labs to track the dependency between these assets. When a raw table arrives, the dependent assets are recomputed. If the sales table fails due to a schema change, Dagster Labs prevents downstream assets from running, highlighting the failed assets in the lineage graph. This allows me to check the logic behind the data, and it also allows me to rematerialize only the affected assets after the fix. This asset-centric approach makes it more reliable and easier to debug when orchestrating individual assets.

I also use Dagster Labs to create scheduled pipeline jobs for running the pipeline.

One of the best features that Dagster Labs offers is the asset-aware data orchestration.

Asset-aware data orchestration is valuable for me because it becomes easy to find out when an asset fails and how the asset failed when the entire pipeline runs. It becomes easy for me to fix the problem, rerun the asset, materialize it, and see it in the lineage graph.

Dagster Labs has positively impacted my organization because whenever there is a change in the data, it becomes easy for me to schedule the job, run the changes, and make changes to be imported to the slowly changing dimension tables. It makes my work easier.

I think it has been great for me to use Dagster Labs, and I would not suggest any areas where it could be better because I have not fully utilized Dagster Labs to its fullest potential. It has been great for me.

I do not have any improvements needed for Dagster Labs that I have not mentioned, and there is nothing else I would like to see changed or added.

I have been using Dagster Labs for quite some time, approximately one and a half years.

The specific outcomes I have noticed since using Dagster Labs include saving time and reducing errors. It becomes easy when I run the pipeline in everyday cases instead of someone manually running it. Dagster Labs makes it easy to run scheduled pipelines more often when a large chunk of data comes into the data table or the database. It reduces dependencies on manual work and personnel, and is also efficient in consuming the data that has changed in the SCD data tables.

I have not used Dagster Labs's AI capabilities, so I have no idea about its governance and security.

Regarding Dagster Labs's AI capabilities, I have not used it, so I would not give any metrics on its accuracy or reliability of output because I have not relied more on the AI capabilities of Dagster Labs.

Dagster Labs is deployed in my organization in a private cloud.

This interview was great, and I would not recommend any changes for the future. I gave this review a rating of eight.