What is our primary use case?
My main use case for Astro by Astronomer is orchestrating the day-to-day pipelines, including the ingestions, transformations, and alerting and error handling.
One specific example of a pipeline I run with Astro by Astronomer is our data ingestion pipeline, where we use Astronomer-hosted Apache Airflow running in Docker to orchestrate end-to-end workflows. The DAG starts by triggering the ingestion job from multiple sources such as Google Sheets, Airbyte, and some APIs, and we have a custom script for that. We can initiate all these using Airflow, perform data quality checks, execute the transformations, trigger the transformation notebooks in AWS Glue, run validation tasks, and send notifications if any stage fails midway. We also have retry logic in Airflow, which is very effective.
What is most valuable?
I find that features such as task grouping and the ability to run tasks in parallel are incredibly helpful. Another important feature is XComs, which allows the transfer of data between specific tasks.
In my opinion, one of the best features Astro by Astronomer offers is that it allows us to focus on building data pipelines instead of managing the Airflow infrastructure. This benefit is complemented by the need to manage some Docker instances where Airflow is hosted, including being careful about updates and billing systems and all other networking considerations. Additionally, I have tried the Airflow CLI, which is similar to Astro CLI, allowing me to test the DAGs locally in the Docker environment. The UI has also improved significantly in recent years, making it very clean and user-friendly.
Astro by Astronomer has positively impacted my organization by automating ingestion pipelines, transformations, and error handling, and it has given us relief. Before using Airflow on Astro by Astronomer, we managed dependencies, tracked failures, scheduled, and monitored separately. Now, everything is packaged in Airflow, allowing us to look into a single place. Before, finding the root cause of failures involved checking multiple places, but now it is centralized in Airflow. From a scalability perspective, we have added more ingestions and transformation workflows into the current pipeline, enabling us to scale up easily by dragging and dropping existing tasks.
What needs improvement?
After using Astro by Astronomer for four years, I can say it has evolved significantly, which is positive; however, improving the debugging experience for complex workflows remains a pain point. While simple pipelines are straightforward, complex workflows become difficult to debug with existing tools. End-to-end tracing and dependency visualization would help identify root causes much more effectively in Airflow. Additionally, the learning curve is becoming more complex for beginners, who may find the multitude of features intimidating.
To enhance Astro by Astronomer, it would be beneficial if it could generate documentation automatically, producing visual documentation for pipelines similar to what we have using DBT.
For how long have I used the solution?
I have been using Astro by Astronomer for almost four years, since the start of my career.
What do I think about the stability of the solution?
In my experience, Astro by Astronomer has been stable, especially with the recent version, which has resolved issues that were present in earlier versions.
What do I think about the scalability of the solution?
Astro by Astronomer has handled growth and increased workloads very effectively; during peak hours when multiple pipelines and transformations run, it scales up effectively, allowing us to spin up multiple Docker instances as needed.
Which solution did I use previously and why did I switch?
I have not used any enterprise solutions before Astro by Astronomer; we previously relied on schedules, cron jobs, and custom scripts to orchestrate the pipeline.
How was the initial setup?
I do not have knowledge of the nitty-gritty details regarding whether we purchased Astro by Astronomer through the AWS Marketplace, as that is handled by our DevOps team.
What was our ROI?
Although I do not have visibility on the return on investment, I can share that using Astro by Astronomer has significantly improved our productivity by saving time, helping us troubleshoot, and speeding up the ingestion pipeline.
What's my experience with pricing, setup cost, and licensing?
My experience regarding pricing, setup cost, and licensing is limited, as it is handled by the DevOps team and finance team, and I am not involved.
Which other solutions did I evaluate?
We evaluated other options before choosing Astro by Astronomer, including Dagster among others, but since some team members were already familiar with Airflow, we decided to move forward with it.
What other advice do I have?
I would rate Astro by Astronomer a seven out of ten.
I chose a seven because it has significantly helped us build a scalable orchestration layer, speeding up the ingestion process and onboarding new ingestion sources. While the logging and monitoring features are good, there is always room for improvement, particularly in the documentation and debugging for complex systems, as it becomes challenging in Airflow as complexity increases.
I am not entirely certain about the AI capabilities of Astro by Astronomer, and I am very skeptical about using AI with corporate enterprise data due to concerns about unauthorized access to our data.
I have tried a few prompts with Astro by Astronomer's AI capabilities, and they work effectively; the suggestions are good.
For our public cloud deployments, we primarily use AWS and Azure, with most of our Docker instances hosted in AWS and Azure used for clients and data engineering tasks.
My advice for others looking into using Astro by Astronomer is to give it a try; it is truly useful compared to relying on cron jobs, CLI, and custom scripts since it provides a managed, reliable, and trusted solution used by thousands of developers and data engineers.
Disclosure: My company does not have a business relationship with this vendor other than being a customer.