What is our primary use case?
I have been using Astro by Astronomer CLI as a local development environment for about a year. During this period, it has been the main tool I use to develop, test, and validate my Apache Airflow projects before deploying them to production. In production, I use Apache Airflow together with Astronomer Cosmos to orchestrate dbt pipelines.
My main use case for Astro by Astronomer is using the CLI as a local development environment for Apache Airflow projects. I use it to develop, test, and validate DAGs before deploying them to production, ensuring everything works correctly. In addition, in my day-to-day work, I use Astronomer Cosmos to integrate and orchestrate dbt pipelines with Apache Airflow, running data transformations on Amazon Athena with Apache Iceberg tables.
In my day-to-day work, I use Astro by Astronomer CLI to develop and test new Apache Airflow DAGs locally before publishing them to the production environment. For example, when I need to create a new data pipeline with dbt using Astronomer Cosmos, I first validate all the orchestration locally with Astro by Astronomer CLI, check that the dependencies, tasks, and integrations are working correctly, and only then do I deploy it to the production environment. This reduces errors and makes the development process much faster and more reliable.
What is most valuable?
In addition to using Astro by Astronomer, I also use Astronomer Cosmos to integrate dbt with Apache Airflow. This makes creating and maintaining DAGs much easier because Cosmos automatically generates the dbt tasks, respecting the dependencies between models, making orchestration simpler and more organized. In our environment, dbt runs happen in containers on Amazon ECS, using AWS Fargate. This model allows us to start resources only during pipeline execution and shut them down afterward, avoiding having dedicated servers running all the time. As a result, we were able to reduce infrastructure costs, maintain a scalable environment, and execute transformations efficiently, especially for workloads that do not need to be active continuously.
The main value of Astronomer Cosmos is simplifying the development and operation of pipelines with Apache Airflow. Astro by Astronomer CLI offers a very consistent local development experience, while Astronomer Cosmos makes it easier to integrate dbt with Airflow by automatically generating the DAGs and respecting the dependencies between models. In addition, this approach integrates very well with Amazon ECS and AWS Fargate to run the dbt jobs. This allows us to scale on demand and pay only for the resources used during pipeline execution, reducing infrastructure costs without sacrificing reliability and ease of maintenance.
A differentiator I consider very important is that Astronomer Cosmos is very flexible and integrates easily with other modern data engineering technologies. In my case, the combination of Astro by Astronomer CLI for local development, Astronomer Cosmos to orchestrate dbt projects, and Amazon ECS with AWS Fargate to run the jobs has brought a very efficient workflow. Besides facilitating the development and maintenance of pipelines, this architecture allowed us to reduce infrastructure costs, because the dbt containers are started only when needed and shut down at the end of execution. This offers a good combination of productivity, scalability, and operational efficiency, especially in environments that run on-demand workloads.
The positive impact of Astro by Astronomer has mainly been on team productivity and the standardization of development. Astro by Astronomer CLI made it much simpler to create and validate Apache Airflow pipelines in a consistent local environment, reducing configuration issues and speeding up testing before deployment. In addition, with Astronomer Cosmos, we were able to integrate dbt with Airflow in a much more organized way, automating the creation of DAGs.
A practical example is that we were able to significantly reduce the time needed to develop and validate new pipelines because all developers work in the same local environment using Astro by Astronomer CLI. This reduced configuration issues and decreased rework during deployments. Another important result was the reduction in costs for running dbt. Instead of keeping dedicated infrastructure running all the time, we started running the jobs in containers on Amazon ECS using AWS Fargate, which are started only when needed and shut down at the end of execution. Although I cannot share exact numbers for confidentiality reasons, we observed a relevant infrastructure cost saving, as well as a more scalable and simpler environment to operate. We also noticed a reduction in failures related to the integration between Airflow and dbt, thanks to the use of Astronomer Cosmos, which automates the creation of DAGs and ensures that dependencies between models are respected.
What needs improvement?
I believe one area for improvement for Astro by Astronomer would be to further expand the documentation and examples for more advanced scenarios, especially involving integrations with AWS, Amazon ECS, AWS Fargate, and Astronomer Cosmos. Although the documentation is good, some more complex use cases require additional research or testing until you find the best approach. It would also be interesting to offer more templates and ready-made best practices for modern architectures with dbt, Airflow, and Kubernetes, making it easier for teams that are just getting started. This would reduce the learning curve and further speed up the implementation of production environments.
The main point would be to provide more content and reference architectures for large-scale corporate environments, especially involving Airflow, dbt, Kubernetes, and AWS. This would help teams adopt best practices more quickly and reduce the time spent on architecture decisions. Otherwise, I consider the experience very positive, and Astro by Astronomer platform meets the needs of development and orchestration of data pipelines very well.
For how long have I used the solution?
I have been working in technology for about twelve years, and specifically as a data engineer for approximately five years. During this time, I have worked at different companies and on different projects, always focused on data platforms, architecture, data processing, and cloud solutions. I currently work on the evolution and support of a data platform using technologies like Apache Airflow, dbt, AWS, and Astronomer Cosmos.
What other advice do I have?
The interview was good and well structured. The questions covered the main aspects of the tool, such as user experience, benefits, improvement points, and business impact. If I could suggest some improvements, I would avoid very similar questions. At times, there was repetition, such as asking if I wanted to add something right after practically every answer. I would give more room for technical examples. Since the audience is in technology, questions about architecture, integrations, implementation challenges, and best practices would generate richer reviews. I would try to reduce administrative questions at the end—name, company, reference, contact—putting them in a form instead of asking all of them by voice. I would allow slightly more natural answers, without interrupting the interviewee between one question and the next. Overall, I found the experience positive, objective, and easy to follow.
In the flow, Cosmos unites data and ideas, and simplicity grows. I would rate this experience a nine out of ten.
Which deployment model are you using for this solution?
Public Cloud
If public cloud, private cloud, or hybrid cloud, which cloud provider do you use?
Amazon Web Services (AWS)
Disclosure: My company does not have a business relationship with this vendor other than being a customer.