Apache Airflow Reviews

We use Apache Airflow for the orchestration of data pipelines.

Since it's widely adopted by the community, Apache Airflow is a user-friendly solution.

Apache Airflow should have better integration with cloud platforms.

I have been using Apache Airflow for a couple of years.

Apache Airflow is not a stable solution.

Around ten people are using the solution in our organization.

The solution's initial setup is difficult and should be done by an experienced person.

Apache Airflow is a cheap solution.

The solution is deployed on the cloud in our organization. Before choosing Apache Airflow, users should try cloud-native services first.

Overall, I rate the solution a seven out of ten.

We utilize Apache Airflow for two primary purposes. Firstly, it serves as the tool for ingesting data from the source system application into our data warehouse. Secondly, it plays a crucial role in our ETL pipeline. After extracting data, it facilitates the transformation process and subsequently loads the transformed data into the designated target tables.

One of its most valuable features is the graphical user interface, providing a visual representation of the pipeline status, successes, failures, and informative developer messages. This graphical interface greatly enhances the user experience by offering clear insights into the pipeline's status.

Enhancements become necessary when scaling it up from a few thousand workflows to a more extensive scale of five thousand or ten thousand workflows. At this point, resource management and threading, become critical aspects. This involves optimizing the utilization of resources and threading within the Kubernetes VM ecosystem.

I have been working with it for five years.

I would rate its stability capabilities nine out of ten.

While it operates smoothly with up to fifteen hundred pipelines, scaling beyond that becomes challenging. The performance tends to drop when dealing with five thousand pipelines or more, leading to the rating of five out of ten.

I would rate the customer service and support nine out of ten.

Positive

The initial setup is straightforward. I would rate it nine out of ten.

The deployment process requires approximately four hours, and the level of involvement from individuals depends on the quantity of pipelines intended for deployment.

The cost is quite affordable. I would rate it two out of ten.

If you have around two thousand pipelines to execute daily within an eight to nine-hour window, Apache Airflow proves to be an excellent solution. I would rate it nine out of ten.

Apache Airflow is utilized for automating data engineering tasks. When creating a sequence of tasks, Airflow can assist in automating them.

The most valuable feature of Apache Airflow is creating and scheduling jobs. Additionally, the reattempt at failed jobs is useful.

We have faced scenarios where Apache Airflow becomes non-responsive, leading to job failures. To resolve such situations, we had to manually reboot Apache Airflow since it doesn't provide an option to restart within the application. This necessitated modifying some configurations to initiate a restart of all Apache Airflow components. Although Apache Airflow is generally dependable, it may occasionally encounter glitches that can disrupt production flows and batches.

I have been using Apache Airflow for approximately three years.

We experienced some glitches using the solution with some errors.

I rate the stability of Apache Airflow a five out of ten.

Apache Airflow is scalable because it is within Amazon AWS.

I rate the scalability of Apache Airflow an eight out of ten.

The technical support is good, they are able to debug issues.

The initial setup of Apache Airflow was simple because it was all managed by Amazon AWS. The process took a few minutes.

The solution is free if you use Amazon AWS.

I would recommend this solution for projects even though there have been glitches. Once the solution has become stable it would be ideal for critical projects.

I rate Apache Airflow a seven out of ten.

This is great software to build data pipelines. However, we had many glitches that were causing some problems in production.

We use the product for scheduling and defining workflows. It helps us extensively to manage complex workflows within Cloudera's ecosystem, particularly for handling and processing data.

The solution has been beneficial in automating and managing our data workflows efficiently. It has integrated well with our Cloudera environment, enabling us to handle complex workflows with greater ease and reliability.

The solution's most valuable feature is its ability to run workflows without saving changes. It allows us to execute tasks without permanently altering our configurations, which is useful for temporary adjustments and testing.

One area for improvement would be to address specific functionalities removed in recent updates that were previously useful for our operations.

Additional features that could enhance the product include more flexibility in parameterization and improved tools for managing and debugging workflows.

I have been working with Airflow for approximately a year and a half, focusing on the current version for the past eight months.

The product has been stable in our environment.

The product is scalable. 

The technical support team has been responsive and helpful. They addressed issues related to removed functionalities and ensured critical features were restored in subsequent updates.

Positive

We previously used Hortonworks but switched to Cloudera CDP. We also used other Cloudera tools but found Airflow to be a better fit for our current needs due to its capabilities in workflow management.

The initial setup was complex due to the integration with various data sources and configuration requirements, but once properly set up, it has proven effective.

The implementation was carried out with guidance from Cloudera's support team, who provided valuable assistance in configuring the solution to meet our requirements.

We evaluated other data workflow solutions but found Airflow the most suitable due to its integration with Cloudera and comprehensive feature set for managing complex workflows.

Airflow integrates well with Cloudera and effectively supports complex operations. However, users should be aware of changes in functionality between versions and plan accordingly.

Overall, I rate it a nine out of ten. 

We use Apache Airflow for the automation and orchestration of model deployment, training, and feature engineering steps. It is a model lifecycle management tool.

We have an integration with Apache Airflow in our portal for messaging. We use group and transformation data from Redshift to Tesco, and then create a call flow to the router. This is a source of data leakage, such as data engineering and machine learning, especially in a HIPAA environment. We need to check the evolution steps in the pipeline. In production, we only have two cases. Sometimes, we need customer data not in the database, which we get from object storage. The call flow from Redshift to Tesco involves transforming the data and then generating it with the router or Kibana router for the policy. The data is then transformed and sent to the dashboard or data warehouse.

Airflow is a pipeline for transferring code by clients, but for experimental model experiments, Apache Airflow does not have any solution. There is a need for more features on experimental evolution steps.

I have been using Apache Airflow for one and a half years.

The product is stable. I rate the solution’s stability an eight out of ten.

20 users are using this solution in our organization. I rate the solution’s scalability an eight out of ten.

The initial setup is not complex and can be done by two people. However, open-source prime solutions have some difficulties. We can schedule Apache Airflow on Kubernetes. Space limitations and installation issues may arise, as we do not have full control over Kubernetes cluster resources, and our administration is limited. I rate the initial setup a six out of ten, where one is difficult, and ten is easy.

I recommend Apache Airflow because it is still profitable and can be used with multiple systems and servers, Kubernetes systems, and dashboard systems. You can use it to get social media and other data, but it can be expensive. Overall, I rate the solution a nine out of ten.

We were using Apache Airflow for our orchestration needs. We used it for all the jobs that we had created in Databricks, Fivetran, or dbt. These were the three primary tools that we were using. There were a few others, but these were the three primary tools. So, Apache Airflow was for the job orchestration and connecting them to each other for building our entire data pipeline. We were also using Apache Airflow for dbt CI/CD purposes.

The most valuable feature is that it's the most popular data orchestration tool in the market right now. It connects to everything you need.

It's open-source. You have a lot of documentation and a lot of people helping out. It has large communities, so if you need something or you want to ask something, you can. Often, someone else would have already asked that question, and they would have already got the answer, and you can just look it up.

Development on Apache Airflow is really fast, and it's easy to use with the newer updates. Everything is in Python, so it's not hard to understand. They also have a graphical view, so if you are not a programmer and you are just an administrator, you can easily track everything and see if everything is working or not. For notifications, it can connect with different messaging tools such as Slack and Teams, as well as with webhooks. It's very easy to use, and it has a lot of features that you would expect from any of the data orchestration tools.

Programmatically, it's very good, and it doesn't have any competitors, but you cannot develop anything in Airflow UI. You need to develop everything within the program. In the market, other tools have come up recently as competitors to Airflow, and they also give graphical programming options, whereas Airflow doesn't provide that feature currently. All the DAGs you want to build need to be coded in Python. It doesn't provide features for graphical programming. You cannot drag and drop something, build a pipeline out of that, or orchestrate that with a drag and drop. They have a graphical feature but only for administration purposes, not for development. They don't have a UI for development.

It doesn't support the Windows system. That's a big drawback because a lot of people are using Windows. 

I used Apache Airflow on my previous project. We had planned to use it in our current project, but due to time issues, we were not able to deploy it. In my previous project, I used it for around eight or nine months.

It's a very stable product.

It's highly scalable. You can scale it as much as you want. It depends on the size, and you need to scale up your instance. We had over 3,000 DAGs in our previous project, and we didn't face any issue with even 8 GB memory in our EC2 instance. If you have a lot of DAGs, you might need to scale up, but it's quite lightweight, so you don't need to worry much about that.

It's open source. It was my first project, and I had a few doubts, but everything I needed was available on the internet, so I never had to contact their support. I might have been able to post my questions on their GitHub, but I didn't need that. Airflow has a very large community, so any questions you ask get answered there.

Its setup wasn't done by us. It was done by the Astronomer team on Azure Community Services. So, it was deployed and set up on Azure Community Service. Everything was taken care of by the Astronomer team.

Apache Airflow has two large and popular distributors. There might be others, but the two popular ones are Bitnami and Astronomer. For us, everything was set up by Astronomer.

It's open source. You can install it locally on your own system. If you are deploying it in the production system, you normally deploy it on some cloud, such as EC2 service, which would have some cost. If you are setting up a Docker container or something for Apache Airflow yourself, which is quite easy, you can do pretty much everything online. I have set it up on my local system, and It doesn't take a long time. You can do customization for your project such as selecting different repository databases or selecting different cellular or web services, which is good.

If you are going with a service provider such as Astronomer or Bitnami, they will charge you because they are a distributor of Airflow. They have some of their own features and their own support. They will charge you if you are going with them.

If you are on a Mac or Linux system, it's very easy to install. You can just go to the Apache website to install it, and you can start working, but Apache Airflow doesn't support Windows Exe installation, so if you have some knowledge of Docker containers for WSL, it'll be useful.

Other than that, Astronomer has an instructor called Marc Lamberti who is very popular in the Airflow community. He has YouTube videos. In five minutes, he can teach you how to set up Airflow or what DAGs are. He has five or six videos, and he gets into the details with his videos. So, if you have no idea about Apache Airflow and you don't want to go through all the documentation, you can start with those videos, but if you have a Mac or Linux system, you can directly install it on your system.

I'd rate it a seven out of ten because it doesn't support Windows, and it doesn't support graphical designing, so we cannot create DAGs in the UI. We can administer and look at DAGs through the UI, but we cannot create DAGs through the UI. Other orchestration tools that are available in the market provide that feature.

We use Apache Airflow to send our data to a third-party system.

We are already on Python. Since Apache works very well on Python, we can manage everything and create pipelines there.

Adding more automated components in Apache Airflow for basic things like exporting the data would be helpful. Apache Airflow is not that easy to use, but we have gotten used to it.

I have been using Apache Airflow for three years.

Apache Airflow is a stable solution.

Apache Airflow is not a scalable solution for our use cases. We have a very huge list of use cases. Over 10 developers use Apache Airflow in our organization.

Apache Airflow's technical support team is good and provides assistance almost 90% of the time.

Apache Airflow's initial setup is easy. It's not that difficult, but it has a learning curve.

Apache Airflow is a cheap solution.

Depending on your use case, if you are looking for a quick solution to work on and know Python, you should go ahead with Apache Airflow.

Apache Airflow is a good enough tool for managing data pipelines. However, the solution is not up to the mark as you scale up and go at the higher performance. Apache Airflow has introduced the DAG connector for managing data pipelines.

Overall, I rate Apache Airflow an eight out of ten.

We use this solution to monitor BD tasks.

The solution is quite configurable so it is easy to code within a configuration kind of environment. 

The ease of learning and using the solution is quite good. The learning curve is low so new users can learn in a short period of time in comparison to other products. 

The following should be improved:

• Dashboards

• Security

• Airflow web UI

• Telemetry for logging, monitoring, and alerting purposes

• Documentation 
  

I have used the solution for six months. 

The solution is 99% stable. We have a few glitches here and there but have been able to fix them. 

The solution is quite scalable. You can grow in terms of users and environment. You can grow to multi-server applications. You can use the solution on desktops, mobile, or other devices. 

We have an internal tech support team so have not needed support from the vendor. 

The setup is straightforward. The time for deployment depends on the environment and user base.

We implement the solution in-house. We have one implementation with 60 users and another with 75 users. 

We have a tech support team that consists of ten engineers who support implementations. They follow up on issues that might arise during the process automation or implementation of the workflow itself. 

For example, our tech support team will resolve a workflow that gets stuck during the MDM workflow engine. The tech team has the knowledge base to resolve any of these issues. 

The solution is open source.

I do not have exposure to use cases for large organizations with a huge user environment, so I cannot speak to the solution's effectiveness in these scenarios. 

I rate the solution an eight out of ten.