Apache Airflow Reviews

Apache Airflow's best feature is its flexibility.

Apache Airflow could be improved with the addition of more frameworks.

I've been using Apache Airflow for four years.

Apache Airflow is stable.

Apache Airflow is scalable.

The initial setup was very easy.

We used an in-house team.

Apache Airflow is open-source and free of charge.

I would rate Apache Airflow eight out of ten.

Currently, I am a lead data scientist. Our primary use cases for Apache Airflow are for all orchestrations, from the basic big data lake to machine learning predictions. It is used for all the MLS processes. It is also used for some ELT, to transform, load, and export all big data from restricted, unrestricted, and all phase processes.

The user experience of Apache Airflow is good. The solution is flexible for all programming languages for all frameworks. I also value that it is used for monitoring. Apache Airflow helps to easily integrate data sources with other products.

Apache Airflow could be improved by integrating some versioning principles. Currently, we have to swap some tags in our flow. It would be interesting if we can check the product and version all of the product at the same time comparing what scripts have changed from last year to this year, or last month to this month.

For example, we have a flow for one project, to version it we need to check it one by one to identify which tags changed and which scripts changed. All of these need to be done manually.

I have been using Apache Airflow for four months.

We have experienced some bugs in Airflow. For example, the solution did not mention all the errors regarding why a process did not work. We had to investigate to try and understand why it was not working.

The solution is easy to scale. We have four people in our organization that use Airflow. One is dedicated to the solution, while the others can use it to adjust the flow of their jobs on their own.

We do not use technical support. We are trained to resolve concerns on our own. If a problem is significant we could call support, however, there is a good developer community that uses Airflow that can help resolve the issue with us.

Positive

Prior to using Airflow, I used Windows SSIS for three years. We made the switch because Windows SSIS uses the drag-and-drop concept, where Airflow requires coding. Also, Windows is orientated to Microsoft products and is not very flexible.

I am a technician, so the initial setup is instinctive. Without experience, it would not be as simple. Experience with configurations with parameters is required. The documentation is good, however, it does not mention some features explicitly requiring some research. 

I would rate the ease of implementation a three out of five.

We have dedicated machine learning ops, so we manage all product deployment ourselves. The deployment takes about four days, including two days of administration. 

Apache Airflow requires maintenance. It is very important to maintain all the source codes and all the data. We are looking for a platform that would facilitate the maintenance of the project.

We use a community edition of Apache Airflow. It is open-source and free. 

Anyone considering Apache Airflow should make sure that they have a good team with experience, including some administration. A strong background will help to understand and exploit the strengths of the platform.

I would rate this solution a nine out of 10 overall.

My team works on commerce services. We use Airflow to synchronize user information or product information from other services. We use the tool for automating data pipelines. We store user history about API calls and show it on a statistics page, like daily or real-time statistics. We use the solution to aggregate API user's data.

Kubernetes from the batch application is the most useful to my team. It uses Python. It is simple. There are not many learning costs. We're using the scheduler. We don't need to care about the batch job every day. We just need to notice when the alerts are firing. It is convenient for us. The product supports many other services, like Kubernetes. I saw some custom applications and programs. The solution integrates very well with other products.

The documents do not precisely define the function of the operators. I had to do some experiments to understand the function of the operators. The documentation must be improved. Some parts of the documentation do not precisely explain the parameters and functions. We often need to do experiments to understand how they work.

I have been using the solution for one and a half years.

I rate the tool’s stability a nine out of ten.

I rate the tool’s scalability a six or seven out of ten. We haven’t horizontally scaled the solution. At least 20% of the teams in my organization are using Airflow to do some batch jobs. There are around 300 users.

I rate the ease of setup an eight out of ten. The product is deployed on the cloud. We release Airflow on Kubernetes. The deployment takes less than five minutes. We use a deployment tool made by our company to deploy the solution.

I am also using Apache Kafka.

I will recommend the product to others. The UI is very simple and easy to learn. There are a lot of users of the product. We can find information easily on Google. Overall, I rate the tool an eight out of ten.

Apache Airflow is useful for workflow automation, making it capable of automating pipelines, data pipelines, and data warehouse processes. I don't have a strong need for Apache Airflow because I do everything with a dbt or data build tool since it has its own integrated workflow process.

I use Fivetran to synchronize my data. I don't need to do any automation on that and don't have any need for workflow automation. I have everything I need.

We were experimenting with the solution. We never reached the point where we would deploy the solution in the production capacity.

The problem with Apache Airflow is that it is an open-source tool. You have to build it into a Kubernetes container, which is not easy to maintain, and I find it to be very clunky.

Additionally, there is room for improvement with DAGs. I had a very hard time building DAGs in Apache Airflow. I decided to use Astronomer, which is on top of Apache Airflow and is supposed to make your life easier. The best part of the solution is the third-party add-on which is Astronomer.

It would be a very nice tool if it could have been an entirely cloud-based solution. Apache Airflow is not so nice when you have a hybrid setup, such as half is on-premises and half of it is on a cloud environment. It should integrate better with the outside world.

I have been using Apache Airflow for a couple of months.

I have no opinion on the solution's stability. The solution did not get to a production capacity. I couldn't even do file processing with Apache Airflow. None of the engineers could actually help me set up Apache Airflow. I had to give up on the product. Just buy a product that works, and you will be done with it.

The initial setup was complex to deploy on the cloud. Installing the software is very difficult. The documentation is very bad. There is no installer where you can press a button, and it does everything for you. One may need a couple of engineers to install the solution, which is an issue with open-source tools. Price-wise, the software falls on the cheaper side. With Apache Airflow, one may spend much more on engineers.

The solution is deployed purely on the cloud.

I didn't experience any ROI using the solution. I could do everything without Apache Airflow since it would have been just a money pit.

I suggest others not use Apache Airflow. If you use Apache Airflow, you will waste your time unless you have a bunch of engineers who already know about the solution.

If you cannot write a DAG within two hours of starting the process, then forget about the tool, and it would be better if you tried to find something else.

Overall, if the tool was working properly, it would be very good, but unfortunately, it is not.

Overall, I rate the solution a five out of ten.

We use this solution to monitor BD tasks.

The solution is quite configurable so it is easy to code within a configuration kind of environment. 

The ease of learning and using the solution is quite good. The learning curve is low so new users can learn in a short period of time in comparison to other products. 

The following should be improved:

• Dashboards

• Security

• Airflow web UI

• Telemetry for logging, monitoring, and alerting purposes

• Documentation 
  

I have used the solution for six months. 

The solution is 99% stable. We have a few glitches here and there but have been able to fix them. 

The solution is quite scalable. You can grow in terms of users and environment. You can grow to multi-server applications. You can use the solution on desktops, mobile, or other devices. 

We have an internal tech support team so have not needed support from the vendor. 

The setup is straightforward. The time for deployment depends on the environment and user base.

We implement the solution in-house. We have one implementation with 60 users and another with 75 users. 

We have a tech support team that consists of ten engineers who support implementations. They follow up on issues that might arise during the process automation or implementation of the workflow itself. 

For example, our tech support team will resolve a workflow that gets stuck during the MDM workflow engine. The tech team has the knowledge base to resolve any of these issues. 

The solution is open source.

I do not have exposure to use cases for large organizations with a huge user environment, so I cannot speak to the solution's effectiveness in these scenarios. 

I rate the solution an eight out of ten. 

I use this solution for scheduling purposes. We have our own Python framework to run jobs, do the extractions, and for transformation loading.

We have 20 people who are using Airflow. It's being used on a daily basis. We don't have any plans to increase usage because we have low data sets.

The solution is deployed on cloud. The cloud provider is Azure.

Everything is in the Python framework now. I would like to see some no-code capabilities and drag and drop abilities in Airflow.

We're expecting a few more improvements in the log generator. Currently, it's very clumsy.

I have used Apache Airflow for three years.

It's stable.

It's scalable. So far, we haven't needed more scalability because it's totally controlled by administrators.

The only difference between Apache Airflow and BPM software is the pricing.

Setup is about medium difficulty. You need to have some prior knowledge and experience with docker containers and AKS.

It's open-source.

I would rate this solution as seven out of ten.

Our primary use case for this solution is scheduling task rates. We capture the data from the SQL Server location and migrate it to the central data warehouse.

The best feature is the customization that can be done using Python. For example, there are use cases where we have to tweak the algorithm and with Apache Script Rate, we have extra functionality that helps to change the underlying process. We can define our algorithms and processes using Python.

The solution could be improved by simplifying the integration process and providing access to its support team to guide integration.

We have been using this solution for two months and it is deployed on-premises.

The solution is stable but primarily depends on the support team and how they manage it.

Apache Airflow is scalable. Approximately 20 people use this solution on my team.

We haven't had any experience with customer service and support.

Previously, we were using SQL server integration tools and integration service SSIS packages. We had project orders and wanted to migrate everything as it was an open source rate and no license was required. We switched to Apache Flow because we are trying to migrate all the projects developed in SSIS using Python.

The initial setup was straightforward. However, if a script is written, it takes four to five minutes to set up.

Apache Airflow is open source, so I cannot comment on licensing costs.

We chose this solution because it was suitable for our business needs.

I rate this solution a seven out of ten. My advice to new users is to have good proficiency with Python language. The solution is good but can be improved by simplifying its integration process.

I'm a data engineer. In the past, I used Airflow for building data pipelines and to populate data warehouses. With my current company, it's a data product or datasets that we sell to biopharma companies.

We are using those pipelines to generate those datasets.

I like the UI rework, it's much easier.

I use XCom for derived variables that need to pass between tasks. I don't really tend to use it for passing data, but only for a derived variable. For example, I don't have to re-query something every time, with one-task uses. I use the JSON comp for overwriting certain parameters.

In our use cases, some of the inputs of the dataset are files that we pulled out of S3. Sometimes they need to re-do those files, but we don't need to change any logic, we just need to redo the bills. Rather than redeploying the code to point to a new S3 bucket, we overwrite it to point to a different S3 key.

I have read that there are many different workflow pipelining tools in the biotech space, such as Snakemake and Nextflow.

There is also a CWL plugin that we may look into at some point. 

Eventually, we might have a use case where a researcher has a pipeline they run locally, and then we want to convert that to a DAG. 

The CWL-Airflow plugin would be useful for that. This might be something to look into later. But that would be like months, or maybe a year from now.

I am using a Celery Executor and I find that it crashes and I can't see any logs. I can only assume that it's a memory issue and have to blindly restart until eventually, it starts up again.

One of the use cases is triggered by input rather than a batch process. For example, we receive a batch of data, it goes through tasks one, two, and three, and a new batch comes in, each subsequent task should be operating on just that data from the prior task.

I am used to working on it as the output gets written to a table and then the next task selects all from that upstream table. It could be coded where you are only writing the data for that portion of the task. It could handle state machines and state changes as opposed to the batch proxy.

I would like to see it more friendly for other use cases.

In my current company, I just introduced it within the last couple of months. But I've used it at my prior two jobs as well.

We are using Version 2.0.1.

We are using the open-source version of Apache Airflow.

I usually create my own custom operators every time. We upgraded to 2.0, but I am not using any of the new features. 

I haven't yet used DAG of DAGs or the new way of using Python functions in the Python operator yet. But we might use DAG of DAGs eventually.

I Love this solution and I would rate it a nine out of ten.