Apache Airflow Reviews

Our usual use cases for Apache Airflow involve ingestion and curation. Ingestion involves collecting data across different types of data sources across the world, such as Hive, traditional databases including OLAP or OLTP systems such as PostgreSQL, MySQL, and SQL Server. The sources can also include CSV files, TSV files, and data lakes such as S3, Azure, and GCP. Apache Airflow has integration with different kinds of data sources, and we build pipelines in Apache Airflow which collect the data from a source and store it in a destination. On a very high level, our destination is Hive backed by AWS S3. On top of S3, we have Hive databases. We have a Cloudera data lake and we create our Hive database in the Cloudera CDP Public Cloud stored in AWS S3. We get the data from different sources and dump the data in AWS S3.

There is also another use case called curation where we integrate and write Spark applications which get data from different sources and do multiple things on the data. We can mask the data, filter the data, and query the data all using Spark and put the data somewhere else. Apache Airflow can trigger the Spark applications, run those applications, and complete them. These are the kinds of use cases we have: ingestion and curation.

The positive impact and benefits I have seen from using Apache Airflow on my company is that since it is an open-source tool and not licensed, we can get that tool as open source and integrate and modify it as much as we can. We can modify Apache Airflow into how we can make it better and how we can tune it in the way we want to use it.

The features and capabilities of Apache Airflow that I have found the most valuable and useful so far include its rich integration with many technologies. Since it is open source, it has many contributions from across the globe and very frequent release versions. For instance, if we have one version today, in one or two months we will get a new version which has all the bug fixes of the earlier version. All the bugs will get fixed and all new features will get added in a very short period of time, and there are many contributors that are contributing to Apache Airflow currently. This is making it rich in a wide tech stack and giving releases very frequently in a year.

The positive impact and benefits I have seen from using Apache Airflow on my company is that since it is an open-source tool and not licensed, we can get that tool as open source and integrate and modify it as much as we can. We can modify Apache Airflow into how we can make it better and how we can tune it in the way we want to use it.

I think there could be improvements or enhancements in Apache Airflow in terms of having a better UI experience. The UX can be done a bit better.

The web interface of Apache Airflow has helped me in tracking and troubleshooting since I'm operational and part of the operational side of Apache Airflow. Mostly, I go into the logs of the web server and scheduler to check what's happening in the back-end instead of looking at the front-end. The UI can give errors related to pipelines, but it can be more improved if we get errors related to import errors and scheduling errors. These areas can be improved.

Other than a better UI experience, I would want to see improvements in the scheduler. Sometimes, for user-made mistakes, the scheduler goes down. I experienced this issue and I'm not sure whether it got fixed right now or not. If a user is building a data pipeline in Apache Airflow and a user makes a mistake in their code, that makes the scheduler go down and eventually Apache Airflow goes down. That is not what is expected. If that gets fixed, then it can do wonders.

The stability and reliability of Apache Airflow cannot be praised as a great reliable application, but for small-scale solutions and small-scale use cases, we can always rely on Apache Airflow. However, we need to have expert guidance and expertise in order to use it for production-grade solutions.

The scalability of Apache Airflow is good in terms of execution and we can define our own executors. There is an auto-scaling feature called KEDA, which is Kubernetes event-driven auto-scaling offered by Apache Airflow. Based on workloads, we can scale Apache Airflow execution part and that is how Apache Airflow manages our workloads.

I do not use any technical support for Apache Airflow. Since it is an open-source project, we do not have official support. We can see what bugs are currently being addressed and what fixed versions are released in the official Git repository. We can find how the workflow is happening and what major bugs are getting fixed. We can go through the repository and get an understanding about those things, but we do not have any official channel for support from the official teams.

Negative

Before Apache Airflow, I did not use a different solution for the same use cases. We have used some enterprise solutions, but I cannot reveal them since they are enterprise tools. We thought of Apache Airflow as an open-source tool, which can be a better alternative.

I decided to switch from the previous solution to Apache Airflow because the previous one is an enterprise tool and Apache Airflow is open-source. The license part will get reduced and we can do as much integration as we want and we can modify it according to our situations.

I participated in the initial setup and deployment of Apache Airflow in our use case and I am part of the deployment of Apache Airflow.

The setup process and deployment process for Apache Airflow involves pulling the Helm charts of Apache Airflow and by using values.yaml, it is a Helm-based deployment on Kubernetes. By modifying the values.yaml based on our requirement, we can deploy it using Helm install Apache Airflow.

Apache Airflow has helped me in workflow management by allowing us to write our Python workflows. We do not use it as an event-based scheduling tool. Instead, we schedule the pipelines based on times. We have our own scheduling intervals and our own schedules where we want to run our data pipeline. For instance, we want to run a pipeline on a bi-weekly, daily, or weekly basis. We schedule pipelines according to our data requirement and we design our pipelines in such a way. I would rate this product a six out of ten.

I am currently working on Apache Airflow.

We use Apache Airflow mainly for orchestration purposes to run SQL scripts in a specific order. Sometimes we have to run them based on dependencies, such as when one script runs, we have to run another script. Primarily to orchestrate the big data pipelines that we have, we use Apache Airflow. We also use Apache Airflow for generating reports and stats every specific interval of three to six hours. We use it for alerting, deriving stats at given intervals, and orchestrating our big data pipelines. These are the three use cases for which we are using Apache Airflow.

The best features of Apache Airflow include its Python-based structure, making it easy to start building DAGs. It has all the features one can think of, such as dependency management. There are use cases where one workflow should be completed before another workflow starts, and Apache Airflow offers this capability. It has a clean UI to debug failures and understand the graph flow.

Apache Airflow has event-based scheduling features. They launched data-aware scheduling about a year ago, which can be used effectively. This helped in workflow management by eliminating the need for constant polling. For instance, when workflow A needs to be completed before workflow B can start, instead of using a sensor that polls every five minutes, event-based scheduling automatically triggers workflow B upon completion of workflow A, saving compute resources.

Using DAG for defining workflows in Apache Airflow is beneficial because it is straightforward. There are no loops, which could occur in shell scripts, allowing for a clean structure. When someone who does not know coding looks at the graph, they can easily understand the sequence of steps.

The web interface is an excellent part of Apache Airflow because it helps monitor everything, from all the DAGs to their execution times. It provides complete information from the UI itself. System properties, environment variables, connections, and secrets can all be managed from the UI.

The main benefits from using Apache Airflow include increased monitoring productivity compared to hard-to-debug shell scripts, significant compute savings through event-based scheduling, and increased accessibility as everyone can work and create their own workflows due to its Python-based nature.

The restarting part or rerunning specific tasks in Apache Airflow is somewhat confusing for users because it does not directly say restart or replay. It says clear, and users have to select upstream, downstream, and other options. It requires understanding of the framework, which most support people do not have. This process can be simplified.

The start date in Apache Airflow is also confusing because it is not straightforward. If you want it to start today, you should give tomorrow's date. Some people do not do that calculation correctly, so that can be improved. Either documentation can be improved, or they should develop some workaround. These are two improvements that should be included in the next releases of Apache Airflow.

I have been working with Apache Airflow for around three years.

For orchestration, we previously used Control-M. The key differences between Control-M and Apache Airflow include accessibility. Not everybody can work with Control-M because it requires shell scripting knowledge, while Apache Airflow does not have this limitation. Control-M is straightforward to learn, but there are not many resources. It is also very old, and users cannot understand the workflows just by looking at the nodes in Control-M.

The initial setup and deployment of Apache Airflow are straightforward. If you follow the instructions, it is easy to start the instance. Later, you need to configure connections with different sources such as BigQuery or GCS.

We found Apache Airflow was the best option when looking to move out, and many enterprises are using it.

Performance-wise, Apache Airflow is good. It receives a rating of nine because we cannot see any lags in our workflows, and due to parallelism, the workflows do not affect each other.

I would suggest other organizations choose Apache Airflow because there is nothing better at this point in time which is open-source.

On a scale of one to ten, I rate Apache Airflow as a product and solution a nine. However, I want improvement in the start date and the rerunning of specific tasks. One more feature I would want to see in Apache Airflow is the ability to hold a specific task from running for a given day, as that feature is available in Control-M but not here.

Neutral

I am using Apache Airflow to orchestrate my jobs, projects, and batch ETL jobs. It manages tasks, orchestrates jobs, and helps in handling batch processes effectively, incorporating integrations like DBT and Great Expectations.

Apache Airflow is easy to scale and its UI improves with each release. Reliability is good, and when integrated with Kubernetes, it performs better compared to on-premises environments. It also facilitates the construction of data pipelines by various teams, including those with limited technical knowledge.

Running frequent jobs, such as every minute or five minutes, is not appropriate for Airflow. It is not suitable for real-time ETL tasks. Stream jobs are not its strength.

I have been using Apache Airflow for approximately five to six years.

Apache Airflow is stable and I have not experienced significant issues. I regard it as a reliable solution.

Apache Airflow scales well, especially when deployed in Kubernetes environments. In stand-alone Linux environments, I found it limited. Kubernetes allows for better scalability.

I generally solve problems internally without technical support, however, forums and community resources like Stack Overflow are helpful.

Positive

Before Apache Airflow, I used CronJob. I switched because Apache Airflow is the industry standard, provides better log tracking, and easier task management for teams with varied technical knowledge.

I prefer using the open-source version rather than the enterprise version, which helps manage costs.

I definitely recommend Apache Airflow and would rate it nine out of ten.

We are using Apache Airflow, which has a modular architecture, integrated with our pipeline and with the code generation using Python and the code pipeline. We are running it in Airflow. It is easy for us to use to build the machine learning models, transport data, and manage our infrastructure.

Apache Airflow is easy for us to use to build machine learning models, transport data, and manage our infrastructure. It generates data using the pipeline, and we use Terraform code to run the airflow engine. The solution is easy for us to install and maintain and is very scalable. Apache Airflow is an open-source platform that allows easy integration with AWS, Azure, and Google Cloud Platform.

There is a minor issue with the manual work in Airflow, as everyday activities are managed manually. There is no dashboard for us to check all the Directed Acyclic Graphs (DAGs); a dashboard would help us analyze the work better.

I have been working with Apache Airflow for six months.

I would rate the stability of the solution as ten out of ten. It is very stable.

I rate the scalability at nine out of ten. The solution is very scalable.

Apache Airflow is an open-source platform, which means we solve issues by ourselves. However, there is enough documentation available, and the community support is good.

Neutral

Apache Airflow is easy for us to install. Our reports allow us to spin up the Airflow container and all the servers automatically within five to ten minutes.

Apache Airflow is a community-based platform and is not a licensed product.

I would rate Apache Airflow from one to ten points overall as ten points, which means it is a very good solution.

Apache Airflow is like a freeway. Just as a freeway allows cars to travel quickly and efficiently from one point to another, Apache Airflow allows data engineers to orchestrate their workflows in a similarly efficient way.

There are a lot of scheduling tools in the market, but Apache Airflow has taken over everything. With the help of airflow operators, any task required for day-to-day data engineering work becomes possible. It manages the entire lifecycle of data engineering workflows.

So, for example, let's say you want to connect to multiple sources, extract data, run your pipeline, and trigger your pipeline through other integrated services. You also want to do this at a specific time.

You have a number of operators that can help you with this. For example, you can use the External Sensor operator to take the dependency of workflows. This means that you can wait for one workflow to complete before triggering another workflow. There are also good operators like the Python operator and the Bash operator. These operators allow you to run your scripts without having to change your code. This is great for traditional projects that are already running on batch.

So, let's say you have a scheduled alert called Informatica. You can use Airflow to trigger your VTech scripts through the Informatica jobs. This way, you don't need to use Informatica as a scheduling engine. That's a good way to decouple your data pipelines from Informatica. Airflow is a powerful tool that can help you to automate your workflows and improve your efficiency.

Every feature in Apache Airflow is valuable. The number of operators and features I've used are mainly related to connectivity services and integrated services because I primarily work with GCP. Specifically, I've utilized the BigQuery connectors and operators, as well as Python operators and other runnable operators like Bash. These common operators have been quite useful in my work.

Another thing that stands out is its ease of use for developers familiar with Python. They can simply write their code and set up their environment to run the code through the scheduling engine. It's quite convenient, especially for those in the data engineering field who are already well-versed in Python. They don't need to install any additional tools or perform complex environment setups. It's straightforward for them.

The graphical interface is good because it runs on a DAG (Directed Acyclic Graph).

One improvement could be the inclusion of a plugin with a drag-and-drop feature. This graphical feature would be beneficial when dealing with connectivity and integration services like connecting to BigQuery or other systems. As a first-time user, although the documentation is available, it would be more user-friendly to have a drag-and-drop interface within the portal. Users could simply drag and drop components to create a pseudo-code, making it more flexible and intuitive.

Therefore, I suggest having a drag-and-drop feature for a more user-friendly experience and better code management.  

Moreover, for admins, there should be improved logging capabilities because Apache Airflow does have logging, but it's limited to some database data. It would be better if everything goes into the server where it's hosted. Probably on the interface level. If something goes well for the developers.

I have been using Apache Airflow since 2014. So, it's been over eight years. We currently use the latest version, 2.4.0

Performance-wise, it's good because I've been using two versions - 2.0 and 2.4.

So, it's stable. The version we've been using is much more stable.

It's pretty scalable.

The initial setup is easy if it's on the cloud, you get everything - scalability, usability, so you don't need to worry about storage. It's pretty scalable.

The ROI is very high. Most companies are adopting Apache Airflow, and it can be used for a wide variety of tasks, including pulling data, summarizing tables, and generating reports. Everything can be done in Python and integrated within Apache Airflow. The efficiency and ease of use it offers contribute to its high ROI.

I've been using Apache Airflow, but I haven't directly compared it with other scheduling tools available in the market. This is because each cloud platform has its own built-in scheduling tool. For instance, if we consider Azure, it has a service called Azure Data Factory, which serves as a scheduling engine.

When you compare Apache Airflow with services like Azure Data Factory and AWS, you'll find that Airflow excels in various aspects. However, one area that could be improved is the integration with hosting services. Currently, Airflow can be hosted on different platforms and machines, which offers flexibility but may require some enhancements to streamline integration with certain hosting services.

Since I have been using Apache Airflow for six to seven years, I would confidently rate the solution a solid ten. We help customers re-design and implement projects using Apache Airflow, and approximately 90% of our work revolves around this powerful tool. So, I rate this product a perfect ten.

The primary use case for us is ETL pipelines. We write some pipelines to ingest the data. That is the primary use. And, secondly, we use it to run some scheduling and orchestration. We need to run some automation jobs every day. So, we just write an Airflow task and pipeline that runs every day or every hour or however we need. Those are the two things we use it for.

Since integrating Airflow, we are efficiently able to build pipelines around it in days. If there is a requirement within days or at the end of the week, we can create a pipeline for it.

The declarative language in Python is very powerful as the learning curve is really less. The UI is also very intuitive, and it makes sense. The core features are strong, which are supported by Apache Airflow variables, DAGs, and connections. Connections make it really extendable to plug-ins and custom modules we can write around it.

The UI is a little bit outdated according to modern standards. The UI can be enhanced to support some modern standards. Maybe small things such as dark mode and some proper aesthetics can be implemented.

I have been working with Airflow for the past two years.

We have not faced any performance issues. Our team follows a custom deployment and uses a Kubernetes runner in the backend, so we can scale it as we need. Scalability-wise, we have not faced any issues.

We use Kubernetes on the backend, which allows us to scale it as needed. We have not faced any issues with scalability.

The team prepared comprehensive user guides and FAQs. I do not remember raising any tickets or concerns with tech support.

Positive

I have heard people using Hadoop and some Informatica flows. Informatica flows were pretty rigid, and custom solutioning was more difficult with those.

We used the help of the Astronomer company for setting up Airflow. Setting up the pipelines is straightforward. We have a CI/CD system, and we just write a Python script, and the pipeline is up and running in minutes.

We used the help of the Astronomer company for setting up Airflow.

I might not have the numbers for the investment. Whatever the investment, we can efficiently build pipelines around it in days. If there is a requirement within days or at the end of the week, we can create a pipeline for it. So, the ROI should be good.

If it is a large deployment, it is good to go with a managed approach where someone else would be managing for us. If it is small, we can go on our own by spinning up some Kubernetes clusters and deploying it in the cloud.

I'd rate the solution nine out of ten.

We use the tool to schedule data pipelines. We also use Apache Airflow to orchestrate dbt, another data processing tool. Airflow helps manage dbt processes, which, in our case, load data from our data lake.

To increase efficiency, it's quite simple to add dbt tasks to an Apache Airflow pipeline or orchestration file. With the tool, you can specify dependencies.

I would like to see workflow integration across the servers. 

I have been using the product for two years. 

The solution is stable, but we do have occasional performance issues. These aren't performance problems, but the Apache Airflow cluster sometimes crashes when too many tasks run simultaneously. 

My team has around 11 people using the tool. Each team has a separate server, so we have about 10-20 different Apache Airflow servers. Altogether, I would estimate that around 200 people in our organization use it.

I haven't contacted the support team directly. Our system team does it. 

Apache Airflow provides templates for deployment, which makes it easy. When deploying the tool or using dbt, we usually use Kubernetes. We configure Kubernetes to generate a Docker file that sets up the Kubernetes servers for us. This means that when we deploy, it automatically goes to production. The whole process can be completed in seven weeks. 

I use the tool's open-source version. 

The solution's maintenance involves upgrades. Our system team handles maintenance for us. Their main tasks are upgrading versions and addressing vulnerabilities. It's hard work, but they manage it well. Maintenance takes about two weeks per year for our system team.

I rate the product a seven out of ten and I recommend it to others. 

We're not using it anywhere near its limits, so scalability hasn't been an issue. We're running it on a virtual server, which we can easily upgrade if needed.

We currently have three end users. We plan to scale it up to 60 users.

The initial setup was easy. It took one day to configure and deploy the product.

For now, we only have one person for maintenance, who knows about Airflow manages the updates, and keeps it running smoothly.

In my company, we use Apache Airflow as an orchestrator because we have a lot of business use cases that involve the automation of people's jobs. For example, if someone takes a file and then moves the file from one folder to another, and we have a lot of scripts to do this in PL/SQL or bash pipelines, we decide to move all of this to be orchestrated through one hub application. Instead of having a few things on the database from Oracle while a few things run on local machines, in our company, we wanted this all to be orchestrated through one thing, which is why we chose Apache Airflow.

I like that Apache Airflow is in Python language, making it easy to use and learn. I like Apache Airflow's versatility. Essentially, if you want to do something, there is generally a webhook that you can use with Apache AirFlow, especially if you use solutions from big companies like Google or Microsoft. Many providers are not from Apache since, with Apache Airflow, it is very easy to develop and integrate applications from various developers.

The only thing I would like Apache to do is to introduce an integration of the database from Oracle because it currently supports Postgres primarily in MySQL. Oracle is something that many companies use, like a production database, for which you have to pay since it is not free and offers more extended support. With Apache Airflow, even though it uses Python and Python has modules that include Oracle databases, it'll be safer and more convenient to do it through Apache Airflow and not through Python scripts. I want to see Apache Airflow have more integrations with more production-based databases since it is an area where the product lacks currently.

I have been using Apache Airflow for a year and a half. Our company has a production environment for Apache Airflow since we are familiarizing ourselves with the product currently. I use Apache Airflow Version 2.6.1, which is the most stable one. Regarding Apache Airflow, I don't know if any recent updates were released. I am an end-user of Apache Airflow.

Considering that in my company, we have Apache Airflow deployed on-premises, I rate the stability of the product a ten out of ten since I haven't seen any issues. If Apache Airflow had been deployed on the cloud, then it wouldn't have been very stable.

I rate Apache Airflow's scalability an eight out of ten because you can scale it however you want since it is in Python. There are some limitations in Apache Airflow. Airflow does not process or hold any data, so if you have many scripts running on this tool, then even small variables stored in the database will eventually overflow the database. By design, Airflow is scalable up to a certain point, but I don't imagine anyone will reach that point. The product's scalability has some limitations, so I cannot give it a ten out of ten, though I think it is pretty much a perfect tool.

I use Apache Airflow daily in my company.

My company had directly contacted the technical support team of Apache, but I used Apache's GitHub Pages, along with its documentation, which was very thorough and helpful. Considering the documentation and stuff Apache provides online as support, I would give it ten out of ten, even though I have not personally spoken to Apache's support team.

Positive

The initial setup is simple. Using pip, you type in the version of the Airflow and download it, which is very convenient.

The solution is deployed on an on-premises model.

The solution can be deployed in a couple of minutes.

Almost ten RPA DevOps engineers in my organization use Apache Airflow.

As far as I know, Apache Airflow is a product that is free of licenses, meaning there is no need to buy a license.

Apache Airflow recently introduced a new way of writing scripts, and I quite like it. It's very convenient to write it in taskflow instead of using it with clauses that Python has, so Apache is improving the technology of Airflow as Python improves.

Apache Airflow does require maintenance.

For Apache Airflow, two engineers are involved in the maintenance phase. As we get more servers in our company, we expect the number of people involved in the maintenance phase to increase from two to five.

I recommend the solution to those requiring an orchestrator to manage all of their different scripts. Considering that Apache Airflow is in Python, you don't need to rewrite anything since all you need to do is write a short script in Python that will execute the scripts you already have in bash or PSQL or whatever you want. If someone needs an orchestrator, Apache Airflow is a perfect product.

I rate the overall product a nine out of ten.