Azure Data Factory Reviews

As a management consultancy company, we help our clients deploy Azure Data Factory or any other cloud-based solution depending on data integration needs. Regarding how we use Azure Data Factory within our company, we are on the Microsoft Stack, so we use the solution primarily for data warehousing and integration.

The feature I found most helpful in Azure Data Factory is the pipeline feature, including being able to connect to different sources.

I also found running Python codes whenever you need to valuable in Azure Data Factory, especially for certain features of the solution, such as data integrations, aggregations, and manipulations.

Azure Data Factory also has built-in security, which is another valuable feature.

I also like that you get access to the whole Azure suite through Azure Data Factory, so the overall architecture design, defining security and access, role-based access management, etc. It's helpful to have the whole suite when designing applications.

Areas for improvement in Azure Data Factory include connectivity and integration.

When you use integration runtime, whenever there's a failure, the backup process in Azure Data Factory takes time, so this is another area for improvement.

Database support in the solution also has room for improvement because Azure Data Factory only currently supports MS SQL and Postgres. I want to see it supporting other databases.

If you want to connect the solution from on-premises to the cloud, you will have to go with a VPN or a pretty expensive route connection. A VPN connection might not work most of the time because you have to download a client and install it, so an interim solution for secure access from on-premise locations to the cloud is what I want to see in Azure Data Factory.

I've been using Azure Data Factory for about a year now.

Azure Data Factory is very stable, so it's a four out of five for me. In some instances, the solution failed, but I wouldn't wholly blame Azure Data Factory because my company connected to some on-premise databases in some cases. Sometimes, you'll get errors from self-hosted integration, faulty connections, or the on-premise server is down, so my rating for stability is a four.

Scalability-wise, Azure Data Factory is a four out of five because Microsoft is still developing certain tiers, which means you can't upgrade an older skill or tier. In contrast, the more modern, newer tiers could be upgraded easily. Rarely will you get stuck in one platform where you have completely destroyed that container and then fit a new container. Most of the time, Azure Data Factory is pretty easy to scale.

We haven't used Microsoft support directly because whenever we have issues with Azure Data Factory, we can find resolutions through their online documentation.

We're using both Azure Data Factory and SSIS.

We had several in-house solutions, but we moved to Azure Data Factory because it was straightforward. From a deployment standpoint, the solution comes with different services, so we didn't have to worry about separate hardware or infrastructure for networking, security, etc.

The initial setup for Azure Data Factory was easy, so I'd rate the setup a four out of five.

The implementation strategy was looking into what my organization needed overall, then planning and direct deployment. My company first did a test, a dummy version, then a UAT with stakeholders before going into production.

It took about two months to complete the deployment for Azure Data Factory.

An in-house team, the digital data engineering team, deployed Azure Data Factory.

We're still computing the ROI from Azure Data Factory. It's too early to comment on that.

My company is on a monthly subscription for Azure Data Factory, but it's more of a pay-as-you-go model where your monthly invoice depends on how many resources you use.

On a scale of one to five, pricing for Azure Data Factory is a four.

It's just the usage fees my company pays monthly. No support fees because my company didn't need support from Microsoft.

If you're not using core Microsoft products, the cost could be slightly higher, for example, when using a Postgres database versus an MS SQL database.

My company uses Azure Data Factory, SSIS, and for a few other instances, Salesforce.

My company currently has about fifty Azure Data Factory users, but not directly exposed to the solution compared to the developers; for example, members of corporate management and other teams apart from the development team are exposed to Azure Data Factory.

Shortly, there could be about two hundred users of Azure Data Factory within the company.

The developer team working directly on Azure Data Factory comprises ten individuals.

For the maintenance of the solution, my company has two to three staff, but it could reach up to eight or ten for the entire product. It's a mix of engineers and business analysts who handle Azure Data Factory maintenance.

I'd rate Azure Data Factory as eight out of ten.

My company is an end user of Azure.

I can't go into specifics about the use case for Azure Data Factory, but it's for analytics related to an assessment.

What I like best about Azure Data Factory is that it allows you to create pipelines, specifically ETL pipelines.

I also like that Azure Data Factory has connectors and solves most of my company's problems. I can't recall a case where I couldn't use the solution for solving problems.

I'm also happy about the Azure Data Factory dashboard.

A room for improvement in Azure Data Factory is its speed. Parallelization also needs improvement. As for the rest of the features of Azure Data Factory, I'm happy.

I cannot suggest an additional feature I'd like to see in Azure Data Factory in the future because some of the features aren't available internally because the features undergo security evaluation first, and my organization controls which features would become available to users.

I've been using Azure Data Factory for the last two years.

We're happy with the stability of Azure Data Factory.

Azure Data Factory is scalable, with clusters available on demand. There isn't any issue with scaling the solution.

We have an internal support team and the Azure Data Factory support team. We raise tickets and follow up on those tickets, and on a scale of one to five, we'd rate support as four because sometimes there are delays. Otherwise, we are satisfied with Azure Data Factory support.

My company didn't set up Azure Data Factory as the Azure team did it.

We outsourced the implementation of Azure Data Factory directly to the Azure team.

I have no idea how much Azure Data Factory costs.

We're using AWS apart from Azure Data Factory. We're trying out Palantir Foundry as well. They are the leading service providers in the data analytics and ETL world.

I'm familiar with Palantir Foundry, but my company just recently got the Palantir Foundry license, so I'm still not using it, but checking it for shortcomings.

I have experience with Azure Data Factory, too.

I'm unsure of the exact version of Azure Data Factory, but I'm using the latest version or whatever's available on Azure.

I have a vague figure of users of Azure Data Factory, but it's more than one thousand to one thousand five hundred people.

I'd tell people who want to use Azure Data Factory that Microsoft offers excellent courses, ESI (Enterprise Skill Initiatives). You should register and take the courses. Azure Data Factory is a solution I'd recommend to others.

I'd rate Azure Data Factory as nine out of ten because it has a lot of connectors, even custom connectors, for data onboarding. It can also integrate with Spark notebooks and allows my organization to parallelize code. Azure Data Factory also has provisions for Spark and SQL scripts or any scripts, plus the infrastructure is highly scalable, so it's a nine for me.

My organization is a customer of Azure Data Factory.

Our primary use case for the solution is data integration and we deploy it only on Azure.

When we were integrating the Ports product with our internal data warehouse, we had to update all the reports to our internal data warehouse on the Ports system database. However, they were not given access to the database company, and they dump some files or provide you with them. In one case, they were providing files. In another case, they provided some APIs where you need to call in a batch of thousands of records multiple times. It works very well with Azure Data Factory to pull the records, parse them quickly and post them in the database and data warehouse.

We have found the bulk load feature very valuable.

The only challenge with Azure Data Factory is its exception-handling mechanism. When the record fails, it's tough to identify and log.

We have been using the solution for a year and a half and are currently using the latest version.

The solution is scalable and we intend to further increase its usage in the future.

I rate customer service and support an eight out of ten.

Positive

We previously used different solutions.

The initial setup is straightforward.

The implementation was done in-house.

I cannot comment on licensing costs because I was not involved.

I rate the solution a six out of ten. The solution is good but its exception handling and logging mechanisms can be improved. I advice users considering this solution to go for it especially if their integrations are heavy on the data side.

We use this solution to ingest data from one of the source systems from SAP. From the SAP HANA view, we push data to our data pond and ingest it into our data warehouse.

Azure Data Factory didn't bring a lot of good when we were also using Alteryx. Alteryx is user-friendly, while Azure Data Factory uses many resources and has issues with parallel workflows. Alteryx helps you diagnose issues quicker than Azure Data Factory because it's on the cloud and has a cold start debugger.

Azure Data Factory has to wake up whenever you are trying to do testing, and it takes about four to five minutes. It's not always online to do a quick test. For example, if we want to test an Excel file to see if the formatting is correct or why the data-flow or pipeline is failing, we need to wait four to five minutes to get the cold start debugger to run. Compared to Alteryx, Azure Data Factory could be better. Nevertheless, we are using it because we have to.

Initially, when we started using it, we didn't like it because it needed to be more mature and had data-flows, so we used the traditional pipeline. After that, Azure Data Factory introduced the concept of data-flows, and it started to become more mature and look more like Alteryx. Azure Data Factory became more user-friendly when data-flows were introduced.

They introduced the concept of Flowlets, but it has bugs. Flowlets are a reusable component that allows you to create data-flows. We can configure a Flowlet as a reusable pipeline and plug it inside different data-flows, so we don't have to rewrite our code or visual transformation.

If we make any changes in our data-flow, it reverts all our changes to the original state of the Flowlet. It does not retain changes, and we must reconfigure the Flowlets repeatedly. We had these issues three months ago so things might have changed. It works fine whenever we plug it in and configure it in our data-flow, but if we make minor changes to it, the Flowlet needs to be reconfigured again and loses the configuration.

We have used this solution for about a month and a half. It is a cloud-based tool, so there are no versions. It is all deployed on Azure Cloud.

Everything is computed inside the SQL server if we're working with pipelines, so we have to be very careful when designing our solution in Azure Data Factory. Alteryx spoiled us because we never cared how it looked in the backend because all the operations were happening on the Alteryx server. But in Azure Data Factory, they run on the capacity of our data warehouse. So Azure Data Factory cannot run your queries, and it directly sends the query to the instance in the SQL server or data warehouse. So we have to be very careful about how we perform certain operations.

We need to have knowledge of SQL and how to optimize our queries. If we are calling a stored procedure, it joins one table in Alteryx. It is pretty easy, and we just put a joint tool. Suppose we want to do it with a stored procedure in the Azure Data Factory. In that case, we have to be very careful about how we write our code. So that is a challenge for our team because we were not looking into how to optimize their SQL queries when fighting queries from Azure Data Factory to the data warehouse.

In addition, the workflows were running very slow, the performance was bad, and some queries were getting timed out because we have a threshold. So we faced many challenges and had to reeducate ourselves on SQL and query optimization.

In regards to scaling, when Azure Data Factory was introduced as your Databricks, it worked similarly to Hadoop or Spark, and it had some Spark clusters in the back end that could scale it as much as it could, and speed up the performance. So it is scalable, especially with Databricks, because a lot of data-related transformations can be performed.

On my team, there are approximately 20 people who work with Azure Data Factory.

We do not have experience with customer service and support.

It does not require any installation and is more like software as a service. You need to create an instance of Azure Data Factory in Azure and configure some of the connections to your databases. You can connect to your block storages and some authentication is necessary for Azure Data Factory.

The setup is straightforward. It doesn't take much time, and it's on cloud. It requires a few clicks, and you can quickly set it up and grant access to the developer. Then the developer can go to the link and start developing within their browser.

We have a team that takes care of the cloud infrastructure, so we raise a ticket and request infrastructure, and they just exceed it based on the naming convention with the project name.

We have an entire team that takes care of the cloud infrastructure. So we raise a ticket when we need infrastructure, which is executed based on the naming convention for the project name.

The nature of our solution is not based on ROI because we are building solutions for other functions within the same organization. In addition, due to the large size of our organization and the services we provide, the ROI is not something we consistently track. It's something discussed with the management, so I can't comment on it.

The cost is based on usage and the computing resources consumed. However, since Azure Data Factory connects with so many different functionalities that Azure provides, such as Azure functions, Logic apps and others in the Azure Data Factory pipelines, additional costs can be acquired by using other tools.

We did not evaluate other options because this solution was aligned with out current work environment. 

I rate the solution a seven out of ten. The solution is good and constantly improving, but the concept of Flowlets can be reconfigured to retain the changes we make. I advise users considering this solution to thoroughly understand what Azure Data Factory is and evaluate what's available in the market. Secondly, to assess the nature of the use cases and the kind of products they will be building before deciding to choose a solution.

The primary use case of this solution is to extract ETLS, transform and load data, and organize database synchronization.

The most valuable feature of this solution is the data flow, which is the same SQL server in important service, integration services, which is a very robust and powerful tool to transform data.

The solution can be improved by decreasing the warmup time which currently can take up to five minutes.

I have been using the solution for two years.

The solution is extremely stable.

The solution is scalable.

Previously I used AWS Glue and SSIS.

The initial setup is straightforward.

The implementation was completed in-house and is immediate because it is a native cloud tool.

I have seen an ROI with the time saved migrating data for reports.

The solution's fees are based on a pay-per-minute use plus the amount of data required to process. The more data you process the more CPUs and time is required which increases the cost of using this solution.

I give the solution ten out of ten.

The only thing you need to deploy the solution is to click on publish.

The solution includes a feature that increases the number of processors used which makes it very powerful and adds to the scalability.

We have three people using the solution in our organization and one engineer that maintains it.

I recommend to any potential user to factor in the five-minute warm-up time that is required for each execution.

We use Azure Data Factory for data transformation, normalization, bulk uploads, data stores, and other ETL-related tasks.

Azure Data Factory allows us to create data analytic stores in a secure manner, run machine learning on our data, and easily adapt to changing schema.

The most valuable feature of Azure Data Factory is that it has a good combination of flexibility, fine-tuning, automation, and good monitoring.

The documentation could be improved. They require more detailed error reporting, data normalization tools, easier connectivity to other services, more data services, and greater compatibility with other commonly used schemas.

I would like to see a better understanding of other common schemas, as well as a simplification of some of the more complex data normalization and standardization issues.

It would be helpful to have visibility, or better debugging, and see parts of the process as they cycle through, to get a better sense of what is and isn't working.

It's essentially just a black box. There is some monitoring that can be done, but when something goes wrong, even simple fixes are difficult to troubleshoot.

I have been working with Azure Data Factory for a couple of years.

There is only one version.

Overall, I believe the stability has been good, but there have been a couple of occasions when Microsoft's resources needed to be allocated were overburdened, and we had to wait for unacceptable amounts of time to get our slot. It has now happened twice which is not ideal.

There is no limit to scalability.

We only have a few users. One is a data scientist, and the other is a data analyst.

We use it to push up various dashboards and reports, it's a transitional product for transferring, transforming, and transitioning data.

It is extensively used, and we intend to expand our use.

You don't really get that kind of support; it's more about documentation and the community support that is available. I would rate it a three out of five compared to others.

You could call them, and pay for their consulting hours directly, but for the most part, we try to figure it out or look through documentation. 

I think their documentation is lagging because it's not as popular of a tool, there's just not a lot, or as much to fall back on.

Neutral

We had only our own tools, and we switched because you get to leverage all of the work done in a SaaS or platform as a service, or however they classify it. As a result, you get more functionality, faster, for less money.

The initial setup is straightforward.

It is a working tool. You can start using it within an hour and then make changes as needed.

We only need one person to maintain the solution; it doesn't take much to keep it running.

It's not a problem; it's a platform.

We completed the deployment ourselves.

We have seen a return on investment. I can't really share many details, but for us, this becomes something that we sell back to our clients.

You pay based on your workload. Depending on how much data you process through it, the cost could range from a few hundred dollars to tens of thousands of dollars.

Pricing is comparable, it's somewhere in the middle.

There are no additional fees to the standard licensing fee.

We looked at some other tools, such as Databricks, AmazonGlue, and MuleSoft.

We already had most of our infrastructure connected to Azure in some way. So the integration of where our data resided appeared to be simpler and safer.

I believe it would be beneficial if they could find someone experienced in some of the tools that are a part of this, such as Spark, not necessarily Data Factory specifically, but some of those other tools that will be very familiar and have a very quick time for productivity. If you're used to doing things in a different way, it may take some time because there isn't as much documentation and community support as there is for some more popular tools.

I would rate Azure Data Factory a seven out of ten.

For developers that are very accustomed to the Microsoft development studio, it's very easy for them to complete end-to-end data integration.

One of the features that still is in development is data privacy to the cloud side of the SAP integration.

I have been using Azure Data Factory for 3 years. 

I rate the stability a 9 out of 10. 

6 developers are using the solution at present. 

The initial setup is not very straightforward. I rate it a seven out of ten. 

The pricing is a bit on the higher end. 

Overall, I rate the solution an 8 out of 10. 

I'm not sure how much information I can provide regarding to some kind of security of my company, but I can tell you that we were migrating integrations from from platform to the other, and the other platform was error data factory.

For me, it was that there are dedicated connectors for different targets or sources, different data sources. For example, there is direct connector to Salesforce, Oracle Service Cloud, etcetera, and that was really helpful.

It's a good idea to take a Microsoft course. Because they are really helpful when you start from your journey with Data Factory.

I have been using Azure Data Factory for one year. 

The solution doesn't have stability issues. 

It is easy to scale. 

The initial setup is straightforward. Deployment is automatic and takes few minutes. 

Overall, I would rate it an eight out of ten.