Azure Data Factory Reviews

We mainly use it to migrate data from on-premises sources to the cloud, such as Oracle and Cisco servers. It's a good solution for integrations within the Azure environment, and it connects well with other Azure data products. However, for external configurations, we use Informatica Cloud or Informatica Data Accelerator (IDA).

For automation, we primarily rely on Snowflake and Informatica. Our strategy is not to depend on a single tool. When it's strictly on-premises to cloud, we use ADF. 

Otherwise, Informatica is more mature and integrates well with various third-party products. We also use Snowflake copy commands to load data into Snowflake. Azure Data Factory doesn't fully meet your automation requirements.

We use Informatica for pipelines that were originally in SSIS, but for new pipelines and ETL processes, we choose either Informatica or Snowflake scripts.

We have multiple banking applications running on SSIS pipelines. We're in the process of upgrading them to a hybrid cloud architecture. For that, we use Azure Data Factory to move data from on-premises to the cloud – mainly for back-end database operations and ETL transformations.

We primarily use it to load data from an on-premises SQL Server to either Blob storage or an Azure SQL data warehouse. For other integrations, especially those outside of Azure, we tend to use Informatica Cloud Services (ICS).

For structured data loading, we use it However, we use Informatica for unstructured or semi-structured data. We also use Snowflake for ETL processes and sometimes for streaming. 

In my opinion, ADF isn't as suitable for streaming – for streaming, Snowflake streamlets or Informatica structured streaming are more reliable. ADF works well for batch processing, though.

I like its integration with SQL pools, its ability to work with Databricks, its pipelines, and the serverless architecture are the most effective features.

There is room for improvement primarily in its streaming capabilities. For structured streaming and machine learning model implementation within an ETL process, it lags behind tools like Informatica. 

Snowflake is also more efficient for loading data into Snowflake, whether from Blob storage or AWS. From our experience, ADF is mainly useful for batch processing. I'm not sure how its streaming capabilities compare to others for industry-wide use cases.

I've been using Azure Data Factory for the past two years.

I would rate the stability an eight out of ten.

When it comes to scalability and handling large datasets, it works well for datasets within the Azure environment because it's tightly integrated. 

However, for third-party integrations – things like SAP HANA, MongoDB, or handling semi-structured and unstructured data from logs – it's not as reliable. ADF excels with tight Azure cloud integration.

There are around seven end-users. Across the enterprise, Informatica is our main tool because it includes data governance (DG/DM), data quality (DQ), data cataloging, API integration, and streaming capabilities – like IDM and Informatica Cloud Services (ICS) as a SaaS platform hosted on either AWS or Azure. 

We are transitioning SSIS pipelines to ADF, but otherwise, Informatica is our central tool.

I would rate the scalability an eight out of ten.

The customer service and support are good. 

Positive

I primarily work with Informatica, Azure, and Snowflake for data pipeline tasks.

We've used Talend, SAP BODS, and AWS Glue. Oracle Data Integrator (ODI).

Now, we primarily use products within the Azure ecosystem, like Snowflake and Informatica Cloud Services (ICS). Specifically, we use Informatica Data Management Cloud (IDMC).

Our goal is to migrate from on-premises to IDMC, and we've been using ADF for the past two years to convert SSIS scripts.

That's likely handled by your cloud infrastructure team.

We have an internal cloud infrastructure team. Our data engineering team is quite large, around 30 people, since multiple projects are happening. Within my immediate team, there are seven people.

The maintenance is primarily handled by our database administrators (DBAs). Our involvement is mainly focused on building data pipelines and the ETL process as part of the data engineering team.

We sometimes need to look into performance issues due to it being in the cloud. However, Snowflake is much easier to maintain – that's all handled by Snowflake itself. Overall, Informatica and Snowflake are less demanding in terms of maintenance compared to ADF.

For our use case, it is not expensive. We take into the picture everything: resources, learning curve, and maintenance. 

I would rate the price a six out of ten. It is moderate pricing. 

I would definitely recommend using it It's a good tool. Because there isn't a comparable native Azure product for cloud-based integrations, using ADF is often necessary. If working with multiple clouds (Azure, AWS, etc.), we end up using tools like Informatica or Snowflake.

Overall, I would rate the solution a seven out of ten. It has some limitations, especially with streaming data.

Compared to Talend, Snowflake, or Informatica, which have rich screen-based GUIs, ADF's visual capabilities are weaker. There's a steeper learning curve, as you need to understand its technical UIs and data flows. 

However, with Microsoft's acquisition of Power BI, I suspect they might integrate these capabilities as 'Microsoft Fabric' in the future.

The primary use case of this solution is to extract ETLS, transform and load data, and organize database synchronization.

The most valuable feature of this solution is the data flow, which is the same SQL server in important service, integration services, which is a very robust and powerful tool to transform data.

The solution can be improved by decreasing the warmup time which currently can take up to five minutes.

I have been using the solution for two years.

The solution is extremely stable.

The solution is scalable.

Previously I used AWS Glue and SSIS.

The initial setup is straightforward.

The implementation was completed in-house and is immediate because it is a native cloud tool.

I have seen an ROI with the time saved migrating data for reports.

The solution's fees are based on a pay-per-minute use plus the amount of data required to process. The more data you process the more CPUs and time is required which increases the cost of using this solution.

I give the solution ten out of ten.

The only thing you need to deploy the solution is to click on publish.

The solution includes a feature that increases the number of processors used which makes it very powerful and adds to the scalability.

We have three people using the solution in our organization and one engineer that maintains it.

I recommend to any potential user to factor in the five-minute warm-up time that is required for each execution.

I can describe a scenario where I was tasked with developing a beta data trace and integration system. GitLab served as the data integration platform responsible for creating pipelines to extract data from various sources, including on-premises and cloud-based systems. Alongside GitLab Data Factory, we utilized Azure Logic Apps for orchestration and Azure Key Vault for securely storing data-related information. This combination enabled us to manage data extraction, transformation, and loading efficiently. The solution also involved Azure Data Lake for further data transformations, culminating in a comprehensive data processing engine that I played a key role in implementing.

The workflow automation features in GitLab, particularly its low code/no code approach, are highly beneficial for accelerating development speed. This feature allows for quick creation of pipelines and offers customization options for integration needs, making it versatile for various use cases. GitLab supports a wide range of connectors, catering to a majority of integration needs. Azure Data Factory's virtual enterprise and monitoring capabilities, the visual interface of GitLab makes it user-friendly and easy to teach, facilitating adoption within teams. While the monitoring capabilities are sufficient out of the box, they may not be as comprehensive as dedicated enterprise monitoring tools. GitLab's monitoring features are manageable for production use, with the option to integrate log analytics or create custom dashboards if needed.

The data flow feature in Azure Data Factory within GitLab is valuable for data transformation tasks, especially for those who may not have expertise in writing complex code. It simplifies the process of data manipulation and is particularly useful for individuals unfamiliar with Spark coding. While there could be improvements for more flexibility, overall, the data flow feature effectively accomplishes its purpose within GitLab's ecosystem.

Azure Data Factory could benefit from improvements in its monitoring capabilities to provide a more robust feature set. Enhancing the ease of deployment to higher environments within Azure DevOps would be beneficial, as the current process often requires extensive scripting and pipeline development. It is also known for the flexibility of the data flow feature, particularly in supporting more dynamic data-driven architectures. These enhancements would contribute to a more seamless and efficient workflow within GitLab.

I have been using Azure Data Factory for the past 3 years. 

I find GitLab to be quite stable overall. I haven't encountered significant issues with its stability and consider it a reliable platform.

In terms of scalability, there are a few aspects of GitLab that I find disappointing. For instance, the limitation on self-hosted integration run time to just four VMs restricts scalability, especially for handling large volumes of data. Improvements are needed in this area to support more than four VMs for scalability. The documentation regarding bandwidth support is unclear, making it difficult to assess the full scalability potential. While GitLab performs well in cloud scalability in terms of compute power, the limitations on self-hosted integration run time are a concern for certain use cases. scalability in GitLab is highly dependent on specific use cases and could benefit from enhancements in self-hosted integration run time capabilities.

As for technical support from Microsoft the response times and solutions can sometimes be delayed. 

Neutral

Our organization did not switch from a previous solution to Azure Data Factory; rather, we implemented Azure Data Factory as a new solution to enable cloud data processing capabilities.

The deployment was handled in-house, and initially, we had a team of five working on the project in 2019. 

We have approximately six to seven data engineers from various departments utilizing the solution in our organization. As for the frequency of using GitLab, it varies depending on the workload and projects, but on average, there is someone working on the platform at least several times a week, as our data engineers are involved in various tasks beyond GitLab.

I am aware of the pricing of Azure Data Factory, but I prefer not to disclose specific details.

In terms of handling complex data transformations and cleansing, Azure Data Factory is capable for simple to medium tasks, but for more complex tasks, we resort to custom coding solutions. Overall, I would recommend Azure Data Factory for data integration and management, and I would rate it an eight out of ten for its flexibility and ability to support third-party integrations.

I can't go into specifics about the use case for Azure Data Factory, but it's for analytics related to an assessment.

What I like best about Azure Data Factory is that it allows you to create pipelines, specifically ETL pipelines.

I also like that Azure Data Factory has connectors and solves most of my company's problems. I can't recall a case where I couldn't use the solution for solving problems.

I'm also happy about the Azure Data Factory dashboard.

A room for improvement in Azure Data Factory is its speed. Parallelization also needs improvement. As for the rest of the features of Azure Data Factory, I'm happy.

I cannot suggest an additional feature I'd like to see in Azure Data Factory in the future because some of the features aren't available internally because the features undergo security evaluation first, and my organization controls which features would become available to users.

I've been using Azure Data Factory for the last two years.

We're happy with the stability of Azure Data Factory.

Azure Data Factory is scalable, with clusters available on demand. There isn't any issue with scaling the solution.

We have an internal support team and the Azure Data Factory support team. We raise tickets and follow up on those tickets, and on a scale of one to five, we'd rate support as four because sometimes there are delays. Otherwise, we are satisfied with Azure Data Factory support.

My company didn't set up Azure Data Factory as the Azure team did it.

We outsourced the implementation of Azure Data Factory directly to the Azure team.

I have no idea how much Azure Data Factory costs.

We're using AWS apart from Azure Data Factory. We're trying out Palantir Foundry as well. They are the leading service providers in the data analytics and ETL world.

I'm familiar with Palantir Foundry, but my company just recently got the Palantir Foundry license, so I'm still not using it, but checking it for shortcomings.

I have experience with Azure Data Factory, too.

I'm unsure of the exact version of Azure Data Factory, but I'm using the latest version or whatever's available on Azure.

I have a vague figure of users of Azure Data Factory, but it's more than one thousand to one thousand five hundred people.

I'd tell people who want to use Azure Data Factory that Microsoft offers excellent courses, ESI (Enterprise Skill Initiatives). You should register and take the courses. Azure Data Factory is a solution I'd recommend to others.

I'd rate Azure Data Factory as nine out of ten because it has a lot of connectors, even custom connectors, for data onboarding. It can also integrate with Spark notebooks and allows my organization to parallelize code. Azure Data Factory also has provisions for Spark and SQL scripts or any scripts, plus the infrastructure is highly scalable, so it's a nine for me.

My organization is a customer of Azure Data Factory.

My main use case for Azure Data Factory is to pull data from on-premises systems. Most data transformation is done through Databricks, but Data Factory mainly pulls data into different services.

The solution handles large volumes of data very well. One of its best features is its ability to integrate data end-to-end, from pulling data from the source to accessing Databricks. This makes it quite useful for our needs.

The main challenge with implementing Azure Data Factory is that it processes data in batches, not near real-time. To achieve near real-time processing, we need to schedule updates more frequently, which can be an issue. Its interface needs to be lighter. 

One specific issue is with parallel executions. When running parallel executions for multiple tables, I noticed a performance slowdown.

I have been working with the product for five years. 

We haven't faced any issues with the tool's stability. 

The solution can handle large datasets. 

I am satisfied with Microsoft's support. They provide solutions to our challenges. 

Positive

The solution is cheap. 

I rate the overall product an eight out of ten. 

We are currently migrating from on-prem to the cloud, and our on-prem tables are getting data from upstream. We used ADF to build a pipeline to facilitate this migration. A team of 15-20 people currently uses ADF, and more will join once it goes live.

ADF is another ETL tool similar to Informatica that can transform data or copy it from on-prem to the cloud or vice versa. Once we have the data, we can apply various transformations to it and schedule our pipeline according to our business needs. ADF integrates with Databricks. We can call our Databricks notebooks and schedule them via ADF. 

I have used Azure Data Factory for about six months.

I rate Azure Data Factory six out of 10 for stability. ADF is stable now, but we had problems recently with indexing on an SQL database. It's slow when dealing with a huge volume of data. It depends on whether the database is configured as general purpose or hyperscale. 

I rate Azure Data Factory eight out of 10 for ease of setup. The deployment time depends on the data volume. Four million records will take longer than four thousand. Migrating our full load from on-prem to the cloud took around 16-18 hours because the volume was 17 million. 

I rate ADF six out of 10 for affordability. The cost depends on the services we use. It's usage-based. 

I rate Azure Data Factory seven out of 10. Companies that want to migrate from on-prem to the cloud have lots of options. I haven't explored them all, but Azure, GCP, and AWS are essentially all the same.

We use Azure Data Factory for data integration.

The most valuable features of the solution are its ease of use and the readily available adapters for connecting with various sources.

Some known bugs and issues with Azure Data Factory could be rectified.

I have been using Azure Data Factory for about two years.

I rate the solution an eight out of ten for stability.

Azure Data Factory is a scalable solution. A team of 16 people from the data analytics team use the solution in our organization.

I rate the solution an eight out of ten for scalability.

On a scale from one to ten, where one is difficult and ten is easy, I rate the solution's initial setup a seven out of ten.

A team of three people deployed Azure Data Factory in three to four days.

The solution's pricing is competitive.

We build data pipelines primarily for integration. Few of them are real-time data transfers, and few of them would be a batch-free file. These would direct the data from various sources to our data warehouse. Azure Data Factory helps build the data pipelines and adaptors.

The solution has built-in features and a control center for us to monitor the status of the pipelines. The solution's email notification also helps us in monitoring. We didn't face any challenges to set up the data pipelines. We know there are some controls, but governance is customized for the organization's requirements. We have our own policies.

Azure Data Factory is deployed on the cloud in our organization. I would recommend Azure Data Factory to other users.

Overall, I rate the solution a nine out of ten.

Our primary use case for the solution is data integration and we deploy it only on Azure.

When we were integrating the Ports product with our internal data warehouse, we had to update all the reports to our internal data warehouse on the Ports system database. However, they were not given access to the database company, and they dump some files or provide you with them. In one case, they were providing files. In another case, they provided some APIs where you need to call in a batch of thousands of records multiple times. It works very well with Azure Data Factory to pull the records, parse them quickly and post them in the database and data warehouse.

We have found the bulk load feature very valuable.

The only challenge with Azure Data Factory is its exception-handling mechanism. When the record fails, it's tough to identify and log.

We have been using the solution for a year and a half and are currently using the latest version.

The solution is scalable and we intend to further increase its usage in the future.

I rate customer service and support an eight out of ten.

Positive

We previously used different solutions.

The initial setup is straightforward.

The implementation was done in-house.

I cannot comment on licensing costs because I was not involved.

I rate the solution a six out of ten. The solution is good but its exception handling and logging mechanisms can be improved. I advice users considering this solution to go for it especially if their integrations are heavy on the data side.