AWS Glue Reviews

I constructed a straightforward ETL job using AWS Glue, wherein I had to load a couple of files in the Teradata database.

AWS Glue is quite better than other tools, but you have to learn it properly before you start using it.

While working on AWS Glue, I could not find any training material for it. Although it's not a problem with the product, the solution could include better documentation.

I have been using AWS Glue for about two months.

AWS Glue is a stable solution.

AWS Glue's initial setup is quite straightforward.

Overall, I rate AWS Glue a seven out of ten.

We use the solution to collect customers' data containing multiple files and convert it into a common database. Later, we send the database for SQL injection.

The solution's most valuable feature is its ability to efficiently collect and catalog the data in the warehouse.

They should improve the solution's performance in case of large amounts of data. Currently, AWS fails to handle massive databases acquired from various sources. Also, it is challenging to queue the data or use a standard code in AWS environment. We need to install a third-party tool to tackle the issue. We need to use another tool to convert the data as well. Thus, we are using multiple tools to handle the database. They should work on this particular area.

We have been using the solution for one year.

It is a stable solution. I rate its stability as an eight.

I rate the solution's scalability as a six.

The initial setup is a bit complex, and I rate the process as a six. We have to install multiple third-party tools whenever we update the security patches or renew the solution. Thus, the deployment process is complicated.

If you already have AWS environment, you can opt for AWS Glue for its ETL operations feature; if you want to process multiple operations, such as creating a table or catalog, or for machine learning purposes better to go for other database tools.

I rate the solution as a seven.

Our company uses the solution for ETL data movement for our customers such as on-premises to cloud, cloud to cloud, and cloud to Snowflake. We also data catalog and schedule ETL jobs. We are able to monitor all jobs through AWS services. 

Transformations are valuable because you can modify or override complex data logic from an open source or Spark to solve issues. 

For example, it is easy to solve issues where volume is good but performance is degrading because you can split jobs into small chunks to more quickly handle data loads. 

The setup and installation is a bit complex without advanced knowledge or training. It would be easier for an AWS expert or someone in DevOps.

Transformations need improvements to be more user friendly and rely less on coding like Matillion. 

I have been using the solution for three years. 

The solution's stability is decent and rates higher than other products. It works well with Snowflake, Azure, GCP, and AWS-supported products. 

A hybrid situation may cause delays in performance. 

The solution is scalable. 

One of our customers used technical support and found them to be helpful. 

The setup and installation is a bit complex. Training or advance knowledge is required. Someone with AWS experience or a DevOps perspective would have fewer issues. 

We install the solution for customers and the timeline depends on the job. 

A complete project will take a few days to a week for deployment. The number of jobs and components determines how many technicians are required for setup, installation, and deployment. Technician requirements can range from two to fifteen. 

Deployment will take a couple of hours for a few announcement jobs that deploy from the CI/CD pipeline.

The solution is my second choice because I prefer Snowflake's capabilities. 

Our company is creating data warehousing in the cloud. Our team includes four data engineers, two data ops, and two data administrators. 

We use S3 to data lake or prepare data from two databases that are contained in MySQL and Oracle. For the migration, we use DMS.

Then, we use the solution to perform data transformation. For Oracle, we use Data Catalog and Data Crawler to create our catalog. Dev Endpoint is used to develop complex data transformations. We then migrate to Studio Notebook where we develop and schedule a complex Spark job. 

Finally, we load the transformed data to Redshift so our data analyst team can visualize it with QuickSight. 

The solution is serverless so it allows us to transform data while optimizing the cost and performance of Spark jobs. 

The solution works with many data sources and services in the cloud. 

Glue Watch monitors our Spark jobs and immediately alerts us to issues so we are able to resolve them quickly. 

The solution does not work with Spark DataFrame. We can use the solution's DynamicFrame for this function but transformations are expensive. 

Not enough resources or services are available to run managed Spark jobs within the solution. We have reached out to Amazon many times regarding this issue. 

The solution should offer features for streaming data in addition to batching data. We can use other products such as Scala or Python but prefer the features be available in the solution. 

I have been using the solution for one year. 

The solution is stable with no issues. 

The solution is scalable. 

Technical support has been good and has handled any issues. 

I rate technical support an eight out of ten. 

Positive

The solution is the best service in its category at this time. Based on project budget and use case, we use either the solution or EMR.

EMR is used for projects that require the latest version of Spark. 

We use the solution for any other versions of Spark. 

I was not involved in the initial setup.

The solution's pricing is based on DPUs so it is a good idea to optimize use or it can get expensive. 

I use Studio Notebook because it is less expensive and jobs can be deleted or clustered to run in one day. 

I rate pricing a four out of ten. 

Our company only uses Amazon cloud because other cloud environments do not offer the same features. 

The solution's Studio uses GCP which is easier than coding in Python Spark or Scala Spark. 

Azure Data Factory's features do not compare to what the solution can do in the cloud. 

The solution is good for teams who do not want to worry about DevOps or who want to optimize cost by using the cloud. 

I rate the solution a seven out of ten. 

We are using AWS Glue for transforming firewalls synced to the Data Lake in the bronze zone. The ATL uses the solution to transform fields in the silver layer and later we will produce the gold zone. We are using the Delta Lake Architecture.

The most valuable feature of AWS Glue is its ease of use and good documentation. Additionally, we can do all the transformations that we need.

The price of the solution could improve.

I have been using AWS Glue for approximately one month.

The stability of AWS Glue is good.

AWS Glue is highly scalable.

There are dozens of customers using this solution.

I have not used the support from AWS Glue but I know their support is good.

I have previously used Azure and Spark for testing.

The initial setup of AWS Glue is simple. In other solutions, such as Spark, the configuration would take a lot longer.

I did the deployment of AWS Glue myself with the AMS console. I am a data engineer.

The overall cost of AWS Glue could be better. It cost approximately $1,000 a month. There is paid support available from AWS Glue.

If the cost of AWS Glue was 50 percent less then we would not move to another solution.

I am moving to the EMR serverless or GCP solution.

I rate AWS Glue a nine out of ten.

The key role of Glue is that it hosts our metadata before rolling out our actual data. This is the major advantage of using this solution and our clients client have been very satisfied with it.

The most valuable aspect of this solution is its automation and ability to sync data from the source to the solution phase. 

The technical support for this solution could be improved. In future, we would like to connect more services like Athena or Kinesis to help control more loads of data.

I have been using this solution for three years. 

This is a stable solution. We have isolated the environment using containerization so that if anything goes wrong, we have higher levels of scalability and availability. To achieve this, we have configured multiple servers for testing, UAT and development.

This is a scalable solution which is supported in our organization by Docker and Kubernetes. We have 2,000 users.

We used a vendor with an internal IT team who provided us with architecture so that we could leverage those services and reach a solution. They have 50 people in the IT team, who continuously help us and monitor the things that we are working on.

Positive

The initial setup was straightforward and took approximately one month. For deployment, we worked in two teams. One person handled all the scripting which we are developing for automation. Two other members handled the database and servers.

This solution is affordable and there is an option to pay for the solution based on your usage. 

I would rate this solution a seven out of ten. 

Glue is a NoSQL-based data ETL tool that has some advantages over IIS and ISAs. It is tailored and customized to use with SQL Server, which works very well in that platform.

If you want to use other data sources, the NoSQL concept makes it very easy, because missing data can be inserted as a new column or with null values.

That is not the case with many other tools. If you have on-premises tools, such as IIS, they don't manage missing data well.

If you want extremely high-performance functionality, you have to use both AWS Glue or Data Lake to store it in some temporary table. First, you will have to do some cleaning of the data, then if you need performance and speed, you have to use IIS with an IBM tool. 

You have to use the right tool in the right places. For example, if you're using Oracle, you have got to use the Oracle tools. If you are using SQL, you have to use the SQL tools. There is no other tool that provides the performance.

It's context-based and project-based. In the projects that I have used, it has worked well.

There is a learning curve to this tool.

I have been working with AWS Glue for four years.

Everything runs on AWS, even if it belongs to a third party. For example, if you have a Netflix subscription, it runs on AWS. We have other products or vendor subscriptions that run on AWS.

Undoubtedly, the cloud is built to handle failure. If you have your devices, and your resources configured correctly, you won't have any issues. I haven't seen a problem.

You have to pay for their technical support, and depending on which level of subscription, you will receive a call within an hour; otherwise, you will have to wait for days.

We also use Azure's Data Lake, and I worked with Tipco in the past, though it's been a few years since we used it.

You should select the best tool for the job or the projects that are currently being worked on. Tipco was heavily used in the previous project we worked on.

It takes some time to learn, but once you get the hang of it, you'll be fine. It's like any other IT tool, where nobody is an expert or isn't an expert, it is just the way you are exposed to a tool. 

You've chosen the right tool if you understand how the data works and what it needs to do. It's like going to Home Depot to get the right tool. You can purchase a set of tools, and it will work for you, but you will still need to purchase something else.

It's one of those tools in which someone must be an expert. After that, all tools and platforms become secondary.

With AWS Glue, you pay more, but if you want to process the data, with speed and performance, you need the correct EC2 instances.

There is a price to pay. It doesn't come free.

Technical support is a paid service, and which subscription you have is dependent on that. You must pay one of them, and it ranges from $15,000 to $25,000 per year. 

You sign up for a level of service, and it does not come for free. As previously stated, everything is based on performance, ELAs.

It was very expensive, at that time. If a company wants to pay the money, it makes my job easier. However, if the company or enterprise does not have the funds to pay for it, then it is a hassle.

In that environment, there is a lot going on. There are some things that you can get for free, and there are some add-ons that you can develop or use that have been tested. It's all about convenience and service. You will get what you pay for if you pay for what you want.

I'm not a fan of any tools; it all depends on the organization I work for, where their data is, what they want to do with it, how quickly they want to get there, and what their budget is, and you work around that. For me, I would not choose one over the other, unless I know the details of the project.

I would rate AWS Glue a nine out of ten.

We are collecting some TV audience data and analyzing it.

Data catalog and triggers are the two best features for me. 

AWS Glue has its own data catalog, which makes it great and really easy to use. Triggers are also really good for scheduling the ETL process. 

The start-up time is really high right now. For instance, when you start up a new job, you have to wait for five or eight minutes before it starts. If the start-up time is reduced to one or two minutes, it will be great.

It will be better to have a direct linkage to Redshift in AWS. If we can use data catalogs from Redshift, it will be so easy to create some data catalogs. Currently, we can only use data catalogs from S3.

We have been using the AWS Glue for approximately one and a half years.

There is no problem related to stability.

Scalability is good. I can reduce or increase the number of DPUs, which I find very useful.

We are trying to increase the usage of AWS Glue because of customer needs. When the data increases, our application needs some more analyzers and user interfaces. We will increase our data analyzer and user interfaces.

I didn't take any technical support because I didn't have a big problem or issue. I just used some information from various communities and forums about the maintenance. 

The pricing is a bit higher than other solutions like Athena and EC2. If the pricing becomes more scaled or flexible, it will be good because you have to pay 44 cents just for one DPU for an hour. If you increase DPUs to 5 or 10, the pricing gets multiplied. 

There are also some time limits like 0 to 10 minutes or 10 to 20 minutes. If the pricing is according to the minutes, it would be better because you have to limit your job to 10 minutes or 20 minutes.

I would recommend AWS Glue. It is a great choice. 

I would rate this solution a nine out of ten.