AWS Batch is a powerful service for managing compute-intensive workloads efficiently. By seamlessly integrating with EC2 and other AWS services, it streamlines the execution of container and batch computing jobs, maximizing resource use and scalability.
| Product | Mindshare (%) |
|---|---|
| AWS Batch | 8.1% |
| AWS Lambda | 15.3% |
| Amazon EC2 | 13.7% |
| Other | 62.900000000000006% |
| Title | Rating | Mindshare | Recommending | |
|---|---|---|---|---|
| Apache Spark | 4.2 | 8.5% | 90% | 69 interviewsAdd to research |
| AWS Lambda | 4.3 | 15.3% | 94% | 92 interviewsAdd to research |
| Company Size | Count |
|---|---|
| Small Business | 5 |
| Large Enterprise | 4 |
| Company Size | Count |
|---|---|
| Small Business | 36 |
| Midsize Enterprise | 15 |
| Large Enterprise | 105 |
AWS Batch provides a comprehensive job scheduling platform, automating resource provisioning and scaling for dynamic workloads. It supports container workloads and offers both EC2 and Fargate options, boosting flexibility and maintaining costs. Users can efficiently run concurrent jobs with customizable resource templates and take advantage of dynamic scaling and memory management tailored to task requirements. Despite its strengths, AWS Batch could benefit from improved job visibility, debugging, and simplified configuration processes. Enhancements in monitoring, integration with AWS services, and pricing adjustments could further optimize performance. Improving IAM privilege setup, documentation, and error handling is essential for smoother operations.
What are the key features of AWS Batch?In industries like data science and analytics, AWS Batch is essential for managing large datasets and running complex simulations. Finance and health sectors leverage its capabilities for log processing, report generation, and other compute-heavy tasks. Businesses benefit from its ability to execute tasks at scale without significant overhead.
AWS Batch was previously known as Amazon Batch.
| Author info | Rating | Review Summary |
|---|---|---|
| Software Engineering Manager – Digital Production Optimization at Yara International ASA | 4.0 | I find AWS Batch highly flexible and scalable for containerized workloads. It's stable with easy setup. However, error handling needs improvement, especially with Spot Instances, and optimal use demands understanding underlying services. |
| Senior Battery Data Engineer at a agriculture with 51-200 employees | 4.5 | I rely on AWS Batch for its excellent scalability, reliability, and cost-effectiveness in data processing. While AWS integration is robust, I find debugging complex due to slow console logs, and job termination sometimes requires multiple attempts. |
| Head of Development at Abyss | 3.5 | I find AWS Batch stable and highly scalable for running secure Python code. However, Fargate's 30-second startup time and the complex initial setup documentation are significant challenges that need improvement. |
| Senior Data Engineer at a tech services company with 5,001-10,000 employees | 4.5 | I use AWS Batch for cost-effective, reliable backup processing of QuickSight assets. It's stable, scalable, and easy to use, utilizing spot instances and containers. I haven't found any significant improvements needed for this straightforward solution. |
| Head of Bioinformatics at Paratus Sciences | 4.5 | AWS Batch offers flexibility similar to HPC environments, allowing scalable compute resources without significant hardware investment. While more costly than some HPC setups, its quick setup and adaptability suit projects with varying resource needs, making cloud deployments efficient and effective. |
| Independent Consultant at a consultancy with 1-10 employees | 4.0 | I leverage AWS Batch for cost-effective, scalable parallel processing of containerized pipelines, appreciating EC2's auto spin-up/down. While powerful, the IAM setup and documentation present a learning curve, despite generally good stability. |
| Works | 3.5 | I use AWS Batch for data processing, valuing its parallelization for large datasets. While setup was easy, I want configuration as code, better automated notifications, and noted some stability issues. I rate it 7/10. |
| DevOps Engineer at ZoomOps Technology | 5.0 | We use AWS Batch to manage containerized workloads with dynamic scaling and integration with AWS services. While it efficiently handles job scheduling, provisioning, and scaling, improvements are needed in pricing, integration, scalability, reliability, and keeping information up-to-date. |
| Architect- Cloud/Automation at a consultancy with 1,001-5,000 employees | 4.5 | I use AWS Batch to run scripts longer than 15 minutes, leveraging its parallelism and scalability. The solution is easy to configure with strong security features. However, the UI needs improvements, especially when handling numerous batch jobs, and it's challenging for junior developers to learn. |
| Assistant Consultant at Tata Consultancy | 4.0 | In my experience with AWS Batch, the tool has provided a good return on investment. However, its documentation requires improvement for better usability. I did not consider any other solutions or use different deployment or cloud providers. |
Positive
Positive
In many ways, it's like using an HPC environment but a lot more flexible. In theory, you could have many different kinds of computing systems and computers, ranging from those geared toward computational speed to larger memory machines or GPU machines.
The idea is to break your computational jobs into smaller jobs that can be run on multiple machines. Any of these virtual machines will orchestrate all of the computation, send out jobs to different machines, wait for them to be done, and then run the next process in the sequence. It's simply a way to run multiple processes on multiple machines in the cloud.
Scalability is the most valuable feature for me. I could run anywhere from 32 cores to over 2,000 cores. So it scales very well, and it's really good for situations where jobs are very heterogeneous, meaning that you have a long-running job that sometimes needs a lot of small compute-intensive machines, for example.
But then, in the second stage, you may need a few very high-memory machines. It's really good for those kinds of situations, for HPCs, where you can really customize and tailor the compute, memory, and GPU requirements for the job.
You can run as many jobs as you want, provided you pay the cost. So, it's about the scalability to run really large jobs in a really short amount of time with a very minimal setup.
One person can set up a compute cluster on AWS Batch. I don't need all the hardware resources, people to maintain those resources, or software installations.
Moreover, there is one other feature in confirmation or call confirmation where you can have templates of what you want to do and just modify those to customize it to your needs.
And these templates basically make it a lot easier for you to get started. So, if you've been doing this for a while, you probably already have a template in your toolbox, and you can use one of those to get started, but you would just customize it. So, these templates help a lot.
The main drawback to using AWS Batch would be the cost. It will be more expensive in some cases than using an HPC. It's more amenable to cases where you have spot requirements.
So, for instance, you don't exactly know how much compute resources you'll need and when you'll need them. So it's much better for that flexibility. But if you're going to be running jobs consistently and using the compute cluster consistently for a lot of time, and it's not going to have a lot of downtime, then the HPC system might be a better alternative. So, really, it boils down to cost versus usage trade-offs. It's going to be more expensive for a lot of people.
In future releases, I would like to see anything that could help make it easier to set up your initial system. And besides improving the GUI a little bit, the interface to it, making it a little bit more descriptive and having more information at your fingertips, so if you could point to the help of what the different features are, you can get quick access to that. That might help.
With most of the AWS services, the difficulty really is getting information and knowledge about the system and seeing examples. So, seeing examples of how it's being used under multiple use cases would be the best way to become familiar with it.
And some of that would just come with experience. You have to just use it and play with it. But in terms of the system itself, it's not that difficult to set up or use.
I have been working with AWS Batch for six years now.
It's a stable product. There are times when your compute access may be down due to AWS having an outage, but there's a lot of redundancy in the system.
So, again, it's one of those things where you get what you pay for. You can have your system up 24/7 all the time if you have implemented some redundancy in there.
So if something goes down, it fails over automatically to another system, another region, or a compute, which, in my experience, has also been very, very, very, extremely rare that something like that happens.
The trade-off again is that you're going to pay for it. Because of that redundancy, you have double the amount of resources needed, or maybe not double, but you do have more resources allocated than you would otherwise need.
So, if you're willing to pay for that, you have that flexibility. If you're willing to tolerate some downtime, then you'll end up spending less. If it's crucial for you to have your system operational all the time with no downtime, then it will cost you more, but you can achieve it.
Currently, I am consulting. But in my previous organization, all of our compute was done in the cloud. So, that was the full informatics team, the stock five people, and all of the DevOps people, that was about three or four people. So, around ten people were on the systems, and all of our compute, all of our storage was done in Amazon cloud. Everything.
It's one of those things where, with AWS, you'll quickly realize that you can get things faster, quicker, and better, but you're going to pay for it. So it's the same thing with support.
With the cheaper support, you'll get a response within 48 hours. If you pay extra, then you can get support within an hour. Again, it really depends on you and how quickly you need the support and the service, and that's all customizable.
And there are those gradients in between. If it's a mission-critical problem where your production system needs to be up right now and there's a problem, you can get support right away, within the analogy or even inline.
If your problem is something that you don't need right away, and maybe you can figure it out in a few days, or you can wait for a response after a few days, then that's going to cost you less. So you have that sort of flexibility.
So it comes down to basically how much you're paying is what you're going to be receiving at the end of the day.
Positive
In my experience, I've only used Amazon and Google Cloud significantly. And Amazon is the reason why I use, by far, the number one kind of compute service.
They just have a lot of maturity, and they've been doing things for a long time. So, they've worked down a lot of the problems that other cloud providers may be affected by. But, again, there is a trade-off that they are typically more expensive than Google or Microsoft.
There are a few parameters here. In the beginning, one of the decisions I needed to make was whether we were going to go with Amazon or Microsoft.
And Microsoft Azure was significantly cheaper, and they were even giving us credit, the equivalent of about a million dollars of credit, to use Microsoft Azure. So, it was going to be a lot less than the AWS.
But there were several reasons why we decided to go with Amazon. One of the advantages is that Amazon is just a much more mature system. And so we could find people who are most familiar working with the system in terms of solutions architects and other companies, third-party companies, in order to build infrastructure.
It is easier to give access to people. There were a lot more services in Amazon, so there was a lot less need for us to build things ourselves. There was a lot more knowledge on Amazon. So, I was pretty familiar with Amazon. I had little general experience with Microsoft, so that was another factor.
And then so given all of these things in the long term, we were thinking of not just over the next couple of years, but long term, maybe five years from now, once we added dedicated our systems to one system, we didn't really want to change them or would end up being cut.
So even though we were going to save a lot upfront, in the long term, that would probably end up being about the same or a little bit more expensive. So that was one of the reasons for choosing AWS Batch.
In terms of using AWS Batch, specifically, we needed computational tasks that, like were very heterogeneous, so sometimes you needed maybe 32 cores on a very large memory machine. Sometimes we needed 2000 cores, on a lot of different machines to be running in parallel. And we really needed computation that was very bursting.
So sometimes we would need 2000 cores running at the same time. And then, sometimes for weeks, we just wouldn't have the system running at all. We didn't want to have to pay for services that we weren't going to be using.
So, the on-demand cost structure of on-demand ended up working out better in the long run. But it also had some flexibility. Some of the machines when we were going to use them for a long time and used them consistently, so we could stay a lot more upfront to reserve it, and that ended up helping us a lot less over the long run.
AWS has integrated well with a lot of different software. We're using things like NextFlow and other systems. So, in many ways, it's the natural fit. These were the reasons we opted for AWS Batch.
It's one of those things where, if you have the appropriate background, experience, and skill set, it's not very difficult to set up.
As with many other AWS systems and services, the main factor is the experience and scale of the solutions architect, administrator, or DevOps person setting up the system.
If it's someone with experience who has been doing this for some time, it's not very difficult to set up. If it's a situation where somebody is very new at it, then it's going to take a long time. But for me, for example, I can set up a batch system in an hour or less.
Mainly all of the AWS systems can be set up through their console and their accounts. There's a fair amount of setup in terms of allocating the resources you want, and especially with security, making sure that the appropriate people have the necessary, but minimal, requirements for security and minimal access to what they need.
You also need to look at what kind and how many compute resources you would want. Because with a lot of the power that you get from these kinds of systems, there's a lot of customization that you can do.
You can add a lot of different features, and you can also remove a lot of features. Security is very important because you don't want to give people access to things that they shouldn't have. At the same time, you want to make sure that people have access to the things that they need to do their work.
AWS Batch can end up saving you money. It really depends on your long-term situation. If you are looking for situations where you want to get started very quickly, there's nothing going to be faster than doing something in the cloud like a data system. So, it doesn't have to be AWS. Microsoft Azure could work, or Google Cloud, etcetera.
So if you want it to be set up very quickly and you want to do things like prototyping, development, etc., it will be more cost-effective for you to use AWS Batch because you don't have to have that upfront investment in very expensive computers. You can just get started and figure out exactly what your compute requirements are going to be, how much power you need, how long you need, etcetera.
It's especially good for situations where the compute requirements vary over time. Like, if you were running an e-commerce site, maybe during Black Friday, you need much more compute resources, but maybe in the summer, you don't sell as many items, so you don't need as many compute resources. So you wouldn't need to invest in a lot of equipment that you're probably not going to be using at the time. So in those kinds of circumstances, in my opinion, it's the best way to go.
In situations where it may not be cost-effective, if you're in a production environment where you expect to run a very consistent workflow over time, it's not going to change very much. You know exactly what you need and what you want, and you have the resources to dedicate people to maintaining the system, administrative people, people looking at the hardware, and you expect to run the system for years, not something that's just going to be running for a short amount of time. Then, for the HPC environment, it would be better.
Honestly, that's going to change in a few years. As time has gone along, people are moving more to the cloud compute environment, and it works better in most of the circumstances.
Overall long-term cost is not as high as it's going to be in the HPC environment. So it really depends on the situation and the circumstances. So, HPCs, in my experience, having been using HPCs for almost 20 years, I would say there are less and fewer circumstances where HPC is going to be a more cost-effective approach.
If you're very new to Amazon and AWS, I would recommend spending six months to a year in training. This advice is particularly relevant if you're coming from an administrative standpoint, where you're either individually responsible or part of a team setting up systems for others. Gaining experience and understanding the nuances of these systems can be the difference between a system that's consistently operational and one that's frequently down or slow.
Moreover, this experience can significantly reduce costs. An experienced professional could slash your costs by 50% to 60%. Hence, it's crucial to amass as much knowledge and experience with the system as possible, particularly if Amazon is a new environment for you.
If you're already familiar with Amazon, then getting up to speed with AWS Batch should be relatively straightforward. It follows a similar implementation procedure to other AWS services. You might need just a few weeks to get accustomed to it, gaining exposure and experience in running different systems.
For end-users, those who need to run computational jobs but aren't involved in the setup, extensive exposure isn't necessary. The systems are already in place, and you'll likely use AWS Batch indirectly through third-party software. It's a simpler process for end-users; you submit your job, and the system, like a "magic black box," delivers the results without you needing to understand the backend processes.
Personally, I would rate the system a nine out of ten. My experience with it has been mostly positive, albeit with a steep learning curve, especially if you're coming from an HPC environment, in which case I was. It's a different way of thinking about how you're going to do your computation. You have a lot more flexibility. It's one of those things like the old adage: the more power you get, the more responsibility you have. And that's the case with AWS services or cloud services in general. You get a lot more power, a lot more flexibility. And at the same time, you require more knowledge to be able to use it, and you need to understand what all of these different features are if you want to really maximize using the system.
The only reason I shy away from a perfect score is the documentation and use cases and examples; things like that, there could be more information on those kinds of things. It would definitely help somebody get used to the system. You really need somebody who's had a significant amount of experience in order to make the most of the system. Otherwise, when somebody's new to the system, and they don't really fully appreciate all of the features. And they get frustrated very quickly because they can't get it to do what they want.
Overall, AWS offers immense capabilities. Though it's not a standalone solution — often requiring third-party software for specific projects — Amazon continually integrates more of these tools. For instance, AWS Parallel Cluster, akin to Batch, can simulate an HPC environment, providing additional flexibility.
Neutral
We use AWS Batch to manage containerized workloads and dynamic scaling. The solution automatically scales computer resources based on the number of jobs and resource requirements. The solution also integrates with other AWS services.
AWS Batch manages the execution of computing workload, including job scheduling, provisioning, and scaling. AWS Batch is specified as a docker container that allows us to run our applications with all their dependencies.
The solution could improve its pricing-based resources like job execution. The solution should include better and seamless integration with other AWS services, like Amazon S3 data storage and EC2 compute resources. The solution should be updated to improve overall performance, including scalability and reliability.
AWS Batch should regularly update its service and provide the most accurate and up-to-date information.
I have been using AWS Batch for one year.
Around three to four people are using the solution in our organization.
The solution’s initial setup is easy.
AWS Batch is a cheap solution.
The solution's monitoring includes checking the performance, health, and resource utilization of our AWS Batch computing jobs. AWS Batch is integrated with CloudWatch, which provides detailed metrics about our AWS Batch jobs and compute environment. I would recommend AWS Batch to other users. It is easy to learn to use the solution.
It is easy to integrate AWS Batch with AWS CloudWatch, which provides metrics and logs to monitor and troubleshoot AWS Batch jobs. AWS web supports the execution of AWS Batch jobs and docker containers. The solution allows the use of containerization applications and uses CLI.
Overall, I rate the solution ten out of ten.
We use the solution to run scripts for more than 15 minutes. We do not get builds for running the scripts. We can deploy it using containers.
Parallelism and scalability are the best features of the solution. The solution is very easy to configure and run by ourselves. We can use the security features to allow only specific service principles like the IAM role. We can easily integrate AWS container images into the product.
When we run a lot of batch jobs, the UI must show the history. The last time we used it, we faced some glitches. Explaining the tool to junior developers was difficult. They were not able to easily grasp how to link the components.
I have been using the solution for two to three years.
I rate the tool’s stability a ten out of ten.
The tool’s scalability is very good. I rate the scalability a ten out of ten. We have less than 20 users.
The support team is good. The support engineers are knowledgeable and provide solutions to our queries. The response time is quicker than Azure’s support team.
Positive
AWS Batch is easier to use than Azure Batch. I prefer AWS over Azure because AWS has good documentation and community support.
The initial setup of AWS Batch is easier than Azure Batch. I rate the ease of setup a ten out of ten. The end-to-end deployment took 25 to 30 minutes.
The pricing is very fair. The price was negligible compared to the number of tasks we run.
It is a very good solution to use. AWS Batch is a suitable solution if users want to run multiple containers of the same image with multiple variables. It is efficient and scalable. The issues are solved much more quickly. Overall, I rate the tool a nine out of ten.
AWS Batch needs to improve its documentation.
I have been using the product for six years.
I rate the solution's stability a nine out of ten.
I rate the tool's scalability a nine out of ten.
AWS Batch's deployment was easy.
We have seen ROI with the tool's use.
AWS Batch's pricing is good.
I rate the product an eight out of ten.
