We use AWS Batch for data processing workloads, such as log processing, report generation, and running compute-heavy jobs asynchronously. In one project, we had a requirement to process large log files daily. Instead of running everything on an always-on server, we containerized the processing job and used AWS Batch to run it on demand. It spins up the compute resources, processes the data, and shuts everything down automatically. AWS Batch is especially useful for workflows that don't need to run continuously but require high compute for short durations. We have evaluated Lambda as well, but it was not suitable for long-running jobs, so we moved to AWS Batch.
Senior Data Engineer at a tech services company with 5,001-10,000 employees
MSP
Top 20
Apr 24, 2025
In my project, I worked on an enhanced backup of some assets in AWS ( /products/amazon-aws-reviews ) QuickSight ( /products/amazon-quicksight-reviews ). AWS ( /products/amazon-aws-reviews ) QuickSight ( /products/amazon-quicksight-reviews ) has a rudimentary 30-day buffer for restoring deleted assets but lacks other versioning options. To address this, I back up data to JSON files and run an AWS Batch ( /products/aws-batch-reviews ) process hourly to detect any changes during business hours. Additionally, a longer process runs every few months to check the data's integrity and restore it to its original state, which takes a couple of hours due to the size of the QuickSight installation.
I use AWS Batch ( /products/aws-batch-reviews ) for my data activities, including building the data science pipeline, processing, storing data, and performing analytics.
Software Engineering Manager – Digital Production Optimization at Yara International ASA
Real User
Top 5
Mar 10, 2025
I use AWS Batch ( /products/aws-batch-reviews ) for planning, scheduling, and computation on containerized workloads. In my previous role, I heavily used AWS Batch ( /products/aws-batch-reviews ) for running numerous simulations. It includes running different types of computations, such as on-demand and scheduled computations, and creating clusters to optimize memory, CPU, or GPU usage. This flexibility allows processing batch jobs effectively, such as data processing and computation.
We use AWS Batch to manage containerized workloads and dynamic scaling. The solution automatically scales computer resources based on the number of jobs and resource requirements. The solution also integrates with other AWS services.
In many ways, it's like using an HPC environment but a lot more flexible. In theory, you could have many different kinds of computing systems and computers, ranging from those geared toward computational speed to larger memory machines or GPU machines. The idea is to break your computational jobs into smaller jobs that can be run on multiple machines. Any of these virtual machines will orchestrate all of the computation, send out jobs to different machines, wait for them to be done, and then run the next process in the sequence. It's simply a way to run multiple processes on multiple machines in the cloud.
AWS Batch is a powerful service for managing compute-intensive workloads efficiently. By seamlessly integrating with EC2 and other AWS services, it streamlines the execution of container and batch computing jobs, maximizing resource use and scalability.AWS Batch provides a comprehensive job scheduling platform, automating resource provisioning and scaling for dynamic workloads. It supports container workloads and offers both EC2 and Fargate options, boosting flexibility and maintaining costs....
We use AWS Batch for data processing workloads, such as log processing, report generation, and running compute-heavy jobs asynchronously. In one project, we had a requirement to process large log files daily. Instead of running everything on an always-on server, we containerized the processing job and used AWS Batch to run it on demand. It spins up the compute resources, processes the data, and shuts everything down automatically. AWS Batch is especially useful for workflows that don't need to run continuously but require high compute for short durations. We have evaluated Lambda as well, but it was not suitable for long-running jobs, so we moved to AWS Batch.
In my project, I worked on an enhanced backup of some assets in AWS ( /products/amazon-aws-reviews ) QuickSight ( /products/amazon-quicksight-reviews ). AWS ( /products/amazon-aws-reviews ) QuickSight ( /products/amazon-quicksight-reviews ) has a rudimentary 30-day buffer for restoring deleted assets but lacks other versioning options. To address this, I back up data to JSON files and run an AWS Batch ( /products/aws-batch-reviews ) process hourly to detect any changes during business hours. Additionally, a longer process runs every few months to check the data's integrity and restore it to its original state, which takes a couple of hours due to the size of the QuickSight installation.
I use AWS Batch ( /products/aws-batch-reviews ) for my data activities, including building the data science pipeline, processing, storing data, and performing analytics.
I use AWS Batch ( /products/aws-batch-reviews ) for planning, scheduling, and computation on containerized workloads. In my previous role, I heavily used AWS Batch ( /products/aws-batch-reviews ) for running numerous simulations. It includes running different types of computations, such as on-demand and scheduled computations, and creating clusters to optimize memory, CPU, or GPU usage. This flexibility allows processing batch jobs effectively, such as data processing and computation.
We use AWS Batch to manage containerized workloads and dynamic scaling. The solution automatically scales computer resources based on the number of jobs and resource requirements. The solution also integrates with other AWS services.
We use the solution to run scripts for more than 15 minutes. We do not get builds for running the scripts. We can deploy it using containers.
In many ways, it's like using an HPC environment but a lot more flexible. In theory, you could have many different kinds of computing systems and computers, ranging from those geared toward computational speed to larger memory machines or GPU machines. The idea is to break your computational jobs into smaller jobs that can be run on multiple machines. Any of these virtual machines will orchestrate all of the computation, send out jobs to different machines, wait for them to be done, and then run the next process in the sequence. It's simply a way to run multiple processes on multiple machines in the cloud.