Senior Data Engineer at a tech services company with 5,001-10,000 employees
MSP
Top 20
2025-04-24T14:49:23Z
Apr 24, 2025
In my project, I worked on an enhanced backup of some assets in AWS ( /products/amazon-aws-reviews ) QuickSight ( /products/amazon-quicksight-reviews ). AWS ( /products/amazon-aws-reviews ) QuickSight ( /products/amazon-quicksight-reviews ) has a rudimentary 30-day buffer for restoring deleted assets but lacks other versioning options. To address this, I back up data to JSON files and run an AWS Batch ( /products/aws-batch-reviews ) process hourly to detect any changes during business hours. Additionally, a longer process runs every few months to check the data's integrity and restore it to its original state, which takes a couple of hours due to the size of the QuickSight installation.
I use AWS Batch ( /products/aws-batch-reviews ) for my data activities, including building the data science pipeline, processing, storing data, and performing analytics.
Software Engineering Manager – Digital Production Optimization at Yara International ASA
Real User
Top 5
2025-03-10T11:04:57Z
Mar 10, 2025
I use AWS Batch ( /products/aws-batch-reviews ) for planning, scheduling, and computation on containerized workloads. In my previous role, I heavily used AWS Batch ( /products/aws-batch-reviews ) for running numerous simulations. It includes running different types of computations, such as on-demand and scheduled computations, and creating clusters to optimize memory, CPU, or GPU usage. This flexibility allows processing batch jobs effectively, such as data processing and computation.
We use AWS Batch to manage containerized workloads and dynamic scaling. The solution automatically scales computer resources based on the number of jobs and resource requirements. The solution also integrates with other AWS services.
In many ways, it's like using an HPC environment but a lot more flexible. In theory, you could have many different kinds of computing systems and computers, ranging from those geared toward computational speed to larger memory machines or GPU machines. The idea is to break your computational jobs into smaller jobs that can be run on multiple machines. Any of these virtual machines will orchestrate all of the computation, send out jobs to different machines, wait for them to be done, and then run the next process in the sequence. It's simply a way to run multiple processes on multiple machines in the cloud.
AWS Batch enables developers, scientists, and engineers to easily and efficiently run hundreds of thousands of batch computing jobs on AWS. AWS Batch dynamically provisions the optimal quantity and type of compute resources (e.g., CPU or memory optimized instances) based on the volume and specific resource requirements of the batch jobs submitted. With AWS Batch, there is no need to install and manage batch computing software or server clusters that you use to run your jobs, allowing you to...
In my project, I worked on an enhanced backup of some assets in AWS ( /products/amazon-aws-reviews ) QuickSight ( /products/amazon-quicksight-reviews ). AWS ( /products/amazon-aws-reviews ) QuickSight ( /products/amazon-quicksight-reviews ) has a rudimentary 30-day buffer for restoring deleted assets but lacks other versioning options. To address this, I back up data to JSON files and run an AWS Batch ( /products/aws-batch-reviews ) process hourly to detect any changes during business hours. Additionally, a longer process runs every few months to check the data's integrity and restore it to its original state, which takes a couple of hours due to the size of the QuickSight installation.
I use AWS Batch ( /products/aws-batch-reviews ) for my data activities, including building the data science pipeline, processing, storing data, and performing analytics.
I use AWS Batch ( /products/aws-batch-reviews ) for planning, scheduling, and computation on containerized workloads. In my previous role, I heavily used AWS Batch ( /products/aws-batch-reviews ) for running numerous simulations. It includes running different types of computations, such as on-demand and scheduled computations, and creating clusters to optimize memory, CPU, or GPU usage. This flexibility allows processing batch jobs effectively, such as data processing and computation.
We use AWS Batch to manage containerized workloads and dynamic scaling. The solution automatically scales computer resources based on the number of jobs and resource requirements. The solution also integrates with other AWS services.
We use the solution to run scripts for more than 15 minutes. We do not get builds for running the scripts. We can deploy it using containers.
In many ways, it's like using an HPC environment but a lot more flexible. In theory, you could have many different kinds of computing systems and computers, ranging from those geared toward computational speed to larger memory machines or GPU machines. The idea is to break your computational jobs into smaller jobs that can be run on multiple machines. Any of these virtual machines will orchestrate all of the computation, send out jobs to different machines, wait for them to be done, and then run the next process in the sequence. It's simply a way to run multiple processes on multiple machines in the cloud.