Rationale

We use Batch for running batch processing jobs in the cloud. The main reasons why we chose it over other alternatives are the following:

• It is SaaS (software as a service), so we do not need to manage any infrastructure directly.
• It is free, so we only have to pay for the Elastic Compute Cloud (EC2) machines we use to process workloads.
• It complies with several certifications from ISO and CSA. Many of these certifications are focused on granting that the entity follows best practices regarding secure cloud-based environments and information security.
• We can monitor job logs using CloudWatch.
• The jobs are highly resilient, which means they rarely go irresponsive. This feature is very important when jobs take several days to finish.
• It supports EC2 spot instances, which considerably decreases EC2 costs.
• All its settings can be written as code using Terraform.
• We can use Nix to queue jobs easily.
• It supports priority-based queuing, which allows us to prioritize jobs by assigning them to one queue or another.
• It supports automatic retries of jobs.
• It integrates with Identity and Access Management (IAM), allowing us to keep a least privilege approach regarding authentication and authorization.
• EC2 workers running jobs can be monitored using CloudWatch.

Alternatives

GitLab CI

We used GitLab CI before implementing Batch. We migrated because GitLab CI is not intended to run scheduled jobs that take many hours, often resulting in jobs becoming irresponsive before they could finish, mainly due to disconnections between the worker running the job and the GitLab CI Bastion. On top of this, GitLab CI has a limit on the number of schedules per project, and running thousands of jobs puts a lot of pressure on the GitLab coordinator and the GitLab CI Bastion.

Buildkite

https://buildkite.com

Pros:

• Handles submission of duplicated jobs
• Gives us logging, monitoring, and stability measurements out-of-the-box
• We can separate costs by having different queues (associated to different deployments)
• Notifications out-of-the-box to email and others
• Support pipelines out-of-the-box
• They have an API to query information about past jobs on a pipeline and trigger new builds, which is much more flexible than Batch's API

Cons:

• Much more expensive.

Kubernetes Jobs

See Jobs in Kubernetes documentation.

Pros:

• Allows better separation of costs.

Cons:

• It requires manually kick-starting a build, because it doesn't listen automatically to the queue like batch does.

Usage

We use Batch for running

• Production background schedules for all our components.
• ARM background tasks like cloning roots and refreshing targets of evaluation.