Introduction to Batch Processing with Azure Batch Services

Lixar Tech

In this day and age of cloud computing where everyone wants power and performance from their services, there are times when large amounts of data need to be processed in batches or CPU-intensive tasks need to be performed. Typically, this is done by running a service that starts a loop and kicks off background tasks that execute these batch or CPU-intensive tasks. In this DIY scenario, you would be responsible for maintaining the infrastructure (hardware and software) and making sure these services are running on at their optimal levels.


Azure Batch is a managed service offered by Microsoft as part of their Azure platform that takes care of the infrastructure needed for running large-scale CPU-intensive processes. These processes run on a managed collection of virtual machines that are scaled automatically to meet the resources necessary to run the tasks and include options for scheduling frequent jobs. The only thing you need to worry about is building the logic that the service will run and uploading the executable and input files it needs to Azure storage. Azure Batch manages the rest!

Azure Batch Workflow

The typical Azure Batch workflow scenario is as follows:

  1. Upload the relevant data necessary to run the tasks (executables and input files) to Azure storage. These files will be downloaded from Azure storage to the compute nodes when the tasks run.
  2. Set up and configure a pool of compute nodes. When a task is run it gets assigned to a compute node from this pool. When setting up the pool you have the options that include specifying the operating system as well as the size of the pool and the VM.
  3. Set up and configure a job to manage a collection of tasks/work items.
  4. Set up and configure a task and assign it to a job. The task defines the computational work to be done which will consist of retrieving the files stored in Azure storage.

The following image outlines the above workflow.

Azure Batch Workflow

Azure Batch Account

In order to begin using Azure Batch Services, you must first configure a Batch Account. You can do this using one of the following:

  • Azure Portal
  • Azure PowerShell Cmdlets
  • Batch Management .NET Library

Creating Batch Account using PowerShell

The following sample script outlines the provisioning of a new Batch Account:

Creating Batch Account using PowerShell

Creating a Batch Account using the Batch Management .NET Library

The Batch Management .NET Library is a Software Development Kit (SDK) that provides a set of Create-Update-Delete (CRUD) operations to help automate the maintenance of your Batch Account. The following code snippet shows the creation of a Batch Account using the Batch Management .NET library:

Creating a Batch Account using the Batch Management .NET Library

Authoring Azure Batch Services

When developing Azure Batch Solutions there are several tools to choose from:

  • Batch .NET: A CLR library available for implementing Azure Batch Solutions
  • Batch PowerShell Cmdlets: A set of PowerShell commands for managing Batch Accounts and solutions
  • Batch REST: A REST API offered by Microsoft for managing Batch Accounts and services. The Batch .NET library, Batch PowerShell cmdlets, and Batch Management.NET libraries all use the REST API behind the scenes to interact with the Batch Service.

In order to use these tools to interact with the Batch Service, you need to provide the account name, URL, and keys as credentials. These can be retrieved using the Batch Management .NET libraries, Batch PowerShell cmdlets, or from the Azure Batch Pane in the Azure Portal as shown below:

Azure Portal

The following sample shows how to instantiate and use the Batch .NET library to submit a simple job to the Batch Services and monitor its output. Firstly, you will need to download the Batch .NET library from NuGet.

nstantiate and use the Batch .NET library

Although the above snippet is very basic and simple in nature, it provides the building blocks needed to get started using Azure Batch Services.