{"id":514683,"date":"2019-11-07T08:28:00","date_gmt":"2019-11-07T07:28:00","guid":{"rendered":"https:\/\/www.devoteam.com\/expert-view\/batch-processing-in-the-cloud\/"},"modified":"2019-11-07T08:28:00","modified_gmt":"2019-11-07T07:28:00","slug":"batch-processing-in-the-cloud","status":"publish","type":"expert-view","link":"https:\/\/devoteam.info\/be\/expert-view\/batch-processing-in-the-cloud\/","title":{"rendered":"Batch processing in the Cloud"},"content":{"rendered":"

Business is scaling \u2013 up, changing their sizes \u2013 and out, increasing the number of branches. If we are speaking about IT infrastructure in growing businesses, the situation is similar. The number of servers (i.e. cores, RAM, disks) is growing rapidly and business is looking for new solutions for empowering this growth.<\/p>\n\n\n

The first solution is to scale up existing infrastructure \u2013 the servers company bought and maintain, then look for new \u201cboxes\u201d to mount in old cases. But DC capacity will end somewhere (and investment budget also). At this point the needed capacity exceeds owned infrastructure. What then? The second scenario. Lift and Shift.<\/p>\n\n\n\n

<\/p>\n\n\n\n

The business would like to move to the cloud, in the hope that the provider can provide a bigger or (cloud sales love this) infinite capacity. Of course it is both not true. Cloud instances (VMs) are limited in totally the same way as Your own \u201cboxes\u201d. It\u2019s because cloud provider is buying the same servers as You can buy directly from Your vendor. But there is one (not so little) difference. Real cloud services provider has a scale. Instead of scale up, infrastructure need to be scaled out. So, if You want more computing power, stop choosing bigger instances \u2013 just start multiplying them.<\/p>\n\n\n\n

The bad thing (and good from larger perspective) is to re-architect your workload to enable it to be distributed across many machines. Your application need to be redesigned to scale out from 1 to 2 or more machines. Next you will be able to scale your business to 3, 10, or 100 servers \u2013 but lift and shift to the Cloud is not only about infrastructure \u2013 it is also about your mind. 100 VMs is a small scale if you want to do something fast \u2013 delivering your product to the customer much faster than your competitors without need to have massive computing power reserve. You can scale out your infrastructure to 100.000 cores and hundreds of terabytes of RAM in minutes and use it only for time you need \u2013 paying only for what you used. This is Cloud.<\/p>\n\n\n\n

A great example of a parallel workloads is report or billing generating procedures in telecommunication companies or banking sector. Rather than calculating 100.000 billings in serial, these billings can be divided up to 100.000 smaller tasks and processed in parallel \u2013 like a batch, without maintaining infrastructure below your workload.<\/p>\n\n\n\n

Batch processing began with mainframe computers. Today, it still plays a central role in business, engineering, science, and other areas that require running lots of automated tasks \u2013 not only processing bills and payroll, but also calculating portfolio risk, designing new products, rendering animated films, testing software, searching for energy, predicting the weather, and finding new cures for disease. There is a tool created for scenarios like this \u2013 Microsoft Azure Batch. With Azure Batch, that power is available to you when you need it, without any capital investment.<\/p>\n\n\n\n

Azure Batch runs the applications that you use on Your workstations and clusters. It is an easy way to Cloud-enable your \u201cexe\u201d files and scripts into hyperscale. Azure Batch provides a queue to receive the work that you want to run and executes your applications. You only need to describe the data that need to be moved to the cloud for processing, how the data should be distributed, what parameters to use for each task, and the command to start the process. Only two elements are mandatory:<\/p>\n\n\n\n

1: Client application\/script that interacts with the Batch and Storage services to execute a parallel workload on compute nodes (virtual machines). It runs on your local workstation or on \u201cmaster node\u201d you configure.<\/p>\n\n\n\n

2: The program that runs on compute nodes in Azure to perform the actual work. In the sample below, TaskApplication.exe parses the text in a file downloaded from Azure Storage (the input file). Then it produces a text file (the output file) that contains a list of the top three words that appear in the input file. After it creates the output file, TaskApplication uploads the file to Azure Storage. This makes it available to the client application for download. TaskApplication runs in parallel on multiple compute nodes in the Batch service.<\/p>\n\n\n\n