{"id":514443,"date":"2022-08-22T15:08:34","date_gmt":"2022-08-22T13:08:34","guid":{"rendered":"https:\/\/www.devoteam.com\/expert-view\/azure-databricks-explained\/"},"modified":"2025-02-18T10:35:53","modified_gmt":"2025-02-18T09:35:53","slug":"azure-databricks-explained","status":"publish","type":"expert-view","link":"https:\/\/devoteam.info\/en-se\/expert-view\/azure-databricks-explained\/","title":{"rendered":"Azure Databricks Explained"},"content":{"rendered":"\n
\"\"<\/figure>\n\n\n\n

Working in the cloud means you\u2019ll have to be familiar with a lot of services, terminologies, and buzzwords. In this article, I\u2019d like to explain one of the most popular analytical services used in the cloud: Databricks.<\/p>\n\n\n\n

What does Databricks do?<\/h2>\n\n\n\n

Databricks<\/a> is a Software-as-a-Service data platform that can be used for almost all your (big) data needs. It\u2019s a very scalable data tool mostly used for data processing & analysis.<\/p>\n\n\n\n

It\u2019s being used in a wide range of industries as its strengths lie in its flexibility and performance. Whether you\u2019re a data scientist, engineer, analyst (or not), understanding Databricks will prove to be helpful for anyone working with data. <\/p>\n\n\n\n

Lastly, as it\u2019s built for the cloud, it integrates very well and runs the same on either Azure<\/a>, AWS<\/a>, or GCP<\/a>. Today we\u2019ll focus on Azure<\/a> Databricks.<\/p>\n\n\n\n

How often does Devoteam work with Azure Databricks?<\/h2>\n\n\n\n

I\u2019ve been working with Azure Databricks<\/a> on a daily basis for my clients for ~3 years now. Since then, I have only become more enthusiastic with it.<\/p>\n\n\n\n

Originally Databricks was a data tool mostly aimed at data scientists and Scala developers (which is still the most popular language) but it has gradually moved to become a larger data platform that integrates amazingly with Azure. You can use it for machine learning, batch processing, streaming, ETL pipelines, data governance, orchestrating, and more. <\/p>\n\n\n\n

It\u2019s also at the forefront of the \u2018lake house\u2019 architecture, which we\u2019ll dedicate a separate blog topic for. In the last months I\u2019ve been experimenting with Databricks SQL – the \u2018new\u2019 environment for data analysts – and the Unity Catalog – Databricks soon-to-be-released data governance solution.<\/p>\n\n\n\n

Even though I\u2019ve developed quite some applications with it and obtained my first certifications, I still feel like there\u2019s much more to explore. Partially it\u2019s because of the wide range of scenarios in which you could use Databricks – more about this later- but also because Databricks keeps releasing new updates and moving forwards. All in all, the direction towards an multifunctional transparent data platform built on open-source is exciting.<\/p>\n\n\n\n

Why is Databricks promising for a technical audience?<\/h2>\n\n\n\n

So far it sounds good, right? But how does it work and what makes Databricks stand out from a technical perspective? <\/p>\n\n\n\n

1. Databricks is made for big data<\/h3>\n\n\n\n

Behind the scenes it runs Apache Spark, which is a very fast open-source big data processing framework. Databricks was actually founded by the team who created Apache Spark at UC Berkeley. This framework performs and scales extremely well by using many tricks like in-memory caching and running processes in parallel.<\/p>\n\n\n\n

That\u2019s why it can handle up to PetaBytes (that\u2019s 1000TB) of data, far surpassing its \u2018predecessor\u2019 MapReduce<\/a>. You can simply choose your cluster and the processing power, and then get started. Not to worry, handling millions of records is easy with even the smallest clusters.<\/p>\n\n\n\n

2. Since it\u2019s high-code software it\u2019s flexible<\/h3>\n\n\n\n

You can choose to run the code in 4(!) different languages – Scala, Python, SQL,or R – in the same notebook. The performance is all equal as these are just a layer on top of the Apache Spark framework. You can easily create your own functions, use libraries, etc. Databricks comes installed with a lot of the libraries pre-installed and installing packages is easy.<\/p>\n\n\n\n

This means you do not have the limitations of low-code\/no-code data processing tools such as Azure Data Factory \/ AWS Glue<\/a> where you\u2019re dependent on the provided functionality.<\/p>\n\n\n\n

3. Ease of collaboration and administration<\/h3>\n\n\n\n

These two important elements also have to be mentioned, you can work with multiple users simultaneously on the same notebook, leave comments, and jump back to previous versions. The different types of users – data analysts, engineers, and scientists – are also divided into separated workspaces if you want, making administration relatively hassle-free. And for the DevOps professionals, yes you can integrate it with your CI\/CD pipelines, docker, and git.<\/p>\n\n\n\n

Why is Azure Databricks good for businesses? <\/h2>\n\n\n\n

I hope by now you\u2019re starting to get an idea of what\u2019s possible with Databricks; but let me show you some examples of use cases of how we\u2019ve used Databricks at a few of my clients. Note that all these use cases were cost-efficient as you pay only for what you use and you can automatically turn off clusters when not used for x<\/em> minutes.<\/p>\n\n\n\n