{"id":738544,"date":"2025-09-15T12:46:00","date_gmt":"2025-09-15T10:46:00","guid":{"rendered":"https:\/\/www.devoteam.com\/expert-view\/cloud-native-data-platform-product\/"},"modified":"2025-09-15T12:46:00","modified_gmt":"2025-09-15T10:46:00","slug":"cloud-native-data-platform-product","status":"publish","type":"expert-view","link":"https:\/\/devoteam.info\/en-se\/expert-view\/cloud-native-data-platform-product\/","title":{"rendered":"Building a Cloud Native Data Platform Like a Product"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">Building a Cloud Native <a href=\"https:\/\/devoteam.info\/en-se\/services\/data\/\" target=\"_blank\" rel=\"noreferrer noopener\">Data Platform<\/a> isn\u2019t just about tools. With today\u2019s cloud services, you can quickly spin up a working datalake: you just need storage (Amazon S3), a technical data catalog (Glue Data Catalog), and a query engine (Athena). But while building a datalake is relatively easy, <strong>industrialising data processing jobs<\/strong> (extraction, transformation) is another story.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">We tackled this challenge by transforming our datalake into a full-fledged Data Platform. In other words, treating it as a software product, designed with modular architecture and product management best practices, to support our client\u2019s growing portfolio of data use cases.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This reflection led us to three main pillars, which we\u2019ll explore in this article:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>The <strong>Data Processing as a Microservice<\/strong> pattern<br><\/li>\n\n\n\n<li>Building a Cloud Native <strong>Data Platform Framework<br><\/strong><\/li>\n\n\n\n<li>Managing the platform with <strong>software product practices<\/strong><strong><br><\/strong><\/li>\n<\/ol>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-1-data-processing-as-a-microservice\"><strong>1. Data Processing as a Microservice<\/strong><\/h2>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-the-limits-of-a-monolithic-approach\"><strong>The Limits of a Monolithic Approach<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">One model for industrialising data jobs is to centralise everything into a single processing block, made up of:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Compute (e.g., EMR Serverless running Spark)<br><\/li>\n\n\n\n<li>An IAM role defining permissions<br><\/li>\n\n\n\n<li>The code and dependencies, packaged in a Docker image<br><\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">This architecture creates issues quickly:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Cost traceability:<\/strong> All jobs run on the same infrastructure, making it hard to split costs by domain or business use case.<br><\/li>\n\n\n\n<li><strong>Noisy neighbours:<\/strong> A single resource-hungry job can impact all others in production.<br><\/li>\n\n\n\n<li><strong>Scalability:<\/strong> Upgrading infrastructure (say, Spark versioning or code dependencies) affects every job at once, risking a global outage if something breaks.<br><\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-the-microservice-approach\"><strong>The Microservice Approach<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Instead, the <strong>Data Processing as a Microservice<\/strong> pattern creates one independent block (compute + IAM + code) per job. Each job becomes its own \u201cautonomous\u201d microservice.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The benefits are clear:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Costs are traceable<\/strong> by domain or business use case.<br><\/li>\n\n\n\n<li><strong>No more noisy neighbours:<\/strong> one job crashing doesn\u2019t take others down.<br><\/li>\n\n\n\n<li><strong>Independent upgrades:<\/strong> infrastructure can evolve job by job, reducing global risk.<br><\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">And the cost? No higher than the monolithic approach. With serverless compute, resources only incur costs when active (punctually running 50 serverless clusters doesn\u2019t cost more than one).<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-2-building-a-cloud-native-data-platform-framework\"><strong>2. Building a Cloud Native Data Platform Framework<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The microservice pattern has a downside: each new job often means rewriting Terraform, Airflow, and AWS config\u2014creating code duplication.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Our solution was to <strong>factorise this logic into a Data Platform Framework<\/strong>, built on three key components:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>Airflow DAG Generator<\/strong><strong><br><\/strong>Like many teams, we use Airflow for orchestration. Writing DAGs was repetitive, so we automated it. A Python library now generates DAGs from YAML config. Data Engineers describe what they need, and the framework builds the DAG.<br><\/li>\n\n\n\n<li><strong>Pipeline Factory<\/strong><strong><br><\/strong>A Terraform module deploys the full block (infra + IAM role + Docker image) from the YAML config we talked about earlier. This removes IaC duplication and speeds up new pipeline creation.<br><\/li>\n\n\n\n<li><strong>Datalake SDK<\/strong><strong><br><\/strong>A Python SDK gives engineers a toolbox of standard utilities, from which here are some examples:<br>\n<ul class=\"wp-block-list\">\n<li>Automatic secret retrieval (AWS Secrets Manager), to easily access source credentials in the extraction code, for example<br><\/li>\n\n\n\n<li>Simplified Dataframe writes to the lake: all the Data Engineer has to do is to return the Dataframe, the Datalake SDK handles the rest<br><\/li>\n\n\n\n<li>SQL-as-code execution, \u00e0 la dbt: the Data Engineers can give a file containing an SQL query, the Datalake SDK will execute the query and write the result in an output table in the Datalake<br><\/li>\n<\/ul>\n<\/li>\n<\/ol>\n\n\n\n<p class=\"wp-block-paragraph\">With this framework, engineers focus on business logic while the platform handles infra and boilerplate code.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-example-pipeline-configuration\"><strong>Example: Pipeline Configuration<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">To make it more tangible, here\u2019s an example of a data pipeline configuration, interpreted by the Airflow DAG Generator and the Pipeline Factory:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" src=\"https:\/\/devoteam.info\/wp-content\/uploads\/2025\/09\/image-3-573x1024.png\" alt=\"\" class=\"wp-image-720788\"\/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">A YAML config defines:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>The pipeline configuration first defines the <strong>DAG settings<\/strong>, such as the schedule interval (which sets the time and frequency of execution) and the option to specify a Slack tag to be notified when the DAG fails.<br><\/li>\n\n\n\n<li>It then defines the task configuration\u2014for instance, a <strong>CustomLambdaSensor<\/strong>, a custom Operator we developed that encapsulates a Sensor within an AWS Lambda function.<br><\/li>\n\n\n\n<li>This task configuration also includes infrastructure details (in this case, an ECS cluster). The setup depends on the type of infrastructure used: <strong>EMR Serverless<\/strong> for big data jobs or <strong>ECS<\/strong> for smaller ones. Since data volumes vary greatly depending on the source, we implemented a hybrid compute system\u2014giving Data Engineers the option to run distributed processing with Spark or non-distributed processing with Pandas to optimise costs.<br><\/li>\n\n\n\n<li>The configuration also specifies the tables read and written by each task. This enables the Pipeline Factory to generate the corresponding IAM roles with least privilege on data and to build the Airflow DAG (for example, Task A writes to Table 1, Task B reads from Table 1, so Task B depends on Task A).<br><\/li>\n<\/ol>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-example-job-code\"><strong>Example: Job Code<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">To make it more concrete, here\u2019s an example of a data processing job using the Datalake SDK:<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" src=\"https:\/\/devoteam.info\/wp-content\/uploads\/2025\/09\/image-3.png\" alt=\"\" class=\"wp-image-720784\"\/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Using the Datalake SDK:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>The Datalake SDK is bundled as a dependency in every processing job, giving Data Engineers access to a ready-made toolbox of data features.<br><\/li>\n\n\n\n<li>Another feature of the Framework is writing data to the datalake. The Data Engineer simply returns the Dataframe they want to write, and the Datalake SDK handles the upload. This abstraction layer makes it easy to implement underlying functionality without impacting the engineers. For example, we added anomaly detection on ingestion volumes, which checks if the ingested volume is abnormal compared to previous runs and raises an alert if necessary. This feature was deployed without affecting users\u2014they didn\u2019t need to make any changes to their pipelines to benefit from it.<br><\/li>\n<\/ol>\n\n\n\n<p class=\"wp-block-paragraph\">The Datalake SDK has a dual implementation: one part for Spark and another for Pandas, <strong>depending on the infrastructure chosen<\/strong> by the Data Engineer. Since the interface contract is nearly identical between the two, it\u2019s relatively easy for engineers to start a job on ECS and later migrate it to EMR Serverless if they realise the data volume is larger than expected.<br><\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-benefits\"><strong>Benefits<\/strong><\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Abstraction of complexity:<\/strong> less Terraform\/Airflow code, easier hiring, faster pipeline dev, shorter use cases time-to-prod<br><\/li>\n\n\n\n<li><strong>Consistency:<\/strong> shared code ensures uniform practices<br><\/li>\n\n\n\n<li><strong>Centralised scalability:<\/strong> framework upgrades (e.g., switching to Iceberg tables) apply everywhere automatically<br><\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-3-managing-the-platform-like-a-software-product\"><strong>3. Managing the Platform Like a Software Product<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Building such a framework means building <strong>a software product<\/strong>. Its sustainability depends on product management best practices:<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-automated-testing\"><strong>Automated Testing<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">The framework must be automatically tested with each update to ensure that every release can be deployed confidently (giving reasonable assurance that it works). To achieve this, we created a test data pipeline that exercises all the features of the Data Platform. This test pipeline is redeployed and executed with every framework change, and if all tests pass, the release can go out.<br><\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-versioned-releases\"><strong>Versioned Releases<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Releases are designed to ensure the stability of existing production pipelines despite changes to the Data Platform.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Imagine if all pipelines used the latest version of the framework. Every new platform feature would automatically propagate to all production pipelines. And any bug that slipped through functional tests could break all jobs at once.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">To prevent this, the framework publishes <strong>versioned releases<\/strong> (e.g., v1.2.3) and allows pipelines to pin the version they use. This way, a new release doesn\u2019t impact existing pipelines Data Engineers can upgrade when they have bandwidth in order to benefit from the latest framework features.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Now, imagine some pipelines are still using an older version and a bug is discovered. The Data Platform team can release a fix in a new version, but engineers may not have the bandwidth to upgrade immediately, potentially creating a significant migration effort depending on the breaking changes. To help users in this situation, we provide <strong>operational maintenance<\/strong> (MCO)\u2014publishing fixes on demand for older versions without forcing pipeline upgrades. At the same time, to manage the platform team\u2019s workload, we limit the lifespan of old versions by decommissioning outdated releases.<br><\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-stable-interface-contracts\"><strong>Stable Interface Contracts<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">When implementing an abstraction layer, the way users interact with the underlying logic is called an <strong>interface contract<\/strong>. If the Data Platform team changes this interface (for example, renaming a parameter in a Datalake SDK function), Data Engineers must update their code to upgrade, which\u2014if it happens too frequently\u2014can cause frustration. This may lead engineers to delay updates, resulting in the gradual \u201cageing\u201d of pipelines in production.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Therefore, it is essential to build a <strong>robust and stable<\/strong> <strong>interface contract<\/strong> that minimises breaking changes and facilitates version upgrades for users.<br><\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-thoughtful-feature-selection\"><strong>Thoughtful Feature Selection<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">This best practice involves balancing two well-known software principles: <strong>DRY<\/strong> versus <strong>YAGNI<\/strong>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Part of the goal in building the Data Platform Framework was to <strong>share code<\/strong> to avoid redundancy (<strong>DRY \u2013 Don\u2019t Repeat Yourself<\/strong>). Once you start sharing, it can be tempting to abstract everything, \u201cjust in case\u201d it might be useful. But sometimes you realise that what you\u2019ve abstracted is only used in a single pipeline (or worse, not at all), making the added functionality unnecessary (<strong>YAGNI \u2013 You Aren\u2019t Gonna Need It<\/strong>).<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Adding a feature to a framework <strong>increases complexity<\/strong>. Framework complexity makes it:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>less intuitive<\/strong> (for both users and maintainers)<\/li>\n\n\n\n<li><strong>harms maintainability<\/strong> (larger codebase)<\/li>\n\n\n\n<li>and <strong>scalability<\/strong> (harder to add new features).<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">In other words, sometimes <strong>declining to implement a requested feature (or implementing it differently than requested) can be the best choice<\/strong> to ensure the long-term sustainability of the framework.<br><\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-conclusion-should-you-build-your-own-data-platform\"><strong>Conclusion: Should You Build Your Own Data Platform?<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">This may seem like an odd question, considering the title of this article\u2014but it\u2019s critical. Sooner or later, someone will ask: <em>\u201c<\/em><strong><em>Why not just move to a managed solution?<\/em><\/strong><em>\u201d<\/em><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">We see two main axes of reflection:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Data Engineering vs YAML Engineering:<\/strong> Platforms abstract infra complexity, shifting engineers toward configuration and business logic. Depending on team culture, this is a blessing or a frustration. We adapted by evolving job descriptions as the framework matured.<br><\/li>\n\n\n\n<li><strong>Build vs Buy:<\/strong> In 2020, building was often necessary. Market solutions (Databricks, Snowflake, dbt) weren\u2019t as mature. Today, the choice is harder. Building offers control over roadmap, stack, and costs (but requires ongoing investment). Managed platforms reduce run costs but tie you to a vendor\u2019s roadmap and pricing.<br><\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">The only way to make the right call is through <strong>KPIs<\/strong>. They are specific to each context, but here are some examples :<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Time-to-prod<\/strong> evolution for use cases<br><\/li>\n\n\n\n<li><strong>Percentage of rollbacks<\/strong> due to platform bugs<br><\/li>\n\n\n\n<li>Ratio of platform team time spent on <strong>build vs run<\/strong><br><\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">In the end, building gives freedom, buying gives simplicity. There\u2019s no universal answer, but tracking the right metrics will tell you whether to <strong>double down on your custom platform or migrate to managed<\/strong>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Building a Cloud Native Data Platform isn\u2019t just about tools. With today\u2019s cloud services, you can quickly spin up a working datalake: you just need storage (Amazon S3), a technical data catalog (Glue Data Catalog), and a query engine (Athena). But while building a datalake is relatively easy, industrialising data processing jobs (extraction, transformation) is [&hellip;]<\/p>\n","protected":false},"featured_media":81664,"template":"","categories":[2464,2332],"tags":[],"industry":[],"class_list":["post-738544","expert-view","type-expert-view","status-publish","has-post-thumbnail","hentry","category-cloud-native-architecture-en-se","category-data-en-se"],"acf":[],"cards":"\n\t<div class=\"single-post-card\">\n\n\t\t<figure class=\"wp-block-post-featured-image\"><a href=\"https:\/\/devoteam.info\/en-se\/expert-view\/cloud-native-data-platform-product\/\" target=\"_self\" ><img width=\"1920\" height=\"1280\" src=\"https:\/\/devoteam.info\/wp-content\/uploads\/2024\/11\/GettyImages-1311007681_Edited.jpg\" class=\"attachment-post-thumbnail size-post-thumbnail wp-post-image\" alt=\"Building a Cloud Native Data Platform Like a Product\" style=\"aspect-ratio:4\/3;width:100%;object-fit:cover;\" decoding=\"async\" loading=\"lazy\" srcset=\"https:\/\/devoteam.info\/wp-content\/uploads\/2024\/11\/GettyImages-1311007681_Edited.jpg 1920w, https:\/\/devoteam.info\/wp-content\/uploads\/2024\/11\/GettyImages-1311007681_Edited-300x200.jpg 300w, https:\/\/devoteam.info\/wp-content\/uploads\/2024\/11\/GettyImages-1311007681_Edited-1024x683.jpg 1024w, https:\/\/devoteam.info\/wp-content\/uploads\/2024\/11\/GettyImages-1311007681_Edited-768x512.jpg 768w, https:\/\/devoteam.info\/wp-content\/uploads\/2024\/11\/GettyImages-1311007681_Edited-1536x1024.jpg 1536w\" sizes=\"auto, (max-width: 1920px) 100vw, 1920px\" \/><\/a><\/figure>\n\n\t\t\n\t\t<div class=\"wp-block-group is-vertical is-layout-flex wp-container-core-group-is-layout-43282307 wp-block-group-is-layout-flex\">\n\t<p style=\"font-style:normal;font-weight:700\" class=\"has-link-color wp-elements-1 wp-block-lp-post-type has-text-color has-primary-color has-small-font-size\">Expert View<\/p>\n\n\t\t\n\t\t<h3 style=\"font-style:normal;font-weight:400\" class=\"wp-block-post-title has-base-font-size\"><a href=\"https:\/\/devoteam.info\/en-se\/expert-view\/cloud-native-data-platform-product\/\" target=\"_self\" >Building a Cloud Native Data Platform Like a Product<\/a><\/h3><\/div>\n\t\t\n\t<\/div>\n\n","yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v28.4 (Yoast SEO v28.4) - https:\/\/yoast.com\/product\/yoast-seo-premium-wordpress\/ -->\n<title>Building a Cloud Native Data Platform Like a Product | Devoteam<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/devoteam.info\/en-se\/expert-view\/cloud-native-data-platform-product\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Building a Cloud Native Data Platform Like a Product\" \/>\n<meta property=\"og:description\" content=\"Building a Cloud Native Data Platform isn\u2019t just about tools. With today\u2019s cloud services, you can quickly spin up a working datalake: you just need storage (Amazon S3), a technical data catalog (Glue Data Catalog), and a query engine (Athena). But while building a datalake is relatively easy, industrialising data processing jobs (extraction, transformation) is [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/devoteam.info\/en-se\/expert-view\/cloud-native-data-platform-product\/\" \/>\n<meta property=\"og:site_name\" content=\"Devoteam\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"9 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/devoteam.info\\\/en-se\\\/expert-view\\\/cloud-native-data-platform-product\\\/\",\"url\":\"https:\\\/\\\/devoteam.info\\\/en-se\\\/expert-view\\\/cloud-native-data-platform-product\\\/\",\"name\":\"Building a Cloud Native Data Platform Like a Product | Devoteam\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/devoteam.info\\\/en-se\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/devoteam.info\\\/en-se\\\/expert-view\\\/cloud-native-data-platform-product\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/devoteam.info\\\/en-se\\\/expert-view\\\/cloud-native-data-platform-product\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/devoteam.info\\\/wp-content\\\/uploads\\\/2024\\\/11\\\/GettyImages-1311007681_Edited.jpg\",\"datePublished\":\"2025-09-15T10:46:00+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/devoteam.info\\\/en-se\\\/expert-view\\\/cloud-native-data-platform-product\\\/#breadcrumb\"},\"inLanguage\":\"en-SE\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/devoteam.info\\\/en-se\\\/expert-view\\\/cloud-native-data-platform-product\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-SE\",\"@id\":\"https:\\\/\\\/devoteam.info\\\/en-se\\\/expert-view\\\/cloud-native-data-platform-product\\\/#primaryimage\",\"url\":\"https:\\\/\\\/devoteam.info\\\/wp-content\\\/uploads\\\/2024\\\/11\\\/GettyImages-1311007681_Edited.jpg\",\"contentUrl\":\"https:\\\/\\\/devoteam.info\\\/wp-content\\\/uploads\\\/2024\\\/11\\\/GettyImages-1311007681_Edited.jpg\",\"width\":1920,\"height\":1280,\"caption\":\"Purple and blue image with a graph showing a decrease.\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/devoteam.info\\\/en-se\\\/expert-view\\\/cloud-native-data-platform-product\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/devoteam.info\\\/en-se\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Expert View\",\"item\":\"https:\\\/\\\/devoteam.info\\\/en-se\\\/expert-view\\\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"Building a Cloud Native Data Platform Like a Product\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/devoteam.info\\\/en-se\\\/#website\",\"url\":\"https:\\\/\\\/devoteam.info\\\/en-se\\\/\",\"name\":\"Devoteam\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/devoteam.info\\\/en-se\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-SE\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Building a Cloud Native Data Platform Like a Product | Devoteam","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/devoteam.info\/en-se\/expert-view\/cloud-native-data-platform-product\/","og_locale":"en_US","og_type":"article","og_title":"Building a Cloud Native Data Platform Like a Product","og_description":"Building a Cloud Native Data Platform isn\u2019t just about tools. With today\u2019s cloud services, you can quickly spin up a working datalake: you just need storage (Amazon S3), a technical data catalog (Glue Data Catalog), and a query engine (Athena). But while building a datalake is relatively easy, industrialising data processing jobs (extraction, transformation) is [&hellip;]","og_url":"https:\/\/devoteam.info\/en-se\/expert-view\/cloud-native-data-platform-product\/","og_site_name":"Devoteam","twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"9 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/devoteam.info\/en-se\/expert-view\/cloud-native-data-platform-product\/","url":"https:\/\/devoteam.info\/en-se\/expert-view\/cloud-native-data-platform-product\/","name":"Building a Cloud Native Data Platform Like a Product | Devoteam","isPartOf":{"@id":"https:\/\/devoteam.info\/en-se\/#website"},"primaryImageOfPage":{"@id":"https:\/\/devoteam.info\/en-se\/expert-view\/cloud-native-data-platform-product\/#primaryimage"},"image":{"@id":"https:\/\/devoteam.info\/en-se\/expert-view\/cloud-native-data-platform-product\/#primaryimage"},"thumbnailUrl":"https:\/\/devoteam.info\/wp-content\/uploads\/2024\/11\/GettyImages-1311007681_Edited.jpg","datePublished":"2025-09-15T10:46:00+00:00","breadcrumb":{"@id":"https:\/\/devoteam.info\/en-se\/expert-view\/cloud-native-data-platform-product\/#breadcrumb"},"inLanguage":"en-SE","potentialAction":[{"@type":"ReadAction","target":["https:\/\/devoteam.info\/en-se\/expert-view\/cloud-native-data-platform-product\/"]}]},{"@type":"ImageObject","inLanguage":"en-SE","@id":"https:\/\/devoteam.info\/en-se\/expert-view\/cloud-native-data-platform-product\/#primaryimage","url":"https:\/\/devoteam.info\/wp-content\/uploads\/2024\/11\/GettyImages-1311007681_Edited.jpg","contentUrl":"https:\/\/devoteam.info\/wp-content\/uploads\/2024\/11\/GettyImages-1311007681_Edited.jpg","width":1920,"height":1280,"caption":"Purple and blue image with a graph showing a decrease."},{"@type":"BreadcrumbList","@id":"https:\/\/devoteam.info\/en-se\/expert-view\/cloud-native-data-platform-product\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/devoteam.info\/en-se\/"},{"@type":"ListItem","position":2,"name":"Expert View","item":"https:\/\/devoteam.info\/en-se\/expert-view\/"},{"@type":"ListItem","position":3,"name":"Building a Cloud Native Data Platform Like a Product"}]},{"@type":"WebSite","@id":"https:\/\/devoteam.info\/en-se\/#website","url":"https:\/\/devoteam.info\/en-se\/","name":"Devoteam","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/devoteam.info\/en-se\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-SE"}]}},"uagb_featured_image_src":{"full":["https:\/\/devoteam.info\/wp-content\/uploads\/2024\/11\/GettyImages-1311007681_Edited.jpg",1920,1280,false],"thumbnail":["https:\/\/devoteam.info\/wp-content\/uploads\/2024\/11\/GettyImages-1311007681_Edited-150x150.jpg",150,150,true],"medium":["https:\/\/devoteam.info\/wp-content\/uploads\/2024\/11\/GettyImages-1311007681_Edited-300x200.jpg",300,200,true],"medium_large":["https:\/\/devoteam.info\/wp-content\/uploads\/2024\/11\/GettyImages-1311007681_Edited-768x512.jpg",768,512,true],"large":["https:\/\/devoteam.info\/wp-content\/uploads\/2024\/11\/GettyImages-1311007681_Edited-1024x683.jpg",1024,683,true],"1536x1536":["https:\/\/devoteam.info\/wp-content\/uploads\/2024\/11\/GettyImages-1311007681_Edited-1536x1024.jpg",1536,1024,true],"2048x2048":["https:\/\/devoteam.info\/wp-content\/uploads\/2024\/11\/GettyImages-1311007681_Edited.jpg",1920,1280,false]},"uagb_author_info":{"display_name":"julien.lemarchal","author_link":"https:\/\/devoteam.info\/en-se\/author\/"},"uagb_comment_info":0,"uagb_excerpt":"Building a Cloud Native Data Platform isn\u2019t just about tools. With today\u2019s cloud services, you can quickly spin up a working datalake: you just need storage (Amazon S3), a technical data catalog (Glue Data Catalog), and a query engine (Athena). But while building a datalake is relatively easy, industrialising data processing jobs (extraction, transformation) is&hellip;","_links":{"self":[{"href":"https:\/\/devoteam.info\/en-se\/wp-json\/wp\/v2\/expert-view\/738544","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devoteam.info\/en-se\/wp-json\/wp\/v2\/expert-view"}],"about":[{"href":"https:\/\/devoteam.info\/en-se\/wp-json\/wp\/v2\/types\/expert-view"}],"version-history":[{"count":0,"href":"https:\/\/devoteam.info\/en-se\/wp-json\/wp\/v2\/expert-view\/738544\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devoteam.info\/en-se\/wp-json\/wp\/v2\/media\/81664"}],"wp:attachment":[{"href":"https:\/\/devoteam.info\/en-se\/wp-json\/wp\/v2\/media?parent=738544"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devoteam.info\/en-se\/wp-json\/wp\/v2\/categories?post=738544"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devoteam.info\/en-se\/wp-json\/wp\/v2\/tags?post=738544"},{"taxonomy":"industry","embeddable":true,"href":"https:\/\/devoteam.info\/en-se\/wp-json\/wp\/v2\/industry?post=738544"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}