{"id":494196,"date":"2024-04-26T06:51:19","date_gmt":"2024-04-26T04:51:19","guid":{"rendered":"https:\/\/www.devoteam.com\/success-story\/leading-manufacturing-company-advanced-in-their-data-maturity-journey-with-the-adoption-of-lakehouse-technology\/"},"modified":"2025-04-04T14:31:16","modified_gmt":"2025-04-04T12:31:16","slug":"leading-manufacturing-company-advanced-in-their-data-maturity-journey-with-the-adoption-of-lakehouse-technology","status":"publish","type":"success-story","link":"https:\/\/devoteam.info\/cz\/success-story\/leading-manufacturing-company-advanced-in-their-data-maturity-journey-with-the-adoption-of-lakehouse-technology\/","title":{"rendered":"Lakehouse adoption in a leading Manufacturing company"},"content":{"rendered":"\n

The customer <\/strong><\/p>\n\n

Our European customer initially focused on consumer electronics before transitioning to healthcare technology and divesting its other divisions. The company is listed in both NYSE and European Stock Exchange. The primary objectives of the customers include optimizing the supply-chain, improving the quality of business operations, enhancing\u00a0operational efficiencies, and reducing cost.\u00a0<\/p>\n\n

The context<\/strong><\/strong><\/strong><\/p>\n\n

Databricks to build the foundation of data platform<\/strong><\/p>\n\n

Before delving into the details of the most recent and fascinating challenge, let’s first establish some context and review past activities. Initially, the organization opted for utilizing the data warehouse capabilities and technologies natively provided by their hyper-scalers. This proved to be quite expensive given the multi-cloud ecosystem while still not satisfying all the needs of the end users. After a thorough selection process, Databricks<\/a> was chosen as the technology to build the foundation of their data platform: transforming and consuming data in a Lakehouse implementation with Delta Lake as the focal point.<\/p>\n\n

The creation of valuable insights in the supply-chain domain lacked<\/strong><\/p>\n\n

The main and significant challenge they faced was the timely access to data from their extensive SAP-dominated IT environment.\u00a0Specifically, the absence of real-time capabilities and a limited extraction window hampered the creation of valuable insights in the supply-chain domain. This obstacle was overcome by introducing a change data capture tool that enabled\u00a0near real-time replication of SAP data.<\/p>\n\n

Following the successful ingestion of SAP data, Devoteam assisted the customer in significantly reducing costs through the implementation of best practices, such as regularly vacuum <\/em>and optimize <\/em>the delta tables, and more importantly migrating some workloads to more suitable infrastructure; leading to an impressive 80% saving on the specific bronze to silver workload.<\/p>\n\n

The challenges<\/strong><\/strong><\/strong><\/p>\n\n

Monitoring and observability with Overwatch<\/strong><\/p>\n\n

We already migrated our customer to their new data platform. The next step was monitoring and observability in order to decrease TCO. Since the usage of Databricks grew quite exponentially, it was soon pretty clear that some monitoring was required. While Overwatch<\/a> served as the best solution back in 2022, Databricks System tables <\/em>now offer most of the necessary information to perform cost-saving measures and facilitate auditing; System tables indeed provide all the required information in a simplified manner without the extra components required for Overwatch.<\/p>\n\n

The pitfalls of data mesh<\/strong><\/p>\n\n

At this point in time, the data landscape is robust, with flexible data ingestion and the adoption of the data mesh paradigm enables different domains to generate insights. Like any approach, data mesh can introduce some pitfalls,\u00a0especially in the early adoption phases when the focus is on generating insights and value; it is not uncommon for data silos to reemerge. For instance, two different domains can rush into the creation of gold tables using the same information from the silver layer leading to a situation similar to the one depicted below:<\/p>\n\n

\"\"<\/figure>\n\n

It is quite clear from the context that both the Red and Blue teams may have inadvertently produced duplicated datasets, consequently inflating costs for both storage and computational resources.<\/p>\n\n

The solution<\/strong><\/p>\n\n

The image above offers insights into addressing this issue:\u00a0data lineage.<\/strong> Data lineage indeed visually depicts how data flows across different datasets. While Databricks with Unity catalog provides data lineage per table, encompassing only the downstream and upstream tables, a complete lineage of all the datasets is required to grasp the broader context and identify potential duplicate datasets. To achieve this objective, we leveraged a new feature from Databricks: the system tables<\/em>, particularly the access <\/em>schema that includes table and column lineage.<\/p>\n\n

Using the information in the system tables, we successfully created several entities including:<\/p>\n\n