{"id":516410,"date":"2019-04-25T10:31:34","date_gmt":"2019-04-25T08:31:34","guid":{"rendered":"https:\/\/www.devoteam.com\/expert-view\/what-is-google-cloud-data-fusion\/"},"modified":"2019-04-25T10:31:34","modified_gmt":"2019-04-25T08:31:34","slug":"what-is-google-cloud-data-fusion","status":"publish","type":"expert-view","link":"https:\/\/devoteam.info\/lu\/expert-view\/what-is-google-cloud-data-fusion\/","title":{"rendered":"What is Google Cloud Data Fusion?"},"content":{"rendered":"
\n

Cloud Data Fusion by Google Cloud is the brand new, fully-managed data engineering product\u00a0within Google Cloud Platform. It will help users to efficiently build and manage ETL\/ELT data pipelines. It intents to shift the focus from code (where data engineers can spend lot of days\/weeks building connectors from a source to a sink) to a focus on insights and action. Built on top of the open-source project\u00a0CDAP<\/a>, it leverages a convenient user interface for building data pipelines in a drag and drop manner.<\/span><\/strong><\/p>\n

\"Google<\/span><\/strong><\/p>\n

Data Fusion is one of\u00a0Google’s major novelties concerning data analytics<\/a>, as announced at Google Cloud Next ’19. It comes at a time where companies struggle to deal with a huge amount of data spread across many data sources, and to fuse them into a central data warehouse<\/strong>. The key challenges of integrating all these data are as follows:<\/p>\n<\/div>\n

<\/p>\n

\"Data<\/figure>\n

Data Fusion is addressing these challenges by making it extremely easy to move data around, with two main focuses:<\/p>\n

    \n
  1. build data pipeline without writing any code<\/strong>: as Data Fusion is built on top of the open-source CDAP project, it already comes with more than 100 connectors and it is constantly growing. Building a pipeline between a source and sink requires therefore only a few clicks.<\/li>\n
  2. Do transformation without writing any code: <\/strong>Data Fusion comes with a set of built-in transformations that you can seamlessly apply to your data.<\/li>\n<\/ol>\n

    The following screenshot shows the interface with a simple pipeline<\/strong>. First step is the connector to the raw database, then there is a wrangling step that does some transformation on a set of columns, and finally the data is sent to two sinks: BigQuery for analytics purposes and Cloud Storage for backup of the data.<\/strong><\/p>\n

    \"Data<\/p>\n

    Some of the other relevant features of Data Fusion<\/strong> are these (described by one of the early adopters):<\/p>\n