{"id":518168,"date":"2024-11-20T15:28:16","date_gmt":"2024-11-20T14:28:16","guid":{"rendered":"https:\/\/www.devoteam.com\/expert-view\/fivetran-connectors-google-cloud-integration\/"},"modified":"2024-11-20T15:28:16","modified_gmt":"2024-11-20T14:28:16","slug":"fivetran-connectors-google-cloud-integration","status":"publish","type":"expert-view","link":"https:\/\/devoteam.info\/sk\/expert-view\/fivetran-connectors-google-cloud-integration\/","title":{"rendered":"Leveraging Google Cloud Functions to Create a Powerful Fivetran Connector to integrate custom APIs"},"content":{"rendered":"<figure class=\"wp-block-post-featured-image\"><img loading=\"lazy\" decoding=\"async\" width=\"2560\" height=\"1920\" src=\"https:\/\/devoteam.info\/wp-content\/uploads\/2024\/10\/GettyImages-1693803686.jpg\" class=\"attachment-post-thumbnail size-post-thumbnail wp-post-image\" alt=\"\" style=\"object-fit:cover;\" srcset=\"https:\/\/devoteam.info\/wp-content\/uploads\/2024\/10\/GettyImages-1693803686.jpg 2560w, https:\/\/devoteam.info\/wp-content\/uploads\/2024\/10\/GettyImages-1693803686-300x225.jpg 300w, https:\/\/devoteam.info\/wp-content\/uploads\/2024\/10\/GettyImages-1693803686-1024x768.jpg 1024w, https:\/\/devoteam.info\/wp-content\/uploads\/2024\/10\/GettyImages-1693803686-768x576.jpg 768w, https:\/\/devoteam.info\/wp-content\/uploads\/2024\/10\/GettyImages-1693803686-1536x1152.jpg 1536w, https:\/\/devoteam.info\/wp-content\/uploads\/2024\/10\/GettyImages-1693803686-2048x1536.jpg 2048w\" sizes=\"auto, (max-width: 2560px) 100vw, 2560px\" \/><\/figure>\n\n\n<p class=\"wp-block-paragraph\"><em>This<\/em> article <em>was written by Harold Snyers \u2013 Senior ML &amp; Data Engineer I Google Cloud Business Unit @ Devoteam Belgium.<\/em><br><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Today, businesses rely on many data sources, including custom APIs. However, extracting and transforming data from APIs for analysis can be complex and time-consuming. This is where Fivetran comes in, offering a powerful solution for seamless data integration. While Fivetran supports hundreds of pre-built connectors, it also empowers you to extract data from unsupported sources using Google Cloud Functions as connectors. This tutorial dives deep into setting up a production-ready environment with Terraform for managing infrastructure as code (IaC).<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-what-is-fivetran\"><strong>What is Fivetran?<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Fivetran is a cloud-based, fully managed ETL (Extract, Transform, Load) service that automates the process of moving data from various sources, including databases, applications, and custom APIs, to your data warehouse. It simplifies data ingestion by handling schema management, data transformations, and scheduling, allowing you to focus on analysing your data and deriving insights.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-leveraging-google-cloud-functions-as-fivetran-connectors\"><strong>Leveraging Google Cloud Functions as Fivetran Connectors<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Fivetran&#8217;s robust capabilities extend to integrate custom APIs through Google Cloud Functions. These functions act as bridges, fetching data from your API, transforming it as needed for Fivetran processing, and delivering it for loading into BigQuery.<span id=\"docs-internal-guid-d4e6367a-7fff-f435-4909-d13e41a61a56\"><\/span><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Simplified Data Ingestion: <\/strong>Automate data extraction and transformation from your custom API, eliminating manual coding and maintenance. Leverage Fivetran&#8217;s built-in monitoring for streamlined oversight.<\/li>\n\n\n\n<li><strong>Scalability and Flexibility: <\/strong>Google Cloud Functions offer serverless execution, scaling automatically to handle varying data volumes from your API. Fivetran will also automatically handle server\/request time-outs when reaching a request limit.<\/li>\n\n\n\n<li><strong>Enhanced Data Quality: <\/strong>Leverage Fivetran&#8217;s transformation capabilities within the Cloud Function or in Fivetran itself to ensure data consistency and accuracy before loading into BigQuery, or handle this later in your pipeline.<\/li>\n\n\n\n<li><strong>Infrastructure as Code (IaC) with Terraform:<\/strong> Terraform allows you to manage your entire infrastructure, including Google Cloud resources and Fivetran configurations. This enables version control, automated deployments, and easier infrastructure management across environments (development, staging, and production).<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-setting-up-the-solution\"><strong>Setting Up the Solution<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Now that we have set the context, let\u2019s explore how you can set this infrastructure up for this data integration pipeline. In my experience, I always prefer to start by setting up everything in the UI, making sure the resources can communicate with each other, and then developing the necessary code infrastructure for it to be production-ready.<\/p>\n\n\n\n<h5 class=\"wp-block-heading\" id=\"h-develop-your-google-cloud-function\">Develop your Google Cloud Function:<span id=\"docs-internal-guid-6887d2c8-7fff-3733-5aee-40ff39931fc3\"><\/span><\/h5>\n\n\n\n<p class=\"wp-block-paragraph\">First, let\u2019s dive deeper into setting up the cloud function. Google Cloud functions support multiple languages (Python, Go and NodeJS.) Choose the language that best suits your requirements. As per the setup tutorial of Fivetran written here, the first task is to develop the Cloud Function code. On Google Cloud, a simple cloud function has a request parameter with specific parameters built in and some custom parameters. Here is an example request for a cloud function. Fivetran will automatically fill in these parameters except for the state.<span id=\"docs-internal-guid-5e76cea3-7fff-f402-dc70-c4fdffdfc9d2\"><\/span><\/p>\n\n\n\n<pre class=\"wp-block-code has-gray-light-background-color has-background\"><code>\n{\n   <mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\"> <strong>'state'<\/strong>: <\/mark><mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-main-color\">{<\/mark><mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\">\n       <strong> 'current_sync_id'<\/strong><\/mark><mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-main-color\">:<\/mark><mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\"> <strong>'bca62c36-*-0693e1230bee'<\/strong>, \n       <strong> 'table_state'<\/strong><\/mark><mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-main-color\">:<\/mark><mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\"> <\/mark><mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-main-color\">{<\/mark><mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\">\n            <strong>'CUSTOMER'<\/strong>: <\/mark><mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-main-color\">{<\/mark><mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\">\n                <strong>'current_sync_unix'<\/strong>: <\/mark><mark style=\"background-color:rgba(0, 0, 0, 0);color:#691aaf\" class=\"has-inline-color\"><strong>None<\/strong><\/mark><mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-main-color\">,<\/mark><mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\"> \n                <strong>'has_more<\/strong>': <\/mark><mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-main-color\">False, <\/mark><mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\">\n                <strong>'last_status'<\/strong><\/mark><mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-main-color\">: {<\/mark><mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\"><strong>'message'<\/strong>: <\/mark><mark style=\"background-color:rgba(0, 0, 0, 0);color:#691aaf\" class=\"has-inline-color\"><strong>None<\/strong><\/mark><mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\">,<strong> 'type': 'OK'<\/strong><\/mark><mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-main-color\">}, <\/mark><mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\">\n                <strong>'next_page_url'<\/strong><\/mark><mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-main-color\">: <\/mark><mark style=\"background-color:rgba(0, 0, 0, 0);color:#691aaf\" class=\"has-inline-color\"><strong>None<\/strong><\/mark><mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-main-color\">,<\/mark><mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\"> \n                <strong>'next_sync_unix'<\/strong><\/mark><mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-main-color\">: <\/mark><mark style=\"background-color:rgba(0, 0, 0, 0);color:#bf0f23\" class=\"has-inline-color\"><strong>1727389295<\/strong><\/mark><mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-main-color\">, <\/mark><mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\">\n                <strong>'rows_read'<\/strong><\/mark><mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-main-color\">:<\/mark><mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\"> <\/mark><mark style=\"background-color:rgba(0, 0, 0, 0);color:#bf0f23\" class=\"has-inline-color\"><strong>0<\/strong><\/mark><mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\">\n      <\/mark><mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-main-color\">      }, \n        }\n    }, <\/mark><mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\">\n    <strong>'secrets'<\/strong><\/mark>:<mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\"> <\/mark><mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-main-color\">{}, <\/mark><mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\">\n   <strong> 'agent'<\/strong><\/mark><mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-main-color\">:<\/mark><mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\"> <strong>'Fivetran Google Cloud Functions Connector\/&lt;id>\/&lt;name>'<\/strong>, \n  <strong>  'sync_id'<\/strong><\/mark><mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-main-color\">:<\/mark><mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\"> <strong>'93d1e969-*-5e5a68d8f4b0'<\/strong><\/mark>\n} \n\n\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">As you see above, we received a state from Fivetran, which we need to keep track of where we are in sync with our custom API. In the response to Fivetran, we need to respect some conventions on what to return, which Fivetran will keep in mind for the next requests it sends to sync the data. Here is a basis for the Fivetran response in our function.<\/p>\n\n\n\n<pre class=\"wp-block-code has-gray-light-background-color has-background\"><code>\nfivetran_response = {\n    <mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\"><strong>\"state\"<\/strong><\/mark>: {\n        <mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\"><strong>\"table_state\"<\/strong><\/mark>: {},\n        <mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\"><strong>\"current_sync_id\"<\/strong><\/mark>: req.get(<mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\"><strong>\"sync_id\"<\/strong><\/mark>, req&#91;<mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\"><strong>\"state\"<\/strong><\/mark>].get(<mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\"><strong>\"current_sync_id\"<\/strong><\/mark>)),\n    },\n    <mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\"><strong>\"insert\"<\/strong><\/mark>: {},\n    <mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\"><strong>\"schema\"<\/strong><\/mark>: {},\n    <mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\"><strong>\"hasMore\"<\/strong><\/mark>: False,\n} \n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">With that, you can build your code to request the data from your API. As with other Fivetran connectors, we set a schedule for the sync from the source to Fivetran. This is set up because Fivetran will manage when to sync from the API and send the request to the cloud function. The cloud function can process and return the data to Fivetran in the correct format. You can read everything you need about formatting your response to Fivetran<a href=\"https:\/\/fivetran.com\/docs\/connectors\/functions\/google-cloud-functions\"> here<\/a>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Nevertheless, what is important here is to keep track of whether you were able to retrieve all data during your sync to your source API or if additional data still needs to be synced. If that is the case, by setting the hasMore variable to True, you tell Fivetran, it needs to send another request as the sync is not done. Of course, then you set up how you manage your API sync. In an example case I developed, some API calls had more data than others, so it was essential to keep track of each table (endpoint), what stage you were at, and if an additional request was needed for it.&nbsp;&nbsp;<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-fivetran\"><strong>Fivetran<\/strong><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Let\u2019s first test the connection with the Fivetran UI.&nbsp;<\/p>\n\n\n\n<h5 class=\"wp-block-heading\" id=\"h-create-the-fivetran-connector\">Create the Fivetran Connector:<\/h5>\n\n\n\n<p class=\"wp-block-paragraph\">Within the Fivetran web interface, navigate to the Connectors section and select &#8220;Custom Function.&#8221; Provide a name and description for your connector. Configure the connector to point to the deployed URL of your Google Cloud Function. And finally, specify the authentication method used by your Cloud Function to access the custom API (e.g., API key, OAuth).<\/p>\n\n\n\n<h5 class=\"wp-block-heading\" id=\"h-set-up-the-bigquery-destination\">Set Up the BigQuery Destination:<\/h5>\n\n\n\n<p class=\"wp-block-paragraph\">In the Fivetran interface, define a destination for the extracted data. Select BigQuery as the destination type and configure the connection details for your BigQuery project and dataset.<\/p>\n\n\n\n<h5 class=\"wp-block-heading\" id=\"h-infrastructure-as-code-iac-with-terraform\">Infrastructure as Code (IaC) with Terraform:<\/h5>\n\n\n\n<p class=\"wp-block-paragraph\">Best practices when working with the Cloud are to use Terraform or another Infrastructure as a Code framework. We will set up the infrastructure with Terraform. With Terraform, we can manage the creation and deployment of resources in Google Cloud Platform (GCP) and Fivetran. This includes your Cloud Function, BigQuery dataset, and Fivetran connector configuration (using the Fivetran API). Terraform allows you to version control your infrastructure code and automate deployments across environments.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">For this post, I will not go through the whole infrastructure setup. It is recommended that you set up some CI\/CD for your Cloud Function Code so that when changes are needed, these are directly updated upon merging your code. So, let\u2019s dive deeper into the setup of your Fivetran connector and destination.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">First, you need to define your Fivetran group to let Fivetran know where to deploy your connectors, destinations, sources, etc. You can do this with the resource fivetran_group.<\/p>\n\n\n\n<pre class=\"wp-block-code has-gray-light-background-color has-background\"><code>\n<mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\"><strong>resource \"fivetran_group\" \"data_warehouse_api_ingestion\" {\n  name = \"data_warehouse_api_ingestion_${var.env}\"<\/strong>\n}<\/mark>\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">We&#8217;ll configure BigQuery as the target data warehouse to define our destination. While Fivetran&#8217;s UI offers a convenient way to explore the required parameters for each destination and connector, I recommend referring to it as a supplement to the official documentation. Terraform setup for a given destination or source is not always clear. I&#8217;ve contacted Fivetran, suggesting they include a small terraform sample code in their documentation for even greater clarity. However, a combination of both resources will ensure a smooth setup process.&nbsp;<\/p>\n\n\n\n<pre class=\"wp-block-code has-gray-light-background-color has-background\"><code>\nresource <strong><mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\">\"fivetran_destination\"<\/mark> <mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\">\"destination_data_warehouse_api_ingestion\"<\/mark><\/strong> {\n  group_id         = fivetran_group.data_warehouse_api_ingestion.<mark style=\"background-color:rgba(0, 0, 0, 0);color:#2347ca\" class=\"has-inline-color\"><strong>id<\/strong><\/mark>\n  region           = <mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\"><strong>\"GCP_EUROPE_WEST3\"<\/strong><\/mark>\n  service          = <mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\"><strong>\"big_query\"<\/strong><\/mark>\n  time_zone_offset = <mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\"><strong>\"+2\"<\/strong><\/mark>\n\n  run_setup_tests    = true\n  trust_certificates = false\n  trust_fingerprints = false\n\n  config {\n    project_id        = var.project_id\n    secret_key        = base64decode(google_service_account_key.fivetran_sa_key.private_key)\n    data_set_location = <mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\"><strong>\"EU\"<\/strong><\/mark>\n  }\n}\n\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">For BigQuery, I defined a module based on the source, a set of tables for the destination. This module will then create a raw dataset, a landing_zone dataset, and a current dataset. The raw dataset will be the name of your connector, as this is how Fivetran works. The landing zone will contain almost a replica of our raw layer but with some preprocessing to respect our naming convention. This layer includes the whole history of our data. Depending on your data platform, you will have different projects or datasets to do your transformations.<\/p>\n\n\n\n<pre class=\"wp-block-code has-gray-light-background-color has-background\"><code>\nmodule <mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\"><strong>\"bigquery_datalake_datasets\"<\/strong><\/mark> {\n  source       = <mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\"><strong>\".\/modules\/bq-datalake-destination\"<\/strong><\/mark>\n  for_each     = var.connectors\n  source_name  = each.value.name\n  with_current = each.value.destination_config.with_current\n}\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Finally, we can proceed with the definition of our connectors. We define the Fivetran group in which the resource belongs and which specific we want as a connector; in our case, this will be \u201cgoogle_cloud_functions\u201d. Our destination schema is our BigQuery dataset name. And then finally, the configuration \u201cconfig\u201d for our given service. We wanted to make the infrastructure as reusable as possible, thus defining each configuration as optional.&nbsp;<\/p>\n\n\n\n<pre class=\"wp-block-code has-gray-light-background-color has-background\"><code>\n\nresource<strong> <mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\">\"fivetran_connector\" \"connector\"<\/mark><\/strong> {\n  for_each           = var.connectors\n  group_id           = fivetran_group.data_warehouse_api_ingestion.<mark style=\"background-color:rgba(0, 0, 0, 0);color:#2347ca\" class=\"has-inline-color\"><strong>id<\/strong><\/mark>\n  service            = each.value.service\n  run_setup_tests    = true\n  trust_certificates = false\n  trust_fingerprints = false\n\n  destination_schema {\n    name = module.bigquery_datalake_datasets&#91;each.key].raw_dataset_name\n  }\n\n  dynamic <mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\"><strong>\"config\"<\/strong><\/mark> {\n    for_each = each.value.config == null ? &#91;] : &#91;<mark style=\"background-color:rgba(0, 0, 0, 0);color:#bf0f23\" class=\"has-inline-color\"><strong>0<\/strong><\/mark>]\n    content {\n      abs_connection_method = each.value.config.abs_connection_method\n      abs_connection_string = each.value.config.abs_connection_string\n      ...\n   api_token                  = each.value.config.api_token == null ? each.value.config.api_token : local.fivetran_passwords&#91;each.value.config.api_token]\n      api_type                   = each.value.config.api_type\n      api_url                    = each.value.config.api_url\n      api_usage                  = each.value.config.api_usage\n    }\n  }\n\n  depends_on = &#91;\n    fivetran_destination.destination_data_warehouse_api_ingestion\n  ]\n}\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">For each connector, we also need to define the scheduling. Optionally, you might want to specify the schema you want to respect if you want a written configuration. Otherwise, you can use the UI to select which tables and columns you want to synchronise.<\/p>\n\n\n\n<pre class=\"wp-block-code has-gray-light-background-color has-background\"><code>\nresource <mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\"><strong>\"fivetran_connector_schedule\" \"connector_schedule\"<\/strong><\/mark> {\n  for_each     = { <mark style=\"background-color:rgba(0, 0, 0, 0);color:#2347ca\" class=\"has-inline-color\"><strong>for<\/strong><\/mark> name, con <mark style=\"background-color:rgba(0, 0, 0, 0);color:#2347ca\" class=\"has-inline-color\"><strong>in<\/strong><\/mark> var.connectors : name => con.schedule <mark style=\"background-color:rgba(0, 0, 0, 0);color:#2347ca\" class=\"has-inline-color\"><strong>if <\/strong><\/mark>con.schedule != null }\n  connector_id = fivetran_connector.connector&#91;each.key].<mark style=\"background-color:rgba(0, 0, 0, 0);color:#2347ca\" class=\"has-inline-color\"><strong>id<\/strong><\/mark>\n\n  sync_frequency  = each.value.sync_frequency\n  daily_sync_time = each.value.daily_sync_time\n\n  paused            = each.value.paused\n  pause_after_trial = each.value.pause_after_trial\n\n  schedule_type = each.value.schedule_type\n}\n\nresource <mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\"><strong>\"fivetran_connector_schema_config\" \"connector_schema_config\"<\/strong><\/mark> {\n  for_each               = { <mark style=\"background-color:rgba(0, 0, 0, 0);color:#2347ca\" class=\"has-inline-color\"><strong>for<\/strong><\/mark> name, con <mark style=\"background-color:rgba(0, 0, 0, 0);color:#2347ca\" class=\"has-inline-color\"><strong>in<\/strong><\/mark> var.connectors : name => con.schema_config <mark style=\"background-color:rgba(0, 0, 0, 0);color:#2347ca\" class=\"has-inline-color\"><strong>if<\/strong><\/mark> con.schema_config != null }\n  connector_id           = fivetran_connector.connector&#91;each.key].<mark style=\"background-color:rgba(0, 0, 0, 0);color:#2347ca\" class=\"has-inline-color\"><strong>id<\/strong><\/mark>\n  schema_change_handling = each.value.schema_change_handling\n  dynamic <mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\"><strong>\"schema\"<\/strong><\/mark> {\n    for_each = each.value.schemas\n    content {\n      name    = schema.key\n      enabled = schema.value.enabled\n      dynamic <mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\"><strong>\"table\"<\/strong><\/mark> {\n        for_each = schema.value.tables\n        content {\n          name      = table.key\n          enabled   = table.value.enabled\n          sync_mode = var.destination.service == <mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\"><strong>\"aws_msk_wh\"<\/strong><\/mark> ? null : table.value.sync_mode\n          dynamic <mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\"><strong>\"column\"<\/strong> <\/mark>{\n            for_each = table.value.columns\n            content {\n              name    = column.key\n              hashed  = column.value.hashed\n              enabled = column.value.enabled\n            }\n          }\n        }\n      }\n    }\n  }\n} \n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Finally, we create the connector variable that we will use to define our different ingestions to BigQuery.&nbsp;<\/p>\n\n\n\n<pre class=\"wp-block-code has-gray-light-background-color has-background\"><code>\nvariable <mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\"><strong>\"connectors\"<\/strong><\/mark> {\n <strong> <mark style=\"background-color:rgba(0, 0, 0, 0);color:#2347ca\" class=\"has-inline-color\">type<\/mark><\/strong> = <mark style=\"background-color:rgba(0, 0, 0, 0);color:#2347ca\" class=\"has-inline-color\"><strong>map<\/strong><\/mark>(<mark style=\"background-color:rgba(0, 0, 0, 0);color:#2347ca\" class=\"has-inline-color\"><strong>object<\/strong><\/mark>({\n    name    = string\n    service = string\n    config = optional(<mark style=\"background-color:rgba(0, 0, 0, 0);color:#2347ca\" class=\"has-inline-color\"><strong>object<\/strong><\/mark>({\n      abs_connection_method = optional(string)\n      abs_connection_string = optional(string)\n      ...\n      api_key                    = optional(string)\n      api_secret                 = optional(string)\n      ...\n    }))\n    schedule = optional(object({\n      sync_frequency    = string\n      daily_sync_time   = string\n      paused            = <mark style=\"background-color:rgba(0, 0, 0, 0);color:#691aaf\" class=\"has-inline-color\"><strong>bool<\/strong><\/mark>\n      pause_after_trial = <mark style=\"background-color:rgba(0, 0, 0, 0);color:#691aaf\" class=\"has-inline-color\"><strong>bool<\/strong><\/mark>\n      schedule_type     = string\n    }))\n    schema_config = optional(<mark style=\"background-color:rgba(0, 0, 0, 0);color:#2347ca\" class=\"has-inline-color\"><strong>object<\/strong><\/mark>({\n      schema_change_handling = optional(string, <mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\"><strong>\"ALLOW_COLUMNS\"<\/strong><\/mark>)\n      schemas = optional(<mark style=\"background-color:rgba(0, 0, 0, 0);color:#2347ca\" class=\"has-inline-color\"><strong>map<\/strong><\/mark>(<mark style=\"background-color:rgba(0, 0, 0, 0);color:#2347ca\" class=\"has-inline-color\"><strong>object<\/strong><\/mark>({\n        enabled = optional(<mark style=\"background-color:rgba(0, 0, 0, 0);color:#691aaf\" class=\"has-inline-color\"><strong>bool<\/strong><\/mark>, false)\n        tables = optional(<mark style=\"background-color:rgba(0, 0, 0, 0);color:#2347ca\" class=\"has-inline-color\"><strong>map<\/strong><\/mark>(<mark style=\"background-color:rgba(0, 0, 0, 0);color:#2347ca\" class=\"has-inline-color\"><strong>object<\/strong><\/mark>({\n          enabled   = optional(<mark style=\"background-color:rgba(0, 0, 0, 0);color:#691aaf\" class=\"has-inline-color\"><strong>bool<\/strong><\/mark>, false)\n          sync_mode = optional(string, <mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\"><strong>\"SOFT_DELETE\"<\/strong><\/mark>)\n          columns = optional(<mark style=\"background-color:rgba(0, 0, 0, 0);color:#2347ca\" class=\"has-inline-color\"><strong>map<\/strong><\/mark>(<mark style=\"background-color:rgba(0, 0, 0, 0);color:#2347ca\" class=\"has-inline-color\"><strong>object<\/strong><\/mark>({\n            enabled = optional(<mark style=\"background-color:rgba(0, 0, 0, 0);color:#691aaf\" class=\"has-inline-color\"><strong>bool<\/strong><\/mark>, false)\n            hashed  = optional(<mark style=\"background-color:rgba(0, 0, 0, 0);color:#691aaf\" class=\"has-inline-color\"><strong>bool<\/strong><\/mark>, false)\n          })), {})\n        })), {})\n      })), {})\n    }))\n    destination_config = optional(\n      <mark style=\"background-color:rgba(0, 0, 0, 0);color:#2347ca\" class=\"has-inline-color\"><strong>object<\/strong><\/mark>({\n        with_current = optional(<mark style=\"background-color:rgba(0, 0, 0, 0);color:#691aaf\" class=\"has-inline-color\"><strong>bool<\/strong><\/mark>, true)\n      }),\n      {\n        with_current = true\n      }\n    )\n  }))\n}\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">And after configuring your terraform variables, you are pretty much done. The only thing remaining is to set up the tfvars with your input variables for each connector, source, and destination. In this current setup, if we use BigQuery as a destination, we define our connector variable with the correct parameters.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Note: As a security practice, we do not want passwords or API keys visible. Therefore, we set up a secret manager for each key and password that we retrieve through the secret name when we need it in the connector. An example is shown in the Okta connection with the API token.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Here is an example of the terraform. tfvars for the connector variable :<\/p>\n\n\n\n<pre class=\"wp-block-code has-gray-light-background-color has-background\"><code>\nconnectors = {\n    <mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\"><strong>\"fivetran_log\"<\/strong><\/mark> = {\n       <mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\"> <strong>\"name\"<\/strong>   <\/mark><mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-main-color\"> =<\/mark><mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\"> <strong>\"fivetran_log\"<\/strong>\n       <strong> \"service\"<\/strong> <\/mark><mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-main-color\">=<\/mark><mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\"> <strong>\"fivetran_log\"<\/strong>\n        <strong>\"config\"<\/strong><\/mark>  = {}\n        schedule = {\n          sync_frequency    =<mark style=\"background-color:rgba(0, 0, 0, 0);color:#bf0f23\" class=\"has-inline-color\"> <strong>360<\/strong><\/mark>\n          daily_sync_time   = <mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\"><strong>\"03:00\"<\/strong><\/mark>\n          paused            = false\n          pause_after_trial = false\n          schedule_type     = <mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\"><strong>\"auto\"<\/strong><\/mark>\n        }\n        destination_config = {\n          with_current = false\n        }\n    },\n  <strong> <mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\"> \"source1\"<\/mark><\/strong><mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-main-color\"> = {<\/mark><mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\">\n   <strong>     \"name\" <\/strong>   <\/mark><mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-main-color\">=<\/mark><mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\"> <strong>\"source1\"<\/strong>\n        <strong>\"service\"<\/strong> <\/mark><mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-main-color\">=<\/mark><mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\"> <strong>\"google_cloud_function\"<\/strong><\/mark>\n        config = {}\n        schedule = {\n            sync_frequency    = <mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\"><strong>\"360\"<\/strong><\/mark>\n            daily_sync_time   = <mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\"><strong>\"06:00\"<\/strong><\/mark>\n            paused            = false\n            pause_after_trial = false\n            schedule_type     = <mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\"><strong>\"auto\"<\/strong><\/mark>\n        }\n    },\n    <mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\"><strong>\"okta\"<\/strong> <\/mark>= {\n        name    = <mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\"><strong>\"okta\"<\/strong><\/mark>\n        service = <mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\"><strong>\"okta\"<\/strong><\/mark>\n        config = {\n            domain = <mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\"><strong>\"domain\"<\/strong><\/mark>\n            sub_domain = <mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\"><strong>\"subdomain\"<\/strong><\/mark>\n            api_token = <mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\"><strong>\"okta_api_key\"<\/strong><\/mark>\n        }\n        schedule = {\n            sync_frequency    = <mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\"><strong>\"720\"<\/strong><\/mark>\n            daily_sync_time   = <mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\"><strong>\"06:00\"<\/strong><\/mark>\n            paused            = false\n            pause_after_trial = false\n            schedule_type     = <mark style=\"background-color:rgba(0, 0, 0, 0);color:#16724c\" class=\"has-inline-color\"><strong>\"auto\"<\/strong><\/mark>\n        },\n        destination_config = {\n            with_current = false\n        }\n    },\n}\n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-conclusion\">Conclusion<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Combining Fivetran&#8217;s data integration capabilities with Google Cloud Functions as connectors, you can establish a robust and automated pipeline for ingesting data from your custom API into BigQuery. Looking at your data platform, we provided a powerful and easy-to-integrate EL (Extract Load) service that you can integrate into your data platform, where you will have all your transformation layers and data marts.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"h-further-exploration\">Further Exploration:<\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">This blog post provides a foundational overview. To delve deeper, explore the following resources:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Google Cloud Function Documentation: <a href=\"https:\/\/cloud.google.com\/functions\">https:\/\/cloud.google.com\/functions<\/a><br>Fivetran Documentation on Custom Connectors: <a href=\"https:\/\/fivetran.com\/docs\/connectors\/functions\">https:\/\/fivetran.com\/docs\/connectors\/functions<\/a><\/p>\n\n\n\n<div class=\"wp-block-columns alignfull is-layout-flex wp-container-core-columns-is-layout-4b1fd674 wp-block-columns-is-layout-flex\" style=\"margin-top:var(--wp--preset--spacing--medium);margin-bottom:var(--wp--preset--spacing--medium)\">\n<div class=\"wp-block-column is-vertically-aligned-center is-layout-flow wp-container-core-column-is-layout-969dc29d wp-block-column-is-layout-flow\" style=\"padding-top:0;padding-right:0;padding-bottom:0;padding-left:0;flex-basis:100%\">\n<div class=\"wp-block-cover is-light\"><span aria-hidden=\"true\" class=\"wp-block-cover__background has-background-dim\" style=\"background-color:#8476bb\"><\/span><img decoding=\"async\" class=\"wp-block-cover__image-background wp-image-8279\" alt=\"\" src=\"https:\/\/devoteam.info\/wp-content\/uploads\/2024\/09\/digital-solutions.jpg\" data-object-fit=\"cover\"\/><div class=\"wp-block-cover__inner-container has-global-padding is-layout-constrained wp-block-cover-is-layout-constrained\">\n<p class=\"has-text-align-center has-large-font-size wp-block-paragraph\"><\/p>\n<\/div><\/div>\n<\/div>\n\n\n\n<div class=\"wp-block-column is-vertically-aligned-stretch has-gray-light-background-color has-background is-layout-flow wp-block-column-is-layout-flow\" style=\"flex-basis:100%\">\n<div class=\"wp-block-group has-main-color has-gray-light-background-color has-text-color has-background has-link-color wp-elements-1 is-layout-flow wp-container-core-group-is-layout-23162e80 wp-block-group-is-layout-flow\" style=\"margin-top:0;margin-bottom:0;padding-top:var(--wp--preset--spacing--medium);padding-right:var(--wp--preset--spacing--large);padding-bottom:var(--wp--preset--spacing--medium);padding-left:var(--wp--preset--spacing--large)\">\n<div class=\"wp-block-group is-vertical is-content-justification-left is-layout-flex wp-container-core-group-is-layout-4a15ae55 wp-block-group-is-layout-flex\" style=\"min-height:0px\">\n<p class=\"has-text-align-left has-main-color has-text-color has-medium-font-size wp-container-content-42c95ffb wp-block-paragraph\">The best time to talk about your Cloud journey is now!<\/p>\n\n\n\n<p class=\"has-text-align-left has-main-color has-text-color has-link-color has-base-font-size wp-elements-2 wp-block-paragraph\">Contact our certified experts to guide your Google Cloud projects.<br><\/p>\n\n\n\n<div class=\"wp-block-buttons is-content-justification-left is-layout-flex wp-container-core-buttons-is-layout-5446dffb wp-block-buttons-is-layout-flex\">\n<div class=\"wp-block-button\"><a class=\"wp-block-button__link wp-element-button\" href=\"https:\/\/devoteam.info\/get-in-touch\/\">Get in touch<\/a><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n","protected":false},"excerpt":{"rendered":"<p>This article was written by Harold Snyers \u2013 Senior ML &amp; Data Engineer I Google Cloud Business Unit @ Devoteam Belgium. Today, businesses rely on many data sources, including custom APIs. However, extracting and transforming data from APIs for analysis can be complex and time-consuming. This is where Fivetran comes in, offering a powerful solution [&hellip;]<\/p>\n","protected":false},"featured_media":493639,"template":"","categories":[2635,2604],"tags":[2751],"industry":[],"class_list":["post-518168","expert-view","type-expert-view","status-publish","has-post-thumbnail","hentry","category-data-sk","category-google-cloud-sk","tag-data-sk"],"acf":[],"cards":"\n\t<div class=\"single-post-card\">\n\n\t\t<figure class=\"wp-block-post-featured-image\"><a href=\"https:\/\/devoteam.info\/sk\/expert-view\/fivetran-connectors-google-cloud-integration\/\" target=\"_self\" ><img width=\"2560\" height=\"1920\" src=\"https:\/\/devoteam.info\/wp-content\/uploads\/2024\/10\/GettyImages-1693803686.jpg\" class=\"attachment-post-thumbnail size-post-thumbnail wp-post-image\" alt=\"Leveraging Google Cloud Functions to Create a Powerful Fivetran Connector to integrate custom APIs\" style=\"aspect-ratio:4\/3;width:100%;object-fit:cover;\" decoding=\"async\" loading=\"lazy\" srcset=\"https:\/\/devoteam.info\/wp-content\/uploads\/2024\/10\/GettyImages-1693803686.jpg 2560w, https:\/\/devoteam.info\/wp-content\/uploads\/2024\/10\/GettyImages-1693803686-300x225.jpg 300w, https:\/\/devoteam.info\/wp-content\/uploads\/2024\/10\/GettyImages-1693803686-1024x768.jpg 1024w, https:\/\/devoteam.info\/wp-content\/uploads\/2024\/10\/GettyImages-1693803686-768x576.jpg 768w, https:\/\/devoteam.info\/wp-content\/uploads\/2024\/10\/GettyImages-1693803686-1536x1152.jpg 1536w, https:\/\/devoteam.info\/wp-content\/uploads\/2024\/10\/GettyImages-1693803686-2048x1536.jpg 2048w\" sizes=\"auto, (max-width: 2560px) 100vw, 2560px\" \/><\/a><\/figure>\n\n\t\t\n\t\t<div class=\"wp-block-group is-vertical is-layout-flex wp-container-core-group-is-layout-43282307 wp-block-group-is-layout-flex\">\n\t<p style=\"font-style:normal;font-weight:700\" class=\"has-link-color wp-elements-4 wp-block-lp-post-type has-text-color has-primary-color has-small-font-size\">Expert View<\/p>\n\n\t\t\n\t\t<h3 style=\"font-style:normal;font-weight:400\" class=\"wp-block-post-title has-base-font-size\"><a href=\"https:\/\/devoteam.info\/sk\/expert-view\/fivetran-connectors-google-cloud-integration\/\" target=\"_self\" >Leveraging Google Cloud Functions to Create a Powerful Fivetran Connector to integrate custom APIs<\/a><\/h3><\/div>\n\t\t\n\t<\/div>\n\n","yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v28.4 (Yoast SEO v28.4) - https:\/\/yoast.com\/product\/yoast-seo-premium-wordpress\/ -->\n<title>Leveraging Google Cloud Functions to Create a Powerful Fivetran Connector to integrate custom APIs | Devoteam<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/devoteam.info\/sk\/expert-view\/fivetran-connectors-google-cloud-integration\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Leveraging Google Cloud Functions to Create a Powerful Fivetran Connector to integrate custom APIs\" \/>\n<meta property=\"og:description\" content=\"This article was written by Harold Snyers \u2013 Senior ML &amp; Data Engineer I Google Cloud Business Unit @ Devoteam Belgium. Today, businesses rely on many data sources, including custom APIs. However, extracting and transforming data from APIs for analysis can be complex and time-consuming. This is where Fivetran comes in, offering a powerful solution [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/devoteam.info\/sk\/expert-view\/fivetran-connectors-google-cloud-integration\/\" \/>\n<meta property=\"og:site_name\" content=\"Devoteam\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"11 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/devoteam.info\\\/sk\\\/expert-view\\\/fivetran-connectors-google-cloud-integration\\\/\",\"url\":\"https:\\\/\\\/devoteam.info\\\/sk\\\/expert-view\\\/fivetran-connectors-google-cloud-integration\\\/\",\"name\":\"Leveraging Google Cloud Functions to Create a Powerful Fivetran Connector to integrate custom APIs | Devoteam\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/devoteam.info\\\/sk\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/devoteam.info\\\/sk\\\/expert-view\\\/fivetran-connectors-google-cloud-integration\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/devoteam.info\\\/sk\\\/expert-view\\\/fivetran-connectors-google-cloud-integration\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/devoteam.info\\\/wp-content\\\/uploads\\\/2024\\\/10\\\/GettyImages-1693803686.jpg\",\"datePublished\":\"2024-11-20T14:28:16+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/devoteam.info\\\/sk\\\/expert-view\\\/fivetran-connectors-google-cloud-integration\\\/#breadcrumb\"},\"inLanguage\":\"en-SK\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/devoteam.info\\\/sk\\\/expert-view\\\/fivetran-connectors-google-cloud-integration\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-SK\",\"@id\":\"https:\\\/\\\/devoteam.info\\\/sk\\\/expert-view\\\/fivetran-connectors-google-cloud-integration\\\/#primaryimage\",\"url\":\"https:\\\/\\\/devoteam.info\\\/wp-content\\\/uploads\\\/2024\\\/10\\\/GettyImages-1693803686.jpg\",\"contentUrl\":\"https:\\\/\\\/devoteam.info\\\/wp-content\\\/uploads\\\/2024\\\/10\\\/GettyImages-1693803686.jpg\",\"width\":2560,\"height\":1920},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/devoteam.info\\\/sk\\\/expert-view\\\/fivetran-connectors-google-cloud-integration\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/devoteam.info\\\/sk\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Expert View\",\"item\":\"https:\\\/\\\/devoteam.info\\\/sk\\\/expert-view\\\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"Leveraging Google Cloud Functions to Create a Powerful Fivetran Connector to integrate custom APIs\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/devoteam.info\\\/sk\\\/#website\",\"url\":\"https:\\\/\\\/devoteam.info\\\/sk\\\/\",\"name\":\"Devoteam\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/devoteam.info\\\/sk\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-SK\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Leveraging Google Cloud Functions to Create a Powerful Fivetran Connector to integrate custom APIs | Devoteam","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/devoteam.info\/sk\/expert-view\/fivetran-connectors-google-cloud-integration\/","og_locale":"en_US","og_type":"article","og_title":"Leveraging Google Cloud Functions to Create a Powerful Fivetran Connector to integrate custom APIs","og_description":"This article was written by Harold Snyers \u2013 Senior ML &amp; Data Engineer I Google Cloud Business Unit @ Devoteam Belgium. Today, businesses rely on many data sources, including custom APIs. However, extracting and transforming data from APIs for analysis can be complex and time-consuming. This is where Fivetran comes in, offering a powerful solution [&hellip;]","og_url":"https:\/\/devoteam.info\/sk\/expert-view\/fivetran-connectors-google-cloud-integration\/","og_site_name":"Devoteam","twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"11 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/devoteam.info\/sk\/expert-view\/fivetran-connectors-google-cloud-integration\/","url":"https:\/\/devoteam.info\/sk\/expert-view\/fivetran-connectors-google-cloud-integration\/","name":"Leveraging Google Cloud Functions to Create a Powerful Fivetran Connector to integrate custom APIs | Devoteam","isPartOf":{"@id":"https:\/\/devoteam.info\/sk\/#website"},"primaryImageOfPage":{"@id":"https:\/\/devoteam.info\/sk\/expert-view\/fivetran-connectors-google-cloud-integration\/#primaryimage"},"image":{"@id":"https:\/\/devoteam.info\/sk\/expert-view\/fivetran-connectors-google-cloud-integration\/#primaryimage"},"thumbnailUrl":"https:\/\/devoteam.info\/wp-content\/uploads\/2024\/10\/GettyImages-1693803686.jpg","datePublished":"2024-11-20T14:28:16+00:00","breadcrumb":{"@id":"https:\/\/devoteam.info\/sk\/expert-view\/fivetran-connectors-google-cloud-integration\/#breadcrumb"},"inLanguage":"en-SK","potentialAction":[{"@type":"ReadAction","target":["https:\/\/devoteam.info\/sk\/expert-view\/fivetran-connectors-google-cloud-integration\/"]}]},{"@type":"ImageObject","inLanguage":"en-SK","@id":"https:\/\/devoteam.info\/sk\/expert-view\/fivetran-connectors-google-cloud-integration\/#primaryimage","url":"https:\/\/devoteam.info\/wp-content\/uploads\/2024\/10\/GettyImages-1693803686.jpg","contentUrl":"https:\/\/devoteam.info\/wp-content\/uploads\/2024\/10\/GettyImages-1693803686.jpg","width":2560,"height":1920},{"@type":"BreadcrumbList","@id":"https:\/\/devoteam.info\/sk\/expert-view\/fivetran-connectors-google-cloud-integration\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/devoteam.info\/sk\/"},{"@type":"ListItem","position":2,"name":"Expert View","item":"https:\/\/devoteam.info\/sk\/expert-view\/"},{"@type":"ListItem","position":3,"name":"Leveraging Google Cloud Functions to Create a Powerful Fivetran Connector to integrate custom APIs"}]},{"@type":"WebSite","@id":"https:\/\/devoteam.info\/sk\/#website","url":"https:\/\/devoteam.info\/sk\/","name":"Devoteam","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/devoteam.info\/sk\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-SK"}]}},"uagb_featured_image_src":{"full":["https:\/\/devoteam.info\/wp-content\/uploads\/2024\/10\/GettyImages-1693803686.jpg",2560,1920,false],"thumbnail":["https:\/\/devoteam.info\/wp-content\/uploads\/2024\/10\/GettyImages-1693803686-150x150.jpg",150,150,true],"medium":["https:\/\/devoteam.info\/wp-content\/uploads\/2024\/10\/GettyImages-1693803686-300x225.jpg",300,225,true],"medium_large":["https:\/\/devoteam.info\/wp-content\/uploads\/2024\/10\/GettyImages-1693803686-768x576.jpg",768,576,true],"large":["https:\/\/devoteam.info\/wp-content\/uploads\/2024\/10\/GettyImages-1693803686-1024x768.jpg",1024,768,true],"1536x1536":["https:\/\/devoteam.info\/wp-content\/uploads\/2024\/10\/GettyImages-1693803686-1536x1152.jpg",1536,1152,true],"2048x2048":["https:\/\/devoteam.info\/wp-content\/uploads\/2024\/10\/GettyImages-1693803686-2048x1536.jpg",2048,1536,true]},"uagb_author_info":{"display_name":"laura.borremans","author_link":"https:\/\/devoteam.info\/sk\/author\/"},"uagb_comment_info":0,"uagb_excerpt":"This article was written by Harold Snyers \u2013 Senior ML &amp; Data Engineer I Google Cloud Business Unit @ Devoteam Belgium. Today, businesses rely on many data sources, including custom APIs. However, extracting and transforming data from APIs for analysis can be complex and time-consuming. This is where Fivetran comes in, offering a powerful solution&hellip;","_links":{"self":[{"href":"https:\/\/devoteam.info\/sk\/wp-json\/wp\/v2\/expert-view\/518168","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devoteam.info\/sk\/wp-json\/wp\/v2\/expert-view"}],"about":[{"href":"https:\/\/devoteam.info\/sk\/wp-json\/wp\/v2\/types\/expert-view"}],"version-history":[{"count":0,"href":"https:\/\/devoteam.info\/sk\/wp-json\/wp\/v2\/expert-view\/518168\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devoteam.info\/sk\/wp-json\/wp\/v2\/media\/493639"}],"wp:attachment":[{"href":"https:\/\/devoteam.info\/sk\/wp-json\/wp\/v2\/media?parent=518168"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devoteam.info\/sk\/wp-json\/wp\/v2\/categories?post=518168"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devoteam.info\/sk\/wp-json\/wp\/v2\/tags?post=518168"},{"taxonomy":"industry","embeddable":true,"href":"https:\/\/devoteam.info\/sk\/wp-json\/wp\/v2\/industry?post=518168"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}