{"id":9337,"date":"2021-07-12T14:57:36","date_gmt":"2021-07-12T13:57:36","guid":{"rendered":"https:\/\/ee.yelkdev.site\/?p=9337"},"modified":"2024-03-28T13:49:24","modified_gmt":"2024-03-28T13:49:24","slug":"how-we-ended-up-creating-language-agnostic-data-pipelines-for-our-customers-at-equal-experts","status":"publish","type":"post","link":"https:\/\/www.equalexperts.com\/blog\/our-thinking\/how-we-ended-up-creating-language-agnostic-data-pipelines-for-our-customers-at-equal-experts\/","title":{"rendered":"How we ended up creating language agnostic data pipelines for our customers at Equal Experts"},"content":{"rendered":"<p><span style=\"font-weight: 400;\">Based on the experience shared in <\/span><a href=\"https:\/\/www.equalexperts.com\/blog\/our-thinking\/how-to-evolve-a-clients-data-strategy-solution-to-provide-more-business-benefits-than-ever-before\/\"><span style=\"font-weight: 400;\">evolving a client\u2019s data architecture<\/span><\/a><span style=\"font-weight: 400;\">, we decided to share a reference implementation of data pipelines. Recalling from the <\/span><a href=\"https:\/\/playbooks.equalexperts.com\/data-pipeline\"><span style=\"font-weight: 400;\">data pipeline playbook<\/span><\/a><span style=\"font-weight: 400;\">.<\/span><\/p>\n<h2><b>What is a Data Pipeline?\u00a0<\/b><\/h2>\n<p>From the EE Data Pipeline playbook:<\/p>\n<img decoding=\"async\" class=\"alignnone wp-image-9339 size-full\" src=\"https:\/\/www.equalexperts.com\/wp-content\/uploads\/2021\/07\/what-is-a-data-pipeline.png\" alt=\"\" width=\"1073\" height=\"601\" srcset=\"https:\/\/www.equalexperts.com\/wp-content\/uploads\/2021\/07\/what-is-a-data-pipeline.png 1073w, https:\/\/www.equalexperts.com\/wp-content\/uploads\/2021\/07\/what-is-a-data-pipeline-300x168.png 300w, https:\/\/www.equalexperts.com\/wp-content\/uploads\/2021\/07\/what-is-a-data-pipeline-768x430.png 768w\" sizes=\"(max-width: 1073px) 100vw, 1073px\" \/>\n<p><span style=\"font-weight: 400;\">A Data Pipeline is created for data analytics purposes and has:<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Data sources &#8211; these can be internal or external and may be structured (e.g. the result of a database call), semi-structured (e.g. a CSV file or a Google Sheet), or unstructured (e.g. text documents or images).<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Ingestion process &#8211; the means by which data is moved from the source into the pipeline (e.g. API call, secure file transfer).<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Transformations &#8211; in most cases data needs to be transformed from the input format of the raw data, to the one in which it is stored. There may be several transformations in a pipeline.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Data Quality\/Cleansing &#8211; data is checked for quality at various points in the pipeline. Data quality will typically include at least validation of data types and format, as well as conforming against master data.\u00a0<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Enrichment &#8211; data items may be enriched by adding additional fields such as reference data.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Storage &#8211; data is stored at various points in the pipeline. Usually at least the landing zone and a structured store such as a data warehouse.<\/span><\/li>\n<\/ul>\n<h2><b>Functional\u00a0<\/b><b>requirements<\/b><\/h2>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Pipelines that are:<\/span>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"2\"><span style=\"font-weight: 400;\">Easy to orchestrate<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"2\"><span style=\"font-weight: 400;\">Support scheduling\u00a0<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"2\"><span style=\"font-weight: 400;\">Support backfilling<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"2\"><span style=\"font-weight: 400;\">Support testing on all the steps<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"2\"><span style=\"font-weight: 400;\">Easy to integrate with custom APIs as sources of data<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"2\"><span style=\"font-weight: 400;\">Easy to integrate in a CI\/CD environment<\/span><\/li>\n<\/ul>\n<\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">The code can be developed in multiple languages to fit each client skill set when python is not a first class citizen.\u00a0<\/span><\/li>\n<\/ul>\n<h2><b>Our strategy<\/b><span style=\"font-weight: 400;\">\u00a0<\/span><\/h2>\n<p><span style=\"font-weight: 400;\">In some situations a tool like <\/span><a href=\"https:\/\/www.matillion.com\/\"><span style=\"font-weight: 400;\">Matillion<\/span><\/a><span style=\"font-weight: 400;\">, <\/span><a href=\"https:\/\/www.stitchdata.com\/\"><span style=\"font-weight: 400;\">Stitchdata<\/span><\/a><span style=\"font-weight: 400;\"> or <\/span><a href=\"https:\/\/fivetran.com\/\"><span style=\"font-weight: 400;\">Fivetran<\/span><\/a><span style=\"font-weight: 400;\"> can be the best approach, although it\u2019s not the best choice for all of our client\u2019s use cases. These ETL tools work well when using the existing pre-made connectors, although when the majority of the data integrations are custom connectors, it\u2019s certainly not the best approach. Apart from the known cost, there is also an extra cost when using these kinds of tools &#8211; the effort to make the data pipelines working in a CI\/CD environment. Also, at Equal Experts, we advocate we should test each step of the pipeline, and if possible, develop them using test driven development &#8211; and this is near impossible in these cases.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">That being said, for the cases when an ETL tool won\u2019t fit our needs, we identified the need of having a reference implementation that we can use for different clients. Since the skill set of each team is different, and sometimes Python is not an acquired skill,<\/span><span style=\"font-weight: 400;\"> it was decided not to use the well known python tools that are used these days for data pipelines like\u00a0 <\/span><a href=\"https:\/\/airflow.apache.org\/\"><span style=\"font-weight: 400;\">Apache Airflow<\/span><\/a><span style=\"font-weight: 400;\"> or <\/span><a href=\"https:\/\/dagster.io\/\"><span style=\"font-weight: 400;\">Dagster.<\/span><\/a><span style=\"font-weight: 400;\">\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">So we designed a solution using <\/span><a href=\"https:\/\/argoproj.github.io\/workflows\"><span style=\"font-weight: 400;\">Argo Workflows<\/span><\/a><span style=\"font-weight: 400;\"> as the orchestrator. We wanted something which allowed us to define the data pipelines as DAGs like Airflow.\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Argo Workflows is a container-native workflow engine for orchestrating parallel jobs on Kubernetes. Argo represents workflows as Dags (Directed Acyclic Graphs), and each step of the workflow is a container. Since data pipelines can be easily modeled as workflow it is a great tool to use. Also, we have freedom to choose which programming language to design the connectors or the transformations, the only requirement is that each step of the pipeline should be containerised.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">For the data transformations, we found that <\/span><a href=\"https:\/\/www.getdbt.com\/\"><span style=\"font-weight: 400;\">dbt<\/span><\/a><span style=\"font-weight: 400;\"> was our best choice. Dbt allows the transformations needed between the staging tables and the analytics tables. Dbt is SQL centric, so there isn&#8217;t a need to learn another language. Also, dbt has features that we wanted like testing and documentation generation and has native connections to Snowflake, BigQuery, Redshift and Postgres data warehouses.\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">With these two tools, that is how we ended up with a language agnostic data pipelines architecture<\/span><span style=\"font-weight: 400;\"> that can be easily reused and adapted in multiple cases and for different clients.<\/span><\/p>\n<h2><b>Reference implementation<\/b><\/h2>\n<p><span style=\"font-weight: 400;\">Because we value knowledge sharing, we have created a public reference implementation of this architecture in the <\/span><a href=\"https:\/\/github.com\/EqualExperts\/language-agnostic-data-pipelines\"><span style=\"font-weight: 400;\">github repo<\/span><\/a><span style=\"font-weight: 400;\"> which shows a pipeline for a simple use case of ingesting UK COVID-19 data (<\/span><span style=\"font-weight: 400;\">https:\/\/api.coronavirus.data.gov.uk<\/span><span style=\"font-weight: 400;\">) as an example.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">The goal of the project is to have a simple implementation that can be used as an accelerator to other teams. It can be easily adapted to make other data pipelines, to integrate in a CI\/CD environment, or to extend the approach and make it work for different scenarios.\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">The sample project uses a local kubernetes cluster to deploy Argo and the containers which represent the data pipeline. Also a database where COVID-19 data is loaded and transformed and an instance of <\/span><a href=\"https:\/\/www.metabase.com\/\"><span style=\"font-weight: 400;\">Metabase<\/span><\/a><span style=\"font-weight: 400;\"> to show the data in a friendly dashboard.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">We\u2019re planning to add into the reference implementation infrastructure as code to deploy the project on AWS and GCP. Also, we might also work in aspects like facilitating the monitoring of the data pipelines when deployed in a cloud, or using <\/span><a href=\"https:\/\/greatexpectations.io\/\"><span style=\"font-weight: 400;\">Great Expectations<\/span><\/a><span style=\"font-weight: 400;\">.<\/span><\/p>\n<h2><b>Transparency is at the heart of our values<\/b><\/h2>\n<p><span style=\"font-weight: 400;\">We value knowledge sharing and collaboration, so we hope that this article, along with the data pipelines playbook will help you to start creating data pipelines in whichever language you choose.\u00a0<\/span><\/p>\n<h2><b>Contact us!<\/b><\/h2>\n<p><span style=\"font-weight: 400;\">For more information on data pipelines in general, take a look at our <\/span><a href=\"https:\/\/playbooks.equalexperts.com\/data-pipeline\"><span style=\"font-weight: 400;\">Data Pipeline Playbook<\/span><\/a><span style=\"font-weight: 400;\">.\u00a0 And if you\u2019d like us to share our experience of data pipelines with you, get in touch using the form below.<\/span><\/p>\n\n\t\t\t\t\t\t<script>\n\t\t\t\t\t\t\twindow.hsFormsOnReady = window.hsFormsOnReady || [];\n\t\t\t\t\t\t\twindow.hsFormsOnReady.push(()=>{\n\t\t\t\t\t\t\t\thbspt.forms.create({\n\t\t\t\t\t\t\t\t\tportalId: 7208712,\n\t\t\t\t\t\t\t\t\tformId: \"83acdf22-cf43-47ba-b91f-0428264b824a\",\n\t\t\t\t\t\t\t\t\ttarget: \"#hbspt-form-1758975560000-7626667077\",\n\t\t\t\t\t\t\t\t\tregion: \"eu1\",\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t})});\n\t\t\t\t\t\t<\/script>\n\t\t\t\t\t\t<div class=\"hbspt-form\" id=\"hbspt-form-1758975560000-7626667077\"><\/div>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Two tools to create a language agnostic data pipelines architecture that can be easily reused and adapted in multiple cases and for different clients plus a public reference implementation of this architecture<\/p>\n","protected":false},"author":164,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"content-type":"","inline_featured_image":false,"footnotes":""},"categories":[5],"tags":[188,184,180,194],"location":[397],"class_list":["post-9337","post","type-post","status-publish","format-standard","hentry","category-our-thinking","tag-data-engineering","tag-data-governance","tag-data-pipeline-playbook","tag-data-pipline"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v25.9 (Yoast SEO v25.9) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>How we ended up creating language agnostic data pipelines for our customers at Equal Experts | Equal Experts<\/title>\n<meta name=\"description\" content=\"How we used two tools to create a scalable language agnostic data pipeline architecture that can be easily reused and adapted.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.equalexperts.com\/blog\/our-thinking\/how-we-ended-up-creating-language-agnostic-data-pipelines-for-our-customers-at-equal-experts\/\" \/>\n<meta property=\"og:locale\" content=\"en_GB\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"How we ended up creating language agnostic data pipelines for our customers at Equal Experts\" \/>\n<meta property=\"og:description\" content=\"Two tools to create a language agnostic data pipelines architecture that can be easily reused and adapted in multiple cases and for different clients plus a public reference implementation of this architecture\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.equalexperts.com\/blog\/our-thinking\/how-we-ended-up-creating-language-agnostic-data-pipelines-for-our-customers-at-equal-experts\/\" \/>\n<meta property=\"og:site_name\" content=\"Equal Experts\" \/>\n<meta property=\"article:published_time\" content=\"2021-07-12T13:57:36+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-03-28T13:49:24+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.equalexperts.com\/wp-content\/uploads\/2021\/07\/Blog_Language_Agnostic_Data_Pipelines_Facebook_1200x630.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"630\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Cl\u00e1udio Diniz\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:title\" content=\"How we ended up creating language agnostic data pipelines for our customers at Equal Experts\" \/>\n<meta name=\"twitter:description\" content=\"Two tools to create a language agnostic data pipelines architecture that can be easily reused and adapted in multiple cases and for different clients plus a public reference implementation of this architecture\" \/>\n<meta name=\"twitter:image\" content=\"https:\/\/www.equalexperts.com\/wp-content\/uploads\/2021\/07\/Blog_Language_Agnostic_Data_Pipelines_Twitter_1024x512.jpg\" \/>\n<meta name=\"twitter:creator\" content=\"@EqualExperts\" \/>\n<meta name=\"twitter:site\" content=\"@EqualExperts\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Cl\u00e1udio Diniz\" \/>\n\t<meta name=\"twitter:label2\" content=\"Estimated reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.equalexperts.com\/blog\/our-thinking\/how-we-ended-up-creating-language-agnostic-data-pipelines-for-our-customers-at-equal-experts\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.equalexperts.com\/blog\/our-thinking\/how-we-ended-up-creating-language-agnostic-data-pipelines-for-our-customers-at-equal-experts\/\"},\"author\":{\"name\":\"Cl\u00e1udio Diniz\",\"@id\":\"https:\/\/www.equalexperts.com\/#\/schema\/person\/28ff89d676b184c93ab62bc91b0af11e\"},\"headline\":\"How we ended up creating language agnostic data pipelines for our customers at Equal Experts\",\"datePublished\":\"2021-07-12T13:57:36+00:00\",\"dateModified\":\"2024-03-28T13:49:24+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.equalexperts.com\/blog\/our-thinking\/how-we-ended-up-creating-language-agnostic-data-pipelines-for-our-customers-at-equal-experts\/\"},\"wordCount\":958,\"publisher\":{\"@id\":\"https:\/\/www.equalexperts.com\/#organization\"},\"image\":{\"@id\":\"https:\/\/www.equalexperts.com\/blog\/our-thinking\/how-we-ended-up-creating-language-agnostic-data-pipelines-for-our-customers-at-equal-experts\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.equalexperts.com\/wp-content\/uploads\/2021\/07\/what-is-a-data-pipeline.png\",\"keywords\":[\"data engineering\",\"data governance\",\"data pipeline playbook\",\"data pipline\"],\"articleSection\":[\"Our Thinking\"],\"inLanguage\":\"en-GB\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.equalexperts.com\/blog\/our-thinking\/how-we-ended-up-creating-language-agnostic-data-pipelines-for-our-customers-at-equal-experts\/\",\"url\":\"https:\/\/www.equalexperts.com\/blog\/our-thinking\/how-we-ended-up-creating-language-agnostic-data-pipelines-for-our-customers-at-equal-experts\/\",\"name\":\"How we ended up creating language agnostic data pipelines for our customers at Equal Experts | Equal Experts\",\"isPartOf\":{\"@id\":\"https:\/\/www.equalexperts.com\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.equalexperts.com\/blog\/our-thinking\/how-we-ended-up-creating-language-agnostic-data-pipelines-for-our-customers-at-equal-experts\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.equalexperts.com\/blog\/our-thinking\/how-we-ended-up-creating-language-agnostic-data-pipelines-for-our-customers-at-equal-experts\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.equalexperts.com\/wp-content\/uploads\/2021\/07\/what-is-a-data-pipeline.png\",\"datePublished\":\"2021-07-12T13:57:36+00:00\",\"dateModified\":\"2024-03-28T13:49:24+00:00\",\"description\":\"How we used two tools to create a scalable language agnostic data pipeline architecture that can be easily reused and adapted.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.equalexperts.com\/blog\/our-thinking\/how-we-ended-up-creating-language-agnostic-data-pipelines-for-our-customers-at-equal-experts\/#breadcrumb\"},\"inLanguage\":\"en-GB\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.equalexperts.com\/blog\/our-thinking\/how-we-ended-up-creating-language-agnostic-data-pipelines-for-our-customers-at-equal-experts\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\/\/www.equalexperts.com\/blog\/our-thinking\/how-we-ended-up-creating-language-agnostic-data-pipelines-for-our-customers-at-equal-experts\/#primaryimage\",\"url\":\"https:\/\/www.equalexperts.com\/wp-content\/uploads\/2021\/07\/what-is-a-data-pipeline.png\",\"contentUrl\":\"https:\/\/www.equalexperts.com\/wp-content\/uploads\/2021\/07\/what-is-a-data-pipeline.png\",\"width\":1073,\"height\":601},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.equalexperts.com\/blog\/our-thinking\/how-we-ended-up-creating-language-agnostic-data-pipelines-for-our-customers-at-equal-experts\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.equalexperts.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"How we ended up creating language agnostic data pipelines for our customers at Equal Experts\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.equalexperts.com\/#website\",\"url\":\"https:\/\/www.equalexperts.com\/\",\"name\":\"Equal Experts\",\"description\":\"Making Software. Better.\",\"publisher\":{\"@id\":\"https:\/\/www.equalexperts.com\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.equalexperts.com\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-GB\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.equalexperts.com\/#organization\",\"name\":\"Equal Experts\",\"url\":\"https:\/\/www.equalexperts.com\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\/\/www.equalexperts.com\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.equalexperts.com\/wp-content\/uploads\/2018\/08\/Equal_Experts_Logo_CMYK_Colour.jpg\",\"contentUrl\":\"https:\/\/www.equalexperts.com\/wp-content\/uploads\/2018\/08\/Equal_Experts_Logo_CMYK_Colour.jpg\",\"width\":719,\"height\":340,\"caption\":\"Equal Experts\"},\"image\":{\"@id\":\"https:\/\/www.equalexperts.com\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/x.com\/EqualExperts\",\"https:\/\/www.linkedin.com\/company\/equal-experts\/?viewAsMember=true\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.equalexperts.com\/#\/schema\/person\/28ff89d676b184c93ab62bc91b0af11e\",\"name\":\"Cl\u00e1udio Diniz\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\/\/www.equalexperts.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/d70fbe38b0540d312610b719e2e75bc9f302aafe3264bf1eb8174eb191c4879d?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/d70fbe38b0540d312610b719e2e75bc9f302aafe3264bf1eb8174eb191c4879d?s=96&d=mm&r=g\",\"caption\":\"Cl\u00e1udio Diniz\"}}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"How we ended up creating language agnostic data pipelines for our customers at Equal Experts | Equal Experts","description":"How we used two tools to create a scalable language agnostic data pipeline architecture that can be easily reused and adapted.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.equalexperts.com\/blog\/our-thinking\/how-we-ended-up-creating-language-agnostic-data-pipelines-for-our-customers-at-equal-experts\/","og_locale":"en_GB","og_type":"article","og_title":"How we ended up creating language agnostic data pipelines for our customers at Equal Experts","og_description":"Two tools to create a language agnostic data pipelines architecture that can be easily reused and adapted in multiple cases and for different clients plus a public reference implementation of this architecture","og_url":"https:\/\/www.equalexperts.com\/blog\/our-thinking\/how-we-ended-up-creating-language-agnostic-data-pipelines-for-our-customers-at-equal-experts\/","og_site_name":"Equal Experts","article_published_time":"2021-07-12T13:57:36+00:00","article_modified_time":"2024-03-28T13:49:24+00:00","og_image":[{"width":1200,"height":630,"url":"https:\/\/www.equalexperts.com\/wp-content\/uploads\/2021\/07\/Blog_Language_Agnostic_Data_Pipelines_Facebook_1200x630.jpg","type":"image\/jpeg"}],"author":"Cl\u00e1udio Diniz","twitter_card":"summary_large_image","twitter_title":"How we ended up creating language agnostic data pipelines for our customers at Equal Experts","twitter_description":"Two tools to create a language agnostic data pipelines architecture that can be easily reused and adapted in multiple cases and for different clients plus a public reference implementation of this architecture","twitter_image":"https:\/\/www.equalexperts.com\/wp-content\/uploads\/2021\/07\/Blog_Language_Agnostic_Data_Pipelines_Twitter_1024x512.jpg","twitter_creator":"@EqualExperts","twitter_site":"@EqualExperts","twitter_misc":{"Written by":"Cl\u00e1udio Diniz","Estimated reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.equalexperts.com\/blog\/our-thinking\/how-we-ended-up-creating-language-agnostic-data-pipelines-for-our-customers-at-equal-experts\/#article","isPartOf":{"@id":"https:\/\/www.equalexperts.com\/blog\/our-thinking\/how-we-ended-up-creating-language-agnostic-data-pipelines-for-our-customers-at-equal-experts\/"},"author":{"name":"Cl\u00e1udio Diniz","@id":"https:\/\/www.equalexperts.com\/#\/schema\/person\/28ff89d676b184c93ab62bc91b0af11e"},"headline":"How we ended up creating language agnostic data pipelines for our customers at Equal Experts","datePublished":"2021-07-12T13:57:36+00:00","dateModified":"2024-03-28T13:49:24+00:00","mainEntityOfPage":{"@id":"https:\/\/www.equalexperts.com\/blog\/our-thinking\/how-we-ended-up-creating-language-agnostic-data-pipelines-for-our-customers-at-equal-experts\/"},"wordCount":958,"publisher":{"@id":"https:\/\/www.equalexperts.com\/#organization"},"image":{"@id":"https:\/\/www.equalexperts.com\/blog\/our-thinking\/how-we-ended-up-creating-language-agnostic-data-pipelines-for-our-customers-at-equal-experts\/#primaryimage"},"thumbnailUrl":"https:\/\/www.equalexperts.com\/wp-content\/uploads\/2021\/07\/what-is-a-data-pipeline.png","keywords":["data engineering","data governance","data pipeline playbook","data pipline"],"articleSection":["Our Thinking"],"inLanguage":"en-GB"},{"@type":"WebPage","@id":"https:\/\/www.equalexperts.com\/blog\/our-thinking\/how-we-ended-up-creating-language-agnostic-data-pipelines-for-our-customers-at-equal-experts\/","url":"https:\/\/www.equalexperts.com\/blog\/our-thinking\/how-we-ended-up-creating-language-agnostic-data-pipelines-for-our-customers-at-equal-experts\/","name":"How we ended up creating language agnostic data pipelines for our customers at Equal Experts | Equal Experts","isPartOf":{"@id":"https:\/\/www.equalexperts.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.equalexperts.com\/blog\/our-thinking\/how-we-ended-up-creating-language-agnostic-data-pipelines-for-our-customers-at-equal-experts\/#primaryimage"},"image":{"@id":"https:\/\/www.equalexperts.com\/blog\/our-thinking\/how-we-ended-up-creating-language-agnostic-data-pipelines-for-our-customers-at-equal-experts\/#primaryimage"},"thumbnailUrl":"https:\/\/www.equalexperts.com\/wp-content\/uploads\/2021\/07\/what-is-a-data-pipeline.png","datePublished":"2021-07-12T13:57:36+00:00","dateModified":"2024-03-28T13:49:24+00:00","description":"How we used two tools to create a scalable language agnostic data pipeline architecture that can be easily reused and adapted.","breadcrumb":{"@id":"https:\/\/www.equalexperts.com\/blog\/our-thinking\/how-we-ended-up-creating-language-agnostic-data-pipelines-for-our-customers-at-equal-experts\/#breadcrumb"},"inLanguage":"en-GB","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.equalexperts.com\/blog\/our-thinking\/how-we-ended-up-creating-language-agnostic-data-pipelines-for-our-customers-at-equal-experts\/"]}]},{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/www.equalexperts.com\/blog\/our-thinking\/how-we-ended-up-creating-language-agnostic-data-pipelines-for-our-customers-at-equal-experts\/#primaryimage","url":"https:\/\/www.equalexperts.com\/wp-content\/uploads\/2021\/07\/what-is-a-data-pipeline.png","contentUrl":"https:\/\/www.equalexperts.com\/wp-content\/uploads\/2021\/07\/what-is-a-data-pipeline.png","width":1073,"height":601},{"@type":"BreadcrumbList","@id":"https:\/\/www.equalexperts.com\/blog\/our-thinking\/how-we-ended-up-creating-language-agnostic-data-pipelines-for-our-customers-at-equal-experts\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.equalexperts.com\/"},{"@type":"ListItem","position":2,"name":"How we ended up creating language agnostic data pipelines for our customers at Equal Experts"}]},{"@type":"WebSite","@id":"https:\/\/www.equalexperts.com\/#website","url":"https:\/\/www.equalexperts.com\/","name":"Equal Experts","description":"Making Software. Better.","publisher":{"@id":"https:\/\/www.equalexperts.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.equalexperts.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-GB"},{"@type":"Organization","@id":"https:\/\/www.equalexperts.com\/#organization","name":"Equal Experts","url":"https:\/\/www.equalexperts.com\/","logo":{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/www.equalexperts.com\/#\/schema\/logo\/image\/","url":"https:\/\/www.equalexperts.com\/wp-content\/uploads\/2018\/08\/Equal_Experts_Logo_CMYK_Colour.jpg","contentUrl":"https:\/\/www.equalexperts.com\/wp-content\/uploads\/2018\/08\/Equal_Experts_Logo_CMYK_Colour.jpg","width":719,"height":340,"caption":"Equal Experts"},"image":{"@id":"https:\/\/www.equalexperts.com\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/x.com\/EqualExperts","https:\/\/www.linkedin.com\/company\/equal-experts\/?viewAsMember=true"]},{"@type":"Person","@id":"https:\/\/www.equalexperts.com\/#\/schema\/person\/28ff89d676b184c93ab62bc91b0af11e","name":"Cl\u00e1udio Diniz","image":{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/www.equalexperts.com\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/d70fbe38b0540d312610b719e2e75bc9f302aafe3264bf1eb8174eb191c4879d?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/d70fbe38b0540d312610b719e2e75bc9f302aafe3264bf1eb8174eb191c4879d?s=96&d=mm&r=g","caption":"Cl\u00e1udio Diniz"}}]}},"_links":{"self":[{"href":"https:\/\/www.equalexperts.com\/wp-json\/wp\/v2\/posts\/9337","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.equalexperts.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.equalexperts.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.equalexperts.com\/wp-json\/wp\/v2\/users\/164"}],"replies":[{"embeddable":true,"href":"https:\/\/www.equalexperts.com\/wp-json\/wp\/v2\/comments?post=9337"}],"version-history":[{"count":0,"href":"https:\/\/www.equalexperts.com\/wp-json\/wp\/v2\/posts\/9337\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.equalexperts.com\/wp-json\/wp\/v2\/media?parent=9337"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.equalexperts.com\/wp-json\/wp\/v2\/categories?post=9337"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.equalexperts.com\/wp-json\/wp\/v2\/tags?post=9337"},{"taxonomy":"location","embeddable":true,"href":"https:\/\/www.equalexperts.com\/wp-json\/wp\/v2\/location?post=9337"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}