{"id":9349,"date":"2021-09-20T08:03:46","date_gmt":"2021-09-20T07:03:46","guid":{"rendered":"https:\/\/ee.yelkdev.site\/?p=9349"},"modified":"2024-03-28T13:48:30","modified_gmt":"2024-03-28T13:48:30","slug":"bringing-tdd-to-the-data-warehouse","status":"publish","type":"post","link":"https:\/\/www.equalexperts.com\/blog\/our-thinking\/bringing-tdd-to-the-data-warehouse\/","title":{"rendered":"Bringing TDD to the Data Warehouse"},"content":{"rendered":"<p><span style=\"font-weight: 400;\">As a follow-up from <\/span><a href=\"https:\/\/www.equalexperts.com\/blog\/our-thinking\/how-we-ended-up-creating-language-agnostic-data-pipelines-for-our-customers-at-equal-experts\/\"><span style=\"font-weight: 400;\">Language Agnostic Data Pipelines<\/span><\/a><span style=\"font-weight: 400;\">, the following post is focused on the use of <\/span><a href=\"https:\/\/www.getdbt.com\/\"><span style=\"font-weight: 400;\">dbt<\/span> <\/a><span style=\"font-weight: 400;\">(data build tool).<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Dbt is a command-line tool that enables us to transform the data inside a Data Warehouse by writing SQL select statements which represent the models. There is also a paid version with a web interface, dbt cloud, but for this article let\u2019s consider just the command-line tool.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">The intent of this article is not to make a tutorial about dbt &#8211; that already exists <\/span><a href=\"https:\/\/docs.getdbt.com\/tutorial\/setting-up\"><span style=\"font-weight: 400;\">here<\/span><\/a><span style=\"font-weight: 400;\">, nor one about TDD, the goal is to illustrate how one of our software development practices, test-driven development, can be used to develop the dbt models.<\/span><\/p>\n<h2><b>Testing strategies in dbt<\/b><\/h2>\n<p><span style=\"font-weight: 400;\">Dbt has two types of tests:<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Schema tests<\/b><span style=\"font-weight: 400;\">: Applied in YAML, returns the number of records that <\/span><i><span style=\"font-weight: 400;\">do not<\/span><\/i><span style=\"font-weight: 400;\"> pass an assertion \u2014 when this number is 0, all records pass and therefore your test passes.<\/span><span style=\"font-weight: 400;\"><img decoding=\"async\" class=\"aligncenter wp-image-9350 size-large\" src=\"https:\/\/www.equalexperts.com\/wp-content\/uploads\/2021\/07\/Claudio-TDD-blog-post-image-1-1200x500.png\" alt=\"\" width=\"1200\" height=\"500\" \/><\/span><\/li>\n<\/ul>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Data tests<\/b><span style=\"font-weight: 400;\">: Specific queries that return 0 records.<img decoding=\"async\" class=\"aligncenter wp-image-9351 size-full\" src=\"https:\/\/www.equalexperts.com\/wp-content\/uploads\/2021\/07\/Claudio-TDD-blog-post-image-2.png\" alt=\"\" width=\"1070\" height=\"206\" srcset=\"https:\/\/www.equalexperts.com\/wp-content\/uploads\/2021\/07\/Claudio-TDD-blog-post-image-2.png 1070w, https:\/\/www.equalexperts.com\/wp-content\/uploads\/2021\/07\/Claudio-TDD-blog-post-image-2-300x58.png 300w, https:\/\/www.equalexperts.com\/wp-content\/uploads\/2021\/07\/Claudio-TDD-blog-post-image-2-768x148.png 768w\" sizes=\"(max-width: 1070px) 100vw, 1070px\" \/><\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">Both tests can be used against staging\/production data to detect data quality issues.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">The second type of test gives us more freedom to write data quality tests. These tests run against a data warehouse loaded with data. They can run on production, on staging, or for instance against a test environment where a sample of data was loaded. These tests can be tied to a data pipeline so they can continuously test the ingested and transformed data.<\/span><\/p>\n<h2><b>Using dbt data tests to compare model results<\/b><\/h2>\n<p><span style=\"font-weight: 400;\">With a little bit of SQL creativity, the data tests (SQL selects) can be <\/span><b>naively*<\/b><span style=\"font-weight: 400;\"> used to test model transformations, comparing the result of a model with a set of expectations:<\/span><\/p>\n<pre><b>with<\/b><span style=\"font-weight: 400;\"> expectations <\/span><b>AS<\/b><span style=\"font-weight: 400;\"> (<\/span>\r\n\r\n<span style=\"font-weight: 400;\">\u00a0\u00a0\u00a0<\/span><b>select<\/b> <span style=\"font-weight: 400;\">'value'<\/span> <b>as<\/b><span style=\"font-weight: 400;\"> column1,<\/span>\r\n\r\n<span style=\"font-weight: 400;\">\u00a0\u00a0\u00a0<\/span><b>union<\/b> <b>all<\/b><span style=\"font-weight: 400;\">\u00a0<\/span>\r\n\r\n<span style=\"font-weight: 400;\">\u00a0\u00a0\u00a0<\/span><b>Select<\/b> <span style=\"font-weight: 400;\">'value 2'<\/span> <b>as<\/b><span style=\"font-weight: 400;\"> column1<\/span>\r\n\r\n<span style=\"font-weight: 400;\">)<\/span>\r\n\r\n\r\n<b>select<\/b><span style=\"font-weight: 400;\"> * <\/span><b>from<\/b><span style=\"font-weight: 400;\"> expectations<\/span>\r\n\r\n<b>except<\/b>\r\n\r\n<b>select<\/b><span style=\"font-weight: 400;\"> * <\/span><b>from<\/b><span style=\"font-weight: 400;\"> analytics.a_model<\/span><\/pre>\n<p><span style=\"font-weight: 400;\">The query returns results when the expectations differ, so in this case dbt reports a test failure. However, this methodology isn\u2019t effective to test the models due to the following facts:<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">The test input is shared among all the tests (this could be overcome by executing dbt test and the data setup for each test, although it\u2019s not practical due to the lack of clarity and the maintainability of test suites).<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">The test input is not located inside the test itself, so it\u2019s not user friendly to code nor easy to understand the goal of each test.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">The dbt test output doesn\u2019t show the differences between the expectations and the actual values, which slows down the development.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">For each test, we need to have a boilerplate query with the previous format (with expectations as&#8230;).<\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">Considering these drawbacks, It doesn&#8217;t seem like the right tool to make model transformation tests.<\/span><\/p>\n<h2><b>A strategy to introduce a kind of &#8216;data unit tests&#8217;<\/b><\/h2>\n<p><span style=\"font-weight: 400;\">It\u2019s possible and common to combine SQL with the templating engine Jinja (<\/span><a href=\"https:\/\/docs.getdbt.com\/docs\/building-a-dbt-project\/jinja-macros\"><span style=\"font-weight: 400;\">https:\/\/docs.getdbt.com\/docs\/building-a-dbt-project\/jinja-macros<\/span><\/a><span style=\"font-weight: 400;\">). Also, It\u2019s possible to define macros which can be used to extend dbt\u2019s functionalities. That being said, let&#8217;s introduce the following macro:<\/span><\/p>\n<pre><span style=\"font-weight: 400;\">unit_test(table_name, input, expectations)<\/span><\/pre>\n<p><span style=\"font-weight: 400;\">The macro receives:<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">A table name (or a view name).<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">An input value that contains a set of inserts.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">A table of expectations.<\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">To illustrate the usage of the macro, here is our last test case refactored:<\/span><\/p>\n<pre><span style=\"font-weight: 400;\">{% <\/span><b>set<\/b><span style=\"font-weight: 400;\"> table_name = <\/span><b>ref<\/b><span style=\"font-weight: 400;\">(<\/span><span style=\"font-weight: 400;\">'a_model'<\/span><span style=\"font-weight: 400;\">) %}<\/span>\r\n\r\n\r\n<span style=\"font-weight: 400;\">{% <\/span><b>set<\/b> <b>input<\/b><span style=\"font-weight: 400;\"> %}<\/span>\r\n\r\n<b>insert<\/b> <b>into<\/b><span style=\"font-weight: 400;\"> a_table(column1) <\/span><b>values<\/b><span style=\"font-weight: 400;\"> (\u2018<\/span><b>value<\/b><span style=\"font-weight: 400;\">\u2019), (\u2018value2\u2019);<\/span>\r\n\r\n<span style=\"font-weight: 400;\">{% endset %}<\/span>\r\n\r\n\r\n<span style=\"font-weight: 400;\">{% <\/span><b>set<\/b><span style=\"font-weight: 400;\"> expectations %}<\/span>\r\n\r\n<b>select<\/b> <span style=\"font-weight: 400;\">'value'<\/span> <b>as<\/b><span style=\"font-weight: 400;\"> column1,<\/span>\r\n\r\n<b>union<\/b> <b>all<\/b><span style=\"font-weight: 400;\">\u00a0<\/span>\r\n\r\n<b>select<\/b> <span style=\"font-weight: 400;\">'value 2'<\/span> <b>as<\/b><span style=\"font-weight: 400;\"> column1<\/span>\r\n\r\n<span style=\"font-weight: 400;\">{% endset %}<\/span>\r\n\r\n\r\n<span style=\"font-weight: 400;\">{{ unit_test(table_name, <\/span><b>input<\/b><span style=\"font-weight: 400;\">, expectations) }}<\/span><\/pre>\n<p><span style=\"font-weight: 400;\">There is some boilerplate when using Jinja to declare the variables to call the unit test macro. Although, it seems a nice tradeoff, because this strategy enables us to:<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Simplify the test query boilerplate.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Setup input data in plain SQL and in the same file.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Setup expectations in plain SQL and in the same file.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Run each test segregated from other tests.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Show differences when a test fails.<\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">To illustrate the usage of this approach, here is a demo video:<\/span><\/p>\n<p><span style=\"font-weight: 400;\"><br \/>\n<iframe title=\"YouTube video player\" src=\"https:\/\/www.youtube.com\/embed\/ND0n6k0Sxjc\" width=\"560\" height=\"315\" frameborder=\"0\" allowfullscreen=\"allowfullscreen\"><\/iframe><br \/>\n<\/span><span style=\"font-weight: 400;\">The previous macro will be available in the repo published with the <\/span><a href=\"https:\/\/www.equalexperts.com\/blog\/our-thinking\/how-we-ended-up-creating-language-agnostic-data-pipelines-for-our-customers-at-equal-experts\/\"><span style=\"font-weight: 400;\">Language Agnostic Data Pipelines<\/span><\/a><span style=\"font-weight: 400;\">.<\/span><\/p>\n<p><b>*naively<\/b><span style=\"font-weight: 400;\"> coded because the use of EXCEPT between both tables fails to detect if duplicate rows exist. It could be fixed easily, but for illustrative purposes, we preferred to maintain the example as simple as we can.<\/span><\/p>\n<h2><b>Bringing software engineering practices to the data world<\/b><\/h2>\n<p><span style=\"font-weight: 400;\">It is also easy to apply other standard software development practices such as integration with a ci\/cd environment in dbt. This\u00a0 is one of the advantages of using it over transforming data inside ETL tools which use a visual programming approach.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Wrapping up, we advocate that data oriented projects should always use the well-known software engineering best practices. We hope that this article shows how you can apply TDD\u00a0 using the\u00a0 emerging DBT data transformation tool.<\/span><\/p>\n<p><a href=\"https:\/\/www.linkedin.com\/in\/pedronsousa\/\"><span style=\"font-weight: 400;\">Pedro Sousa<\/span><\/a><span style=\"font-weight: 400;\">\u200b paired on this journey with me. He is taking the journey from software engineering to data engineering in our current project, and he helped on the blog post.<\/span><\/p>\n<h2><b>Contact us!<\/b><\/h2>\n<p><span style=\"font-weight: 400;\">For more information on data pipelines in general, take a look at our <\/span><a href=\"https:\/\/playbooks.equalexperts.com\/data-pipeline\"><span style=\"font-weight: 400;\">Data Pipeline Playbook<\/span><\/a><span style=\"font-weight: 400;\">.\u00a0 And if you\u2019d like us to share our experience of data pipelines with you, get in touch using the form below.<\/span><\/p>\n\n\t\t\t\t\t\t<script>\n\t\t\t\t\t\t\twindow.hsFormsOnReady = window.hsFormsOnReady || [];\n\t\t\t\t\t\t\twindow.hsFormsOnReady.push(()=>{\n\t\t\t\t\t\t\t\thbspt.forms.create({\n\t\t\t\t\t\t\t\t\tportalId: 7208712,\n\t\t\t\t\t\t\t\t\tformId: \"83acdf22-cf43-47ba-b91f-0428264b824a\",\n\t\t\t\t\t\t\t\t\ttarget: \"#hbspt-form-1758975567000-3550943157\",\n\t\t\t\t\t\t\t\t\tregion: \"eu1\",\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t})});\n\t\t\t\t\t\t<\/script>\n\t\t\t\t\t\t<div class=\"hbspt-form\" id=\"hbspt-form-1758975567000-3550943157\"><\/div>\n","protected":false},"excerpt":{"rendered":"<p>Learn why we advocate that data oriented projects should always use the well-known software engineering best practices.  This article shows how you can apply TDD  using the  emerging dbt data transformation tool.<\/p>\n","protected":false},"author":164,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"content-type":"","inline_featured_image":false,"footnotes":""},"categories":[5],"tags":[188,180,186,207],"location":[397],"class_list":["post-9349","post","type-post","status-publish","format-standard","hentry","category-our-thinking","tag-data-engineering","tag-data-pipeline-playbook","tag-data-pipelines","tag-data-warehouse"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v25.9 (Yoast SEO v25.9) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Bringing TDD to the Data Warehouse | Equal Experts<\/title>\n<meta name=\"description\" content=\"An illustration of how one of our software development practices, test-driven development, can be used to develop dbt models.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.equalexperts.com\/blog\/our-thinking\/bringing-tdd-to-the-data-warehouse\/\" \/>\n<meta property=\"og:locale\" content=\"en_GB\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Bringing TDD to the Data Warehouse\" \/>\n<meta property=\"og:description\" content=\"Learn why we advocate that data oriented projects should always use the well-known software engineering best practices. This article shows how you can apply TDD using the emerging dbt data transformation tool.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.equalexperts.com\/blog\/our-thinking\/bringing-tdd-to-the-data-warehouse\/\" \/>\n<meta property=\"og:site_name\" content=\"Equal Experts\" \/>\n<meta property=\"article:published_time\" content=\"2021-09-20T07:03:46+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-03-28T13:48:30+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.equalexperts.com\/wp-content\/uploads\/2021\/07\/Bringing_TDD_to_Data_Warehouse_FB.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"630\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Cl\u00e1udio Diniz\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:title\" content=\"Bringing TDD to the Data Warehouse\" \/>\n<meta name=\"twitter:description\" content=\"Learn why we advocate that data oriented projects should always use the well-known software engineering best practices. This article shows how you can apply TDD using the emerging dbt data transformation tool.\" \/>\n<meta name=\"twitter:creator\" content=\"@EqualExperts\" \/>\n<meta name=\"twitter:site\" content=\"@EqualExperts\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Cl\u00e1udio Diniz\" \/>\n\t<meta name=\"twitter:label2\" content=\"Estimated reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.equalexperts.com\/blog\/our-thinking\/bringing-tdd-to-the-data-warehouse\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.equalexperts.com\/blog\/our-thinking\/bringing-tdd-to-the-data-warehouse\/\"},\"author\":{\"name\":\"Cl\u00e1udio Diniz\",\"@id\":\"https:\/\/www.equalexperts.com\/#\/schema\/person\/28ff89d676b184c93ab62bc91b0af11e\"},\"headline\":\"Bringing TDD to the Data Warehouse\",\"datePublished\":\"2021-09-20T07:03:46+00:00\",\"dateModified\":\"2024-03-28T13:48:30+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.equalexperts.com\/blog\/our-thinking\/bringing-tdd-to-the-data-warehouse\/\"},\"wordCount\":827,\"publisher\":{\"@id\":\"https:\/\/www.equalexperts.com\/#organization\"},\"image\":{\"@id\":\"https:\/\/www.equalexperts.com\/blog\/our-thinking\/bringing-tdd-to-the-data-warehouse\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.equalexperts.com\/wp-content\/uploads\/2021\/07\/Claudio-TDD-blog-post-image-1-1200x500.png\",\"keywords\":[\"data engineering\",\"data pipeline playbook\",\"data pipelines\",\"data warehouse\"],\"articleSection\":[\"Our Thinking\"],\"inLanguage\":\"en-GB\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.equalexperts.com\/blog\/our-thinking\/bringing-tdd-to-the-data-warehouse\/\",\"url\":\"https:\/\/www.equalexperts.com\/blog\/our-thinking\/bringing-tdd-to-the-data-warehouse\/\",\"name\":\"Bringing TDD to the Data Warehouse | Equal Experts\",\"isPartOf\":{\"@id\":\"https:\/\/www.equalexperts.com\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.equalexperts.com\/blog\/our-thinking\/bringing-tdd-to-the-data-warehouse\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.equalexperts.com\/blog\/our-thinking\/bringing-tdd-to-the-data-warehouse\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.equalexperts.com\/wp-content\/uploads\/2021\/07\/Claudio-TDD-blog-post-image-1-1200x500.png\",\"datePublished\":\"2021-09-20T07:03:46+00:00\",\"dateModified\":\"2024-03-28T13:48:30+00:00\",\"description\":\"An illustration of how one of our software development practices, test-driven development, can be used to develop dbt models.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.equalexperts.com\/blog\/our-thinking\/bringing-tdd-to-the-data-warehouse\/#breadcrumb\"},\"inLanguage\":\"en-GB\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.equalexperts.com\/blog\/our-thinking\/bringing-tdd-to-the-data-warehouse\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\/\/www.equalexperts.com\/blog\/our-thinking\/bringing-tdd-to-the-data-warehouse\/#primaryimage\",\"url\":\"https:\/\/www.equalexperts.com\/wp-content\/uploads\/2021\/07\/Claudio-TDD-blog-post-image-1-1200x500.png\",\"contentUrl\":\"https:\/\/www.equalexperts.com\/wp-content\/uploads\/2021\/07\/Claudio-TDD-blog-post-image-1-1200x500.png\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.equalexperts.com\/blog\/our-thinking\/bringing-tdd-to-the-data-warehouse\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.equalexperts.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Bringing TDD to the Data Warehouse\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.equalexperts.com\/#website\",\"url\":\"https:\/\/www.equalexperts.com\/\",\"name\":\"Equal Experts\",\"description\":\"Making Software. Better.\",\"publisher\":{\"@id\":\"https:\/\/www.equalexperts.com\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.equalexperts.com\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-GB\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.equalexperts.com\/#organization\",\"name\":\"Equal Experts\",\"url\":\"https:\/\/www.equalexperts.com\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\/\/www.equalexperts.com\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.equalexperts.com\/wp-content\/uploads\/2018\/08\/Equal_Experts_Logo_CMYK_Colour.jpg\",\"contentUrl\":\"https:\/\/www.equalexperts.com\/wp-content\/uploads\/2018\/08\/Equal_Experts_Logo_CMYK_Colour.jpg\",\"width\":719,\"height\":340,\"caption\":\"Equal Experts\"},\"image\":{\"@id\":\"https:\/\/www.equalexperts.com\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/x.com\/EqualExperts\",\"https:\/\/www.linkedin.com\/company\/equal-experts\/?viewAsMember=true\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.equalexperts.com\/#\/schema\/person\/28ff89d676b184c93ab62bc91b0af11e\",\"name\":\"Cl\u00e1udio Diniz\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\/\/www.equalexperts.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/d70fbe38b0540d312610b719e2e75bc9f302aafe3264bf1eb8174eb191c4879d?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/d70fbe38b0540d312610b719e2e75bc9f302aafe3264bf1eb8174eb191c4879d?s=96&d=mm&r=g\",\"caption\":\"Cl\u00e1udio Diniz\"}}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Bringing TDD to the Data Warehouse | Equal Experts","description":"An illustration of how one of our software development practices, test-driven development, can be used to develop dbt models.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.equalexperts.com\/blog\/our-thinking\/bringing-tdd-to-the-data-warehouse\/","og_locale":"en_GB","og_type":"article","og_title":"Bringing TDD to the Data Warehouse","og_description":"Learn why we advocate that data oriented projects should always use the well-known software engineering best practices. This article shows how you can apply TDD using the emerging dbt data transformation tool.","og_url":"https:\/\/www.equalexperts.com\/blog\/our-thinking\/bringing-tdd-to-the-data-warehouse\/","og_site_name":"Equal Experts","article_published_time":"2021-09-20T07:03:46+00:00","article_modified_time":"2024-03-28T13:48:30+00:00","og_image":[{"width":1200,"height":630,"url":"https:\/\/www.equalexperts.com\/wp-content\/uploads\/2021\/07\/Bringing_TDD_to_Data_Warehouse_FB.jpg","type":"image\/jpeg"}],"author":"Cl\u00e1udio Diniz","twitter_card":"summary_large_image","twitter_title":"Bringing TDD to the Data Warehouse","twitter_description":"Learn why we advocate that data oriented projects should always use the well-known software engineering best practices. This article shows how you can apply TDD using the emerging dbt data transformation tool.","twitter_creator":"@EqualExperts","twitter_site":"@EqualExperts","twitter_misc":{"Written by":"Cl\u00e1udio Diniz","Estimated reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.equalexperts.com\/blog\/our-thinking\/bringing-tdd-to-the-data-warehouse\/#article","isPartOf":{"@id":"https:\/\/www.equalexperts.com\/blog\/our-thinking\/bringing-tdd-to-the-data-warehouse\/"},"author":{"name":"Cl\u00e1udio Diniz","@id":"https:\/\/www.equalexperts.com\/#\/schema\/person\/28ff89d676b184c93ab62bc91b0af11e"},"headline":"Bringing TDD to the Data Warehouse","datePublished":"2021-09-20T07:03:46+00:00","dateModified":"2024-03-28T13:48:30+00:00","mainEntityOfPage":{"@id":"https:\/\/www.equalexperts.com\/blog\/our-thinking\/bringing-tdd-to-the-data-warehouse\/"},"wordCount":827,"publisher":{"@id":"https:\/\/www.equalexperts.com\/#organization"},"image":{"@id":"https:\/\/www.equalexperts.com\/blog\/our-thinking\/bringing-tdd-to-the-data-warehouse\/#primaryimage"},"thumbnailUrl":"https:\/\/www.equalexperts.com\/wp-content\/uploads\/2021\/07\/Claudio-TDD-blog-post-image-1-1200x500.png","keywords":["data engineering","data pipeline playbook","data pipelines","data warehouse"],"articleSection":["Our Thinking"],"inLanguage":"en-GB"},{"@type":"WebPage","@id":"https:\/\/www.equalexperts.com\/blog\/our-thinking\/bringing-tdd-to-the-data-warehouse\/","url":"https:\/\/www.equalexperts.com\/blog\/our-thinking\/bringing-tdd-to-the-data-warehouse\/","name":"Bringing TDD to the Data Warehouse | Equal Experts","isPartOf":{"@id":"https:\/\/www.equalexperts.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.equalexperts.com\/blog\/our-thinking\/bringing-tdd-to-the-data-warehouse\/#primaryimage"},"image":{"@id":"https:\/\/www.equalexperts.com\/blog\/our-thinking\/bringing-tdd-to-the-data-warehouse\/#primaryimage"},"thumbnailUrl":"https:\/\/www.equalexperts.com\/wp-content\/uploads\/2021\/07\/Claudio-TDD-blog-post-image-1-1200x500.png","datePublished":"2021-09-20T07:03:46+00:00","dateModified":"2024-03-28T13:48:30+00:00","description":"An illustration of how one of our software development practices, test-driven development, can be used to develop dbt models.","breadcrumb":{"@id":"https:\/\/www.equalexperts.com\/blog\/our-thinking\/bringing-tdd-to-the-data-warehouse\/#breadcrumb"},"inLanguage":"en-GB","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.equalexperts.com\/blog\/our-thinking\/bringing-tdd-to-the-data-warehouse\/"]}]},{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/www.equalexperts.com\/blog\/our-thinking\/bringing-tdd-to-the-data-warehouse\/#primaryimage","url":"https:\/\/www.equalexperts.com\/wp-content\/uploads\/2021\/07\/Claudio-TDD-blog-post-image-1-1200x500.png","contentUrl":"https:\/\/www.equalexperts.com\/wp-content\/uploads\/2021\/07\/Claudio-TDD-blog-post-image-1-1200x500.png"},{"@type":"BreadcrumbList","@id":"https:\/\/www.equalexperts.com\/blog\/our-thinking\/bringing-tdd-to-the-data-warehouse\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.equalexperts.com\/"},{"@type":"ListItem","position":2,"name":"Bringing TDD to the Data Warehouse"}]},{"@type":"WebSite","@id":"https:\/\/www.equalexperts.com\/#website","url":"https:\/\/www.equalexperts.com\/","name":"Equal Experts","description":"Making Software. Better.","publisher":{"@id":"https:\/\/www.equalexperts.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.equalexperts.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-GB"},{"@type":"Organization","@id":"https:\/\/www.equalexperts.com\/#organization","name":"Equal Experts","url":"https:\/\/www.equalexperts.com\/","logo":{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/www.equalexperts.com\/#\/schema\/logo\/image\/","url":"https:\/\/www.equalexperts.com\/wp-content\/uploads\/2018\/08\/Equal_Experts_Logo_CMYK_Colour.jpg","contentUrl":"https:\/\/www.equalexperts.com\/wp-content\/uploads\/2018\/08\/Equal_Experts_Logo_CMYK_Colour.jpg","width":719,"height":340,"caption":"Equal Experts"},"image":{"@id":"https:\/\/www.equalexperts.com\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/x.com\/EqualExperts","https:\/\/www.linkedin.com\/company\/equal-experts\/?viewAsMember=true"]},{"@type":"Person","@id":"https:\/\/www.equalexperts.com\/#\/schema\/person\/28ff89d676b184c93ab62bc91b0af11e","name":"Cl\u00e1udio Diniz","image":{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/www.equalexperts.com\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/d70fbe38b0540d312610b719e2e75bc9f302aafe3264bf1eb8174eb191c4879d?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/d70fbe38b0540d312610b719e2e75bc9f302aafe3264bf1eb8174eb191c4879d?s=96&d=mm&r=g","caption":"Cl\u00e1udio Diniz"}}]}},"_links":{"self":[{"href":"https:\/\/www.equalexperts.com\/wp-json\/wp\/v2\/posts\/9349","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.equalexperts.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.equalexperts.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.equalexperts.com\/wp-json\/wp\/v2\/users\/164"}],"replies":[{"embeddable":true,"href":"https:\/\/www.equalexperts.com\/wp-json\/wp\/v2\/comments?post=9349"}],"version-history":[{"count":0,"href":"https:\/\/www.equalexperts.com\/wp-json\/wp\/v2\/posts\/9349\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.equalexperts.com\/wp-json\/wp\/v2\/media?parent=9349"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.equalexperts.com\/wp-json\/wp\/v2\/categories?post=9349"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.equalexperts.com\/wp-json\/wp\/v2\/tags?post=9349"},{"taxonomy":"location","embeddable":true,"href":"https:\/\/www.equalexperts.com\/wp-json\/wp\/v2\/location?post=9349"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}