{"id":10354,"date":"2021-11-23T08:17:24","date_gmt":"2021-11-23T08:17:24","guid":{"rendered":"https:\/\/ee.yelkdev.site\/?p=10354"},"modified":"2023-09-21T23:49:56","modified_gmt":"2023-09-21T22:49:56","slug":"writing-unit-tests-for-dbt-with-tdd","status":"publish","type":"post","link":"https:\/\/www.equalexperts.com\/blog\/our-thinking\/writing-unit-tests-for-dbt-with-tdd\/","title":{"rendered":"Writing Unit Tests for dbt – with TDD"},"content":{"rendered":"

This blog describes the exploration that <\/span>Cl\u00e1udio Diniz<\/span><\/a> and I have been doing around the dbt tool, trying to find a way to write unit tests using TDD. You can check this <\/span>post<\/span><\/a> to understand how it all started.<\/span><\/p>\n

This is not a tutorial on dbt, and it\u2019s not a detailed technical document on our framework. You can check our demo dbt project <\/span>here<\/span><\/a> if you want to see the details.<\/span><\/p>\n

Note: What we are calling a framework is a set of Jinja macros. Throughout this document, we will call it a framework for simplicity.<\/span><\/i><\/p>\n

Why do we need unit tests in dbt?<\/b><\/h2>\n

One of the software engineering practices that we follow is TDD. We strongly believe that it helps us create better code and achieve a good level of confidence in its correctness. So, why not use this in our dbt projects? \"\"<\/span><\/p>\n

Dbt is code<\/b><\/h3>\n

A dbt project, like a \u201cregular\u201d software project, consists of several files of SQL code. And we want to be sure that the logic of this code is correct before it jumps to production.<\/span><\/p>\n

We think that using TDD can also help us to write better SQL code. Using TDD, we write the test, make it pass with the simplest SQL code we can think of, and then refactor it, knowing we have a test to assure it still works. It would be awesome if we could do this on our dbt projects! <\/span><\/p>\n

But wait, dbt already has tests<\/b><\/h3>\n

That\u2019s right, but dbt tests were mainly designed for data tests. They are used to check the correctness of the data produced by the models. We want to write unit tests on our SQL code. We want to be sure that the logic of the code is correct.<\/span><\/p>\n

A word on <\/b>unit tests<\/i><\/b><\/h3>\n

The line between unit and integration tests is sometimes a bit blurred. This is also true with these tests in dbt.<\/span><\/p>\n

We can think of a dbt model as a function, where the inputs are other models, and the output is the result of its SQL. A unit test in dbt would be a test where we provide fake inputs to a model, and then we check the results against some expectations.<\/span><\/p>\n

However, a model in dbt can belong to a long chain of models (see image below), each transforming the data in its own rules. We could test a model by providing fake inputs to the first models in that chain and asserting the results on the final model. We would be checking all the intermediate transformations along the way. This, on the other end, could be called an integration test.<\/span><\/p>\n

\"\"

A complex dbt model hierarchy<\/p><\/div>\n

These integration tests are harder to write because we have to think of how the data is transformed throughout all those models until it reaches the model we want to test. However, they provide an extra level of confidence. As usual, we need to keep a good balance between these two types of tests.<\/span><\/p>\n

As we will see, using this definition, our framework will allow us to create both unit and integration tests.<\/span><\/p>\n

What do you want to achieve?<\/b><\/h2>\n

We came up with a list of requirements that we would like to fulfill when writing tests in dbt:<\/span><\/p>\n