{"id":10030,"date":"2021-09-07T13:23:31","date_gmt":"2021-09-07T12:23:31","guid":{"rendered":"https:\/\/ee.yelkdev.site\/?p=10030"},"modified":"2024-04-16T09:48:09","modified_gmt":"2024-04-16T08:48:09","slug":"data-tips-the-data-warehouse","status":"publish","type":"post","link":"https:\/\/www.equalexperts.com\/blog\/our-thinking\/data-tips-the-data-warehouse\/","title":{"rendered":"Data tips – the data warehouse"},"content":{"rendered":"

Whatever your role, you might come into contact with a data warehouse at some point. They’re incredibly powerful tools that I’ve frequently seen ignored or used in ways that cause problems. In this post, I’ll share a few tips for handling data in your data warehouse effectively. Whilst the focus here is helping data scientists, these tips have helped other tech professionals answer their own data questions and avoid unnecessary work, too.<\/span><\/p>\n

Gaps in traditional data science training<\/b><\/h2>\n

Traditional data science training focuses on academic techniques, not practicalities of dealing with data in the real world. The odds are you’ll learn five different ways to perform a clustering, but you’ll not write a test or a SQL query.<\/span><\/p>\n

Fresh data scientists, landing in their first professional roles, can find that the reality of data science is not what they expected. They must deal with messy and unreliable terabyte-scale datasets that they can’t just load into a Pandas or R dataframe to clean and refine. Their code needs to run efficiently enough to be useful and cost-effective. It needs to run in environments other than their laptop, robustly interact with remote services and be maintained over time and by other people.<\/span><\/p>\n

This mismatch between training, expectation and reality can cause real problems*. At one end of the scale, you get data scientists producing code that needs to be “productionised” (well, rewritten) – by “data engineers” to be fit for its real-world purpose. At the scarier end of the scale you get data science time investments that have to be thrown in the bin <\/span> because a couple of early mistakes mean it just doesn’t work in practice.<\/span><\/p>\n

(* this blog post isn\u2019t about the other big expectation mismatch – how you\u2019ll likely be spending a good chunk of your time doing pretty basic things like writing simple reports, not the cool data science stuff… sorry <\/span><\/i>)<\/span><\/i><\/p><\/blockquote>\n

Data science training is adapting to make more room for practical skills. Here’s a few tips to avoid making mistakes I\u2019ve helped fix for fresh and seasoned data scientists out there in the field today.<\/span><\/p>\n

Use your data warehouse effectively<\/b><\/h1>\n

There\u2019s a good chance that your data science is applied to data that came from a SQL-based analytics system like <\/span>Google BigQuery<\/span><\/a>, <\/span>AWS Redshift<\/span><\/a>, <\/span>Azure Analytics<\/span><\/a>, <\/span>Snowflake<\/span><\/a> or even <\/span>Spark SQL<\/span><\/a>. These systems are engineered to efficiently answer complex queries against big datasets, not to serve up large quantities of raw data. You might be able to:<\/span><\/p>\n

SELECT<\/span>\r\n\r\n\u00a0\u00a0*<\/span>\r\n\r\nFROM client_transactions<\/span><\/pre>\n
 <\/p>\n
…streaming out fifty columns by ten million rows to find the top ten clients by spend in <\/span>Pandas<\/span><\/a>, but it\u2019ll be an awful lot faster and more robust if you do more work in the data warehouse and retrieve just the ten rows and two columns you’re really interested in:<\/span><\/p>\n
SELECT<\/span>\r\n\r\n\u00a0\u00a0client_id,<\/span>\r\n\r\n\u00a0\u00a0SUM(transaction_value) total_spend<\/span>\r\n\r\nFROM<\/span>\r\n\r\n\u00a0\u00a0client_transactions<\/span>\r\n\r\nGROUP BY<\/span>\r\n\r\n\u00a0\u00a0client_id<\/span>\r\n\r\nORDER BY<\/span>\r\n\r\n\u00a0\u00a0total_spend DESC<\/span>\r\n\r\nLIMIT 10<\/span><\/pre>\n
 <\/p>\n
As the table you’re accessing grows, the benefits of doing the work in the data warehouse will too.<\/span><\/p>\n
I recall working with a data scientist, fresh out of university, who did not want to learn SQL and took the former approach. It worked well enough while the product was in development and soft launch. Three months after public launch, they were trying to download tens of gigabytes of data from the data warehouse. We ran out of hardware and networking options to keep the complex analytics pipeline they had built on their laptop running and it ground to an undignified halt.<\/span><\/i><\/p><\/blockquote>\n
Often, some of the work has to be done in your code outside the data warehouse. These tips encourage you to do as much of the work as you can in your data warehouse before you pull just what you need out to process in your code. Modern data warehouses are capable of doing more right there in the warehouse, <\/span>including<\/span><\/a> building<\/span><\/a> machine learning models.<\/span><\/p>\n
We\u2019ll focus on SQL for the next few tips as it\u2019s so prevalent, but similar principles and capabilities are often available in whatever data storage and parallel processing capability you are using. The tips apply in principle to any query language, so you may be able to apply some of them when working with systems like <\/span>Neo4J<\/span><\/a> or <\/span>MongoDB<\/span><\/a>. I’ll typically refer to Google’s BigQuery data warehouse, as I find it has excellent SQL feature coverage and the documentation is clear and well-written.<\/span><\/p>\n
Tip 1: Learn and use the query language<\/b><\/h2>\n
SQL<\/span><\/a> has been around for a long time – even older than I have (just). The <\/span>foundations were laid back in 1970<\/span><\/a>. It\u2019s an enormously expressive and powerful language. You can probably do more than you think with it, and making the effort to try and push as much work as you can into the database, building your SQL knowledge as you go, will pay dividends quickly. You get the added bonus that non-data scientists could understand and be able to help with your queries, as SQL is commonly used by other data folk that perform analytics and reporting functions.<\/span><\/p>\n
There\u2019s a lot more power and flexibility in modern SQL than the basic\u00a0SELECT<\/code>,\u00a0WHERE<\/code>, and\u00a0GROUP BY<\/code>\u00a0would suggest.\u00a0Every database has a query optimiser that looks at your query and statistics about the tables involved before selecting a strategy to answer most efficiently. <\/span>Self-joins<\/span><\/a>, <\/span>nested columns, STRUCT types<\/span><\/a>, <\/span>user-defined functions<\/span><\/a> and <\/span>window or analytic functions<\/span><\/a> are a few capabilities common to many data warehouses that I\u2019ve found very helpful in expressively solving trickier problems, and they are less-well known amongst data scientists.<\/span><\/p>\n
I found the first couple of paragraphs in <\/span><\/i>BigQuery’s window function documentation<\/span><\/i><\/a> provided a really useful mental model of what they are, how they differ from normal aggregations and when they might be useful.<\/span><\/i><\/p><\/blockquote>\n
Once you start using SQL, you\u2019ll find yourself writing larger and more complex queries. Let’s talk about how you can control that complexity and keep your queries manageable.<\/span><\/p>\n
Tip 2: Use common table expressions<\/b><\/h2>\n
You can think of <\/span>Common Table Expressions (CTEs)<\/span><\/a> as virtual tables within your query. As your simple query becomes more complex, you might be tempted to bolt together multiple sub-queries like this:<\/span><\/p>\n
SELECT a, b, c<\/span>\r\n\r\nFROM (SELECT ...)<\/span>\r\n\r\nINNER JOIN (SELECT ...)<\/span><\/pre>\n
 <\/p>\n
This will work, and I\u2019ve seen plenty of queries done this way. They can easily grow into SQL behemoths hundreds of lines long that are really difficult to understand and reason about. The sub-queries are a big part of the problem, nesting complex logic inside other queries. Instead of creating a large, monolithic query, you can pull out those complex queries into CTEs and give them meaningful names.<\/span><\/p>\n
WITH<\/span>\r\n\r\n\u00a0\u00a0\u00a0\u00a0clients AS (SELECT ....),<\/span>\r\n\r\n\u00a0\u00a0\u00a0\u00a0transactions AS (SELECT \u2026)<\/span>\r\n\r\nSELECT a, b, c<\/span>\r\n\r\nFROM clients<\/span>\r\n\r\nINNER JOIN transactions<\/span><\/pre>\n
 <\/p>\n
Now you can see what that query is all about. As well as making your queries easier to understand, breaking up a large query like this means you can query the CTEs to see what they are producing while you develop and debug your query. <\/span>Here<\/span><\/a> is a more in-depth tutorial on CTEs.<\/span><\/p>\n
Equal Experts Associate Dawid du Toit highlights one thing to watch out for when using CTEs. If you use the CTE multiple times during your query (for example, a self-join from the CTE back onto itself) you may find the CTE evaluated twice. In this, materializing the data into a temporary table may be a better option if the cost of the query is a significant concern.<\/span><\/i><\/p><\/blockquote>\n
Tip 3: Use views<\/b><\/h2>\n
So you\u2019ve started using CTEs and you\u2019re writing much more expressive queries. Things can still become complicated, and it\u2019s not always convenient to change your query to inspect your CTEs – for example, when your query is running out there in production and can\u2019t be modified.<\/span><\/p>\n
Once you\u2019ve got a CTE, it\u2019s straightforward to convert into a database <\/span>view<\/span><\/a>. A view is just a query you define in the database, so you can just take your CTE and wrap it in a <\/span>CREATE VIEW<\/strong> statement. You can then use the view as if it were a table – which means you can query it and inspect the data it contains at any time without modifying your deployed queries.<\/span><\/p>\n
Views are usually heavily optimised and don’t have to introduce any performance or cost penalty. For example, implementing “<\/span><\/i>predicate pushdown<\/span><\/i><\/a>” allows a data warehouse to optimise the data and computation the view uses based on how it is queried – as if the view was just part of the query.<\/span><\/i><\/p><\/blockquote>\n
Views are logical infrastructure, and lend themselves to deployment with an “infrastructure as code” mindset the same way as we might deploy a server or the database itself.<\/span><\/p>\n
CREATE VIEW clients AS (SELECT ....);<\/span>\r\n\r\nCREATE VIEW transactions AS (SELECT \u2026);<\/span><\/pre>\n
 <\/p>\n
Then, we can use these views in queries, whether executed ad-hoc or as part of a data pipeline.<\/span><\/p>\n
SELECT a, b, c<\/span>\r\n\r\nFROM clients<\/span>\r\n\r\nINNER JOIN transactions<\/span><\/pre>\n
 <\/p>\n
Database views provide a much more flexible interface to underlying tables. The underlying data will eventually need to change. If consumers access it through views, the views can adapt to protect those consumers from the change, avoiding broken queries and irate consumers.<\/span><\/i><\/p><\/blockquote>\n
Views are also a great way to avoid data engineering complexity. I’ve seen plenty of complex data processing pipelines that populate tables with refined results from other tables, ready for further processing. In every case I can recall, a view is a viable alternative approach, and it avoids managing the data in intermediate tables. No complex data synchronisation like backfilling, scheduling or merging challenges here! If your data warehouse supports it, a <\/span>materialized view<\/span><\/a> can work where there are genuine reasons, such as performance or cost, to avoid directly querying the source table.<\/span><\/p>\n
Tip 4: Use user defined functions<\/b><\/h2>\n
Databases have supported <\/span>User Defined Functions (UDFs)<\/span><\/a> for a long time. This useful capability allows you to capture logic right in the database, give it a meaningful name, and reuse it in different queries. A simple example of a UDF might parse a list from a comma-separated string, and return the first element. For example, given a function like this (using the BigQuery SQL dialect)<\/span><\/p>\n
CREATE FUNCTION my_dataset.get_first_post(post_list STRING)<\/span>\r\n\r\nRETURNS STRING<\/span>\r\n\r\nAS (<\/span>\r\n\r\n\u00a0\u00a0SPLIT(post_list, ',')[SAFE_OFFSET(0)]<\/span>\r\n\r\n)<\/span><\/pre>\n
 <\/p>\n
Specify the return type of your query, even if it can be inferred. That way, you’ll know if you’re returning something totally different to what you expected.<\/span><\/i><\/p><\/blockquote>\n
We can use the function in other queries:<\/span><\/p>\n
WITH post_lists AS (<\/span><\/p>\n
\u00a0\u00a0SELECT ‘a,b,c,d’ AS post_list<\/span><\/p>\n
)<\/span><\/p>\n
SELECT<\/span><\/p>\n
\u00a0\u00a0post_list,<\/span><\/p>\n
\u00a0\u00a0my_dataset.get_first_post(post_list) first_post<\/span><\/p>\n
FROM post_lists<\/span><\/p>\n
 <\/p>\n
The result would be as follows:<\/span><\/p>\n\n\n\n\n
post_list<\/b><\/td>\nfirst_post<\/b><\/td>\n<\/tr>\n
a,b,c,d<\/span><\/td>\na<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
 <\/p>\n
The benefits might not be clear from this simple function, but I recently implemented a <\/span>Levenshtein Distance<\/span><\/a> function using <\/span>BigQuery’s JavaScript UDF capability<\/span><\/a>, allowing us to find likely typos in a large dataset in the data warehouse.<\/span><\/p>\n
Dawid recalls seeing UDFs that take a parameter which then go and query another table. This can completely kill your query performance, for example, where you could just have just joined with the other table. Be wary of using UDFs in this way!<\/span><\/i><\/p><\/blockquote>\n
A benefit that might not be immediately obvious is that you can deploy a fix or improvement to your UDF and all queries using it pick up the change when they are next executed. Be aware that there may be little of the structural flexibility you might be used to in other programming languages. For example, you might not be able to “overload” a UDF to handle an additional argument, nor can you provide a default value to an argument.<\/span><\/p>\n
These limitations can make handling change even harder than in other languages – but I’ve found it’s still a big improvement over copying and pasting.<\/span><\/p>\n
Tip 5: Understand what\u2019s efficient<\/b><\/h2>\n
Doing your number-crunching in your data warehouse is likely to be the most cost-efficient place to do it. After all, modern data warehouses are really just massively parallel compute clusters.<\/span><\/p>\n
There are typically ways to make your processing more efficient – which always means cheaper and usually means faster. For example, a clickstream table might stretch back five years and be 150Tb in total size. <\/span>Partitioning<\/span><\/a> that table by the event date means that the data warehouse can just scan yesterday\u2019s data rather than the whole table to give you a total of some metric for yesterday – so long as you construct your query to take advantage of it. In this case, it’s likely as straightforward as ensuring your query\u00a0WHERE<\/code> clause contains a simple date constraint on the partition column.<\/span><\/p>\n
Understanding how your data warehouse is optimised (or how to optimise it, if you\u2019re setting up the table), can make a real difference to your query performance and cost.<\/span><\/i><\/p><\/blockquote>\n
Writing better SQL, faster<\/b><\/h1>\n
As you gain competency in writing SQL (or any language), you\u2019ll find your queries and programs becoming more complex. That will make tracking down problems more difficult. Here\u2019s a few tips to help you find and squash your bugs more easily, be more confident that your code does what it\u2019s supposed to, and avoid the problems in the first place!<\/span><\/p>\n
Tip 6: Test your UDFs<\/b><\/h2>\n
You can quite easily test your UDFs. Here\u2019s a cut-down version of the query I wrote to test that Levenshtein distance function I mentioned earlier:<\/span><\/p>\n
--- create a virtual table of function inputs and expected outputs, via a CTE<\/span>\r\n\r\nWITH examples AS (<\/span>\r\n\r\n\u00a0\u00a0\u00a0\u00a0SELECT 'kittens' AS term_1, 'mittens' AS term_2, 1 AS expected<\/span>\r\n\r\n\u00a0\u00a0\u00a0\u00a0UNION ALL SELECT 'kittens', 'kittens', 0<\/span>\r\n\r\n)<\/span>\r\n\r\n\r\n\r\n\r\n--- create a new CTE that injects the actual outputs from the function<\/span>\r\n\r\n, test AS (<\/span>\r\n\r\n\u00a0\u00a0\u00a0\u00a0SELECT<\/span>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0*,<\/span>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0my_dataset.levenshtein_distance(term_1, term_2) actual<\/span>\r\n\r\n\u00a0\u00a0\u00a0\u00a0FROM examples<\/span>\r\n\r\n)<\/span>\r\n\r\n\r\n\r\n\r\nSELECT<\/span>\r\n\r\n\u00a0\u00a0\u00a0\u00a0*,<\/span>\r\n\r\n\u00a0\u00a0\u00a0\u00a0IF(TO_JSON_STRING(actual) = TO_JSON_STRING(expected), 'pass', 'fail') result<\/span>\r\n\r\nFROM tes<\/span>\r\n\r\n\r\n<\/pre>\n
Results:<\/span><\/p>\n\n\n\n\n\n
term_1<\/b><\/td>\nterm_2<\/b><\/td>\nexpected<\/b><\/td>\nactual<\/b><\/td>\nresult<\/b><\/td>\n<\/tr>\n
kittens<\/span><\/td>\nmittens<\/span><\/td>\n1<\/span><\/td>\n1<\/span><\/td>\npass<\/span><\/td>\n<\/tr>\n
kittens<\/span><\/td>\nkittens<\/span><\/td>\n0<\/span><\/td>\n0<\/span><\/td>\npass<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
 <\/p>\n
If the query produces any rows that do not contain ‘pass’ in the result column, we know that there was an issue – that the actual function output did not match our expectation.<\/span><\/p>\n
Although not needed for these examples, I typically use a function like\u00a0TO_JSON_STRING<\/code><\/span><\/i>\u00a0that \u201cstringifies\u201d the expected and actual content in a deterministic manner so that structured values and NULL are compared correctly. Be aware that <\/span><\/i>the documentation for TO_JSON_STRING<\/span><\/i><\/a> does not guarantee deterministic order of keys in STRUCTs, but I’ve seen no problems in practice.<\/span><\/i><\/p><\/blockquote>\n
Tests have one more huge benefit that might not be immediately obvious. When you do data science for an organisation, it\u2019s likely that someone else will one day need to understand your work. They might even need to make changes. Having some runnable tests means that both of those things will be easier, and that person will be much more likely to think fondly of you.<\/span><\/p>\n
Tip 7: Test your views<\/b><\/h2>\n
Your views will depend on tables, and as you use these techniques you\u2019ll find that you end with views that depend on other views. It\u2019s handy to know that your views actually do what they are supposed to. You can test views in a similar fashion to UDFs, but it is a little more involved because unlike a UDF, a view must explicitly refer to the tables or views that provide the data it uses.<\/span><\/p>\n
I have rolled my own testing solutions for these networks of views before. You can create test data, for example in the form of CSV files, that you upload to temporary tables. You can then create a temporary version of your views that use these tables with known, fixed content. The same testing approach outlined for UDFs can now check that your views produce the right data, for the same benefits!<\/span><\/p>\n
A templating engine like <\/span><\/i>Jinja templates<\/span><\/i><\/a> and <\/span><\/i>envsubst<\/span><\/i><\/a> can provide a relatively straightforward way to change values in queries and views that vary between environments using environment variables. A query containing\u00a0{{my_dataset}}.table<\/code>\u00a0can be adjusted to\u00a0test.my_table<\/code>\u00a0in the test environment and\u00a0prod.my_table<\/code>\u00a0in production.<\/span><\/i><\/p><\/blockquote>\n
Tip 8: Use dbt<\/b><\/h2>\n
I\u2019ll also mention some tooling here. We’ve found dbt quite effective for managing networks of views, like those described above. It directly supports flexible references for source views and tables, provides built-in data and view testing capabilities and can even generate documentation, as in <\/span>this example<\/span><\/a>. <\/span>Dataform<\/span> is an alternative providing similar capabilities. Given that it\u2019s recently been acquired by Google, it\u2019ll likely be making an appearance as part of GCP soon, (thanks to Equal Experts Associate Gareth Rowlands for pointing this out!)<\/span><\/p>\n
Data Studio team member <\/span><\/i>Cl\u00e1udio Diniz<\/span><\/i><\/a> has also been using dbt. You can find more details in <\/span><\/i>his post<\/span><\/i><\/a> and the related example <\/span><\/i>Github repo<\/span><\/i><\/a>.\u00a0<\/span><\/i><\/p><\/blockquote>\n
Tip 9: Your time is valuable <\/b><\/h2>\n
Organisations often promote directives to save costs wherever possible. It\u2019s not the first time I\u2019ve come across posters, screens and mailshots for employees that remind staff to think about how much things cost and save every penny they can. In principle, that\u2019s a great directive. Unfortunately, it can create focus on small costs that are easy to quantify at the expense of larger costs that are more obscure.<\/span><\/p>\n
It\u2019s great to create focus on a query that would process 10Tb but could execute for 1\/1000th that cost with the addition of a\u00a0WHERE<\/code>\u00a0clause that takes advantage of partitioning. Why not save easy money?<\/p>\n
On the other hand, I\u2019ve seen occasions when hours or days of work are invested to make a query more efficient, perhaps saving 2p per day at the expense of – let\u2019s be conservative – two days of a data scientists\u2019 time. At an average salary of \u00a350k, that means we\u2019re looking at a <\/span>\u00a3400 investment to save \u00a37.30 per year<\/b> – and we\u2019re ignoring the <\/span>opportunity cost<\/b> that your time could be spent working on something else.<\/span><\/p>\n
This is a naive estimate. How much you cost your company is complicated. I\u2019m using the gross salary as an estimate of employee cost. I think the true cost of a typical employee to a typical company is still likely to be higher than the gross salary after factors like tax breaks and equipment costs are taken into account.<\/span><\/i><\/p><\/blockquote>\n
Don\u2019t forget to consider what your time is worth!<\/span><\/p>\n
Bonus Tip: Google it!<\/b><\/h2>\n
The last tip I\u2019d share with you is to check for existing solutions before you build it yourself. The odds are that the problem you are thinking about isn\u2019t novel – others have likely come across that problem before. If you\u2019re lucky, you\u2019ll be able to understand and use their solution directly. If not, you\u2019ll learn about someone else\u2019s experience and you can use that to inform your solution.<\/span><\/p>\n
Reading about a solution that didn\u2019t work is a lot quicker than inventing it and discovering that it doesn\u2019t work yourself.<\/span><\/i><\/p><\/blockquote>\n
The end\u2026?<\/b><\/h2>\n
If you got this far, thanks for reading this post! I hope some of the tips here will be useful.<\/span><\/p>\n
There\u2019s more to productive data science than effective use of your data warehouse, so watch this space for more hints and tips in future.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"
Whatever your role, you might come into contact with a data warehouse at some point. They’re incredibly powerful tools that are frequently ignored or used in ways that cause problems. This post contains tips for handling data in your data warehouse effectively.<\/p>\n","protected":false},"author":168,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"content-type":"","inline_featured_image":false,"footnotes":""},"categories":[5],"tags":[185,207],"location":[397],"class_list":["post-10030","post","type-post","status-publish","format-standard","hentry","category-our-thinking","tag-data","tag-data-warehouse"],"acf":[],"yoast_head":"\nData tips - the data warehouse | Equal Experts<\/title>\n<meta name=\"description\" content=\"Hints and tips for data scientists and other tech professionals on how to handle data in your data warehouse effectively.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.equalexperts.com\/blog\/our-thinking\/data-tips-the-data-warehouse\/\" \/>\n<meta property=\"og:locale\" content=\"en_GB\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Data tips - the data warehouse\" \/>\n<meta property=\"og:description\" content=\"Whatever your role, you might come into contact with a data warehouse at some point. They're incredibly powerful tools that are frequently ignored or used in ways that cause problems. This post contains tips for handling data in your data warehouse effectively.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.equalexperts.com\/blog\/our-thinking\/data-tips-the-data-warehouse\/\" \/>\n<meta property=\"og:site_name\" content=\"Equal Experts\" \/>\n<meta property=\"article:published_time\" content=\"2021-09-07T12:23:31+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-04-16T08:48:09+00:00\" \/>\n<meta name=\"author\" content=\"Paul Brabban\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:title\" content=\"Data tips - the data warehouse\" \/>\n<meta name=\"twitter:description\" content=\"Whatever your role, you might come into contact with a data warehouse at some point. They're incredibly powerful tools that are frequently ignored or used in ways that cause problems. This post contains tips for handling data in your data warehouse effectively.\" \/>\n<meta name=\"twitter:creator\" content=\"@EqualExperts\" \/>\n<meta name=\"twitter:site\" content=\"@EqualExperts\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Paul Brabban\" \/>\n\t<meta name=\"twitter:label2\" content=\"Estimated reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"14 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.equalexperts.com\/blog\/our-thinking\/data-tips-the-data-warehouse\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.equalexperts.com\/blog\/our-thinking\/data-tips-the-data-warehouse\/\"},\"author\":{\"name\":\"Paul Brabban\",\"@id\":\"https:\/\/www.equalexperts.com\/#\/schema\/person\/ad309d5a8484849a75e1bdd9fe56878c\"},\"headline\":\"Data tips – the data warehouse\",\"datePublished\":\"2021-09-07T12:23:31+00:00\",\"dateModified\":\"2024-04-16T08:48:09+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.equalexperts.com\/blog\/our-thinking\/data-tips-the-data-warehouse\/\"},\"wordCount\":3077,\"publisher\":{\"@id\":\"https:\/\/www.equalexperts.com\/#organization\"},\"keywords\":[\"data\",\"data warehouse\"],\"articleSection\":[\"Our Thinking\"],\"inLanguage\":\"en-GB\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.equalexperts.com\/blog\/our-thinking\/data-tips-the-data-warehouse\/\",\"url\":\"https:\/\/www.equalexperts.com\/blog\/our-thinking\/data-tips-the-data-warehouse\/\",\"name\":\"Data tips - the data warehouse | Equal Experts\",\"isPartOf\":{\"@id\":\"https:\/\/www.equalexperts.com\/#website\"},\"datePublished\":\"2021-09-07T12:23:31+00:00\",\"dateModified\":\"2024-04-16T08:48:09+00:00\",\"description\":\"Hints and tips for data scientists and other tech professionals on how to handle data in your data warehouse effectively.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.equalexperts.com\/blog\/our-thinking\/data-tips-the-data-warehouse\/#breadcrumb\"},\"inLanguage\":\"en-GB\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.equalexperts.com\/blog\/our-thinking\/data-tips-the-data-warehouse\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.equalexperts.com\/blog\/our-thinking\/data-tips-the-data-warehouse\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.equalexperts.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Data tips – the data warehouse\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.equalexperts.com\/#website\",\"url\":\"https:\/\/www.equalexperts.com\/\",\"name\":\"Equal Experts\",\"description\":\"Making Software. Better.\",\"publisher\":{\"@id\":\"https:\/\/www.equalexperts.com\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.equalexperts.com\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-GB\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.equalexperts.com\/#organization\",\"name\":\"Equal Experts\",\"url\":\"https:\/\/www.equalexperts.com\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\/\/www.equalexperts.com\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.equalexperts.com\/wp-content\/uploads\/2018\/08\/Equal_Experts_Logo_CMYK_Colour.jpg\",\"contentUrl\":\"https:\/\/www.equalexperts.com\/wp-content\/uploads\/2018\/08\/Equal_Experts_Logo_CMYK_Colour.jpg\",\"width\":719,\"height\":340,\"caption\":\"Equal Experts\"},\"image\":{\"@id\":\"https:\/\/www.equalexperts.com\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/x.com\/EqualExperts\",\"https:\/\/www.linkedin.com\/company\/equal-experts\/?viewAsMember=true\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.equalexperts.com\/#\/schema\/person\/ad309d5a8484849a75e1bdd9fe56878c\",\"name\":\"Paul Brabban\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\/\/www.equalexperts.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/a3520a00b97202664d60d70e71cb1aa5e1de19cd19a34f37d5622a973493db53?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/a3520a00b97202664d60d70e71cb1aa5e1de19cd19a34f37d5622a973493db53?s=96&d=mm&r=g\",\"caption\":\"Paul Brabban\"}}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Data tips - the data warehouse | Equal Experts","description":"Hints and tips for data scientists and other tech professionals on how to handle data in your data warehouse effectively.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.equalexperts.com\/blog\/our-thinking\/data-tips-the-data-warehouse\/","og_locale":"en_GB","og_type":"article","og_title":"Data tips - the data warehouse","og_description":"Whatever your role, you might come into contact with a data warehouse at some point. They're incredibly powerful tools that are frequently ignored or used in ways that cause problems. This post contains tips for handling data in your data warehouse effectively.","og_url":"https:\/\/www.equalexperts.com\/blog\/our-thinking\/data-tips-the-data-warehouse\/","og_site_name":"Equal Experts","article_published_time":"2021-09-07T12:23:31+00:00","article_modified_time":"2024-04-16T08:48:09+00:00","author":"Paul Brabban","twitter_card":"summary_large_image","twitter_title":"Data tips - the data warehouse","twitter_description":"Whatever your role, you might come into contact with a data warehouse at some point. They're incredibly powerful tools that are frequently ignored or used in ways that cause problems. This post contains tips for handling data in your data warehouse effectively.","twitter_creator":"@EqualExperts","twitter_site":"@EqualExperts","twitter_misc":{"Written by":"Paul Brabban","Estimated reading time":"14 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.equalexperts.com\/blog\/our-thinking\/data-tips-the-data-warehouse\/#article","isPartOf":{"@id":"https:\/\/www.equalexperts.com\/blog\/our-thinking\/data-tips-the-data-warehouse\/"},"author":{"name":"Paul Brabban","@id":"https:\/\/www.equalexperts.com\/#\/schema\/person\/ad309d5a8484849a75e1bdd9fe56878c"},"headline":"Data tips – the data warehouse","datePublished":"2021-09-07T12:23:31+00:00","dateModified":"2024-04-16T08:48:09+00:00","mainEntityOfPage":{"@id":"https:\/\/www.equalexperts.com\/blog\/our-thinking\/data-tips-the-data-warehouse\/"},"wordCount":3077,"publisher":{"@id":"https:\/\/www.equalexperts.com\/#organization"},"keywords":["data","data warehouse"],"articleSection":["Our Thinking"],"inLanguage":"en-GB"},{"@type":"WebPage","@id":"https:\/\/www.equalexperts.com\/blog\/our-thinking\/data-tips-the-data-warehouse\/","url":"https:\/\/www.equalexperts.com\/blog\/our-thinking\/data-tips-the-data-warehouse\/","name":"Data tips - the data warehouse | Equal Experts","isPartOf":{"@id":"https:\/\/www.equalexperts.com\/#website"},"datePublished":"2021-09-07T12:23:31+00:00","dateModified":"2024-04-16T08:48:09+00:00","description":"Hints and tips for data scientists and other tech professionals on how to handle data in your data warehouse effectively.","breadcrumb":{"@id":"https:\/\/www.equalexperts.com\/blog\/our-thinking\/data-tips-the-data-warehouse\/#breadcrumb"},"inLanguage":"en-GB","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.equalexperts.com\/blog\/our-thinking\/data-tips-the-data-warehouse\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.equalexperts.com\/blog\/our-thinking\/data-tips-the-data-warehouse\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.equalexperts.com\/"},{"@type":"ListItem","position":2,"name":"Data tips – the data warehouse"}]},{"@type":"WebSite","@id":"https:\/\/www.equalexperts.com\/#website","url":"https:\/\/www.equalexperts.com\/","name":"Equal Experts","description":"Making Software. Better.","publisher":{"@id":"https:\/\/www.equalexperts.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.equalexperts.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-GB"},{"@type":"Organization","@id":"https:\/\/www.equalexperts.com\/#organization","name":"Equal Experts","url":"https:\/\/www.equalexperts.com\/","logo":{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/www.equalexperts.com\/#\/schema\/logo\/image\/","url":"https:\/\/www.equalexperts.com\/wp-content\/uploads\/2018\/08\/Equal_Experts_Logo_CMYK_Colour.jpg","contentUrl":"https:\/\/www.equalexperts.com\/wp-content\/uploads\/2018\/08\/Equal_Experts_Logo_CMYK_Colour.jpg","width":719,"height":340,"caption":"Equal Experts"},"image":{"@id":"https:\/\/www.equalexperts.com\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/x.com\/EqualExperts","https:\/\/www.linkedin.com\/company\/equal-experts\/?viewAsMember=true"]},{"@type":"Person","@id":"https:\/\/www.equalexperts.com\/#\/schema\/person\/ad309d5a8484849a75e1bdd9fe56878c","name":"Paul Brabban","image":{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/www.equalexperts.com\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/a3520a00b97202664d60d70e71cb1aa5e1de19cd19a34f37d5622a973493db53?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/a3520a00b97202664d60d70e71cb1aa5e1de19cd19a34f37d5622a973493db53?s=96&d=mm&r=g","caption":"Paul Brabban"}}]}},"_links":{"self":[{"href":"https:\/\/www.equalexperts.com\/wp-json\/wp\/v2\/posts\/10030","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.equalexperts.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.equalexperts.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.equalexperts.com\/wp-json\/wp\/v2\/users\/168"}],"replies":[{"embeddable":true,"href":"https:\/\/www.equalexperts.com\/wp-json\/wp\/v2\/comments?post=10030"}],"version-history":[{"count":0,"href":"https:\/\/www.equalexperts.com\/wp-json\/wp\/v2\/posts\/10030\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.equalexperts.com\/wp-json\/wp\/v2\/media?parent=10030"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.equalexperts.com\/wp-json\/wp\/v2\/categories?post=10030"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.equalexperts.com\/wp-json\/wp\/v2\/tags?post=10030"},{"taxonomy":"location","embeddable":true,"href":"https:\/\/www.equalexperts.com\/wp-json\/wp\/v2\/location?post=10030"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}