{"id":8966,"date":"2021-06-10T09:24:31","date_gmt":"2021-06-10T08:24:31","guid":{"rendered":"https:\/\/ee.yelkdev.site\/?p=8966"},"modified":"2024-03-28T13:48:28","modified_gmt":"2024-03-28T13:48:28","slug":"six-essential-practices-of-data-pipelines","status":"publish","type":"post","link":"https:\/\/www.equalexperts.com\/blog\/our-thinking\/six-essential-practices-of-data-pipelines\/","title":{"rendered":"Six essential practices of data pipelines"},"content":{"rendered":"

A carefully managed data pipeline can provide you with seamless access to reliable and well-structured datasets. <\/span><\/p>\n

A generalised form of transferring data from a source system A to a source system B, data pipelines are developed in small pieces, and integrated with data, logic and algorithms to perform complex transformations. To do this effectively, there are some essential practices that need to be adhered to.<\/span><\/p>\n

In our <\/span>data pipeline playbook <\/span><\/a>we have identified eleven practices to follow when creating a data pipeline.\u00a0 Here we touch on six of these practices such as how to start by using a steel thread, and in our next blog post we will talk about iteratively creating your data models as well as observing the pipeline.\u00a0 Applying these practices will allow you to integrate new data sources faster at a higher quality as outlined in our recent post on the <\/span>benefits of a data pipeline<\/span><\/a>.<\/span><\/p>\n

About this series<\/b><\/h2>\n

This is part four in our six-part series on the data pipeline, taken from our<\/span> latest playbook<\/span><\/a>. First we looked at the basics, in <\/span>What is a data pipeline<\/span><\/a>. Next we looked at the <\/span>six main benefits of an effective data pipeline<\/span><\/a>. In part three we considered the \u2018must have\u2019 <\/span>key principles of data pipeline projects<\/span><\/a>. Now we look at the six key practices needed for a data pipeline. Before we get into the details we just want to cover off what\u2019s coming in the rest of the series. In part five<\/a> we look at more of those practices, and in part six<\/a> we look at the many pitfalls you can encounter in a data pipeline project.\u00a0<\/span><\/p>\n

The growing need for good data engineering<\/b><\/h2>\n

Today, data engineers serve a wider audience than just a few years ago. As there is a growing need for organisations to apply machine learning techniques to their data, new challenges are faced by data engineers in order to remain relevant. Essential to every project is the ability to reliably deliver large-volume data sets so that data scientists can train more accurate models.<\/span><\/p>\n

Aside from dealing with larger data volumes, these pipelines need to be flexible in order to accommodate the variety of data and the increasingly high processing velocity required. The following practices are those that we feel are essential to successful projects, the minimum requirement for success. They are based on our collective knowledge and experience gained across <\/span>many data pipeline engagements.\u00a0\u00a0<\/span><\/p>\n

Practice 1: Build for the right latency<\/b><\/h2>\n

When designing the pipeline, it\u2019s important to consider what level of latency you need. What is your speed of decision? How quickly do you need the data? Building and running a low latency, real-time data pipeline will be significantly more expensive, so make sure that you know you need one before embarking on that path. You should also ask how fast your pipeline can be. Is it even possible for you to have a real-time data pipeline? If all your data sources are produced by daily batch jobs, then the best latency you can reach will be daily updates, and the extra cost of real-time implementations will not provide any business benefits.<\/span><\/p>\n

If you do need to be within real-time or near real-time, then this needs to be a key factor at each step of the pipeline. The speed of the pipe is conditioned by the speed of the slowest stage.<\/span><\/p>\n

And be careful not to confuse the need for a real-time decision engine with the need for a real-time historical data store, such as a data warehouse for the data scientists. Decision models are created from stores of historical data and need to be validated before deployment into production. Model release usually takes place at a slower cadence (e.g., weekly or monthly). Of course, the deployed model will need to work on a live data stream, but we consider this part of the application development. This is not the appropriate use for a data warehouse or similar.<\/span><\/p>\n

Practice 2: Keep raw data<\/b><\/h2>\n

Ingestions should start by storing raw data in the pipeline without making any changes. In most environments, data storage is cheap, and it is common to have all the ingested data persisted and unchanged. Typically, this is done via cloud file storage (S3, GCP Cloud Storage, Azure Storage), or HDFS for on-premise data.<\/span><\/p>\n

Keeping this data allows you to reprocess it without re-ingestion if any business rule changes, and it also retains the possibility of new pipelines based on this data if, for example, a new dashboard is needed.<\/span><\/p>\n

Practice 3: Break transformations into small tasks<\/b><\/h2>\n

Pipelines are usually composed of several transformations of the data, activities such as format validation, conformance against master data, enrichment, imputation of missing values, etc. Data pipelines are no different from other software and should thus follow modern software development practices of breaking down software units into small reproducible tasks. Each task should target a single output and be deterministic and idempotent. If we run a transformation on the same data multiple times, the results should always be the same.<\/span><\/p>\n

By creating easily tested tasks, we increase the quality and confidence in the pipeline, as well as enhance the pipeline maintainability. If we need to add or change something on the transformation, we have the guarantee that if we rerun it, the only changes will be the ones we made.<\/span><\/p>\n

Practice 4: Support backfilling<\/b><\/h2>\n

If the pipelines are mature at the start of development, it may not be possible to fully evaluate whether the pipeline is working correctly or not. Is this metric unusual because this is what always happens on Mondays, or is it a fault in the pipeline? We may well find at a later date that some of the ingested data was incorrect. Imagine you find out that during a month, a source was reporting incorrect results, but for the rest of the time, the data was correct.<\/span><\/p>\n

We should engineer our pipelines so that we can correct them as our understanding of the dataflows matures. We should be able to backfill the stored data when we have identified a problem in the source or at some point in the pipeline, and ideally, it should be possible to backfill just for the corresponding period of time, leaving the data for other periods untouched.<\/span><\/p>\n

Practice 5: Start with a steel thread<\/b><\/h2>\n

When starting at a greenfield site, we typically build up data pipelines iteratively around a steel thread \u2013 first a thin data pipe which is a thin slice through the architecture. This progressively validates the quality and security of the data. The first thread creates an initial point of value – probably a single data source, with some limited processing, stored where it can be accessed by at least one data user. The purpose of this first thread is to provide an initial path to data and uncover unexpected blockers, so it is selected for simplicity rather than having the highest end-user value. Bear in mind that in the first iteration, you will need to:<\/span><\/p>\n