![\"\"](\"https:\/\/www.equalexperts.com\/wp-content\/uploads\/2021\/04\/datapipeline_blog1_content-1200x736.png\")
<\/span><\/p>\nAbout this series<\/b><\/h2>\n
This is part one in our six part series on the data pipeline, taken from our latest playbook. Here we look at the very basics – what is a data pipeline and who is it used by? Before we get into the details, we just want to cover off what\u2019s coming in the rest of the series. In part two<\/a>, we look at the six main benefits of a good data pipeline, part three<\/a> considers the \u2018must have\u2019 key principles of data pipeline projects, and parts four<\/a> and five<\/a> cover the essential practices of a data pipeline. Finally, in part six<\/a> we look at the many pitfalls you can encounter in a data pipeline project.<\/p>\n There is a lot of untapped value in the data that your organisation holds. Data that is critical if you take data analysis seriously. Put to good use, data can identify valuable business insights on your customers and your operations. However, to find these insights, the data has to be regularly, or even continuously, transported from the place where it is generated to a place where it can be analysed.<\/p>\n A data pipeline, consolidates data from all your disparate sources into one (or multiple) destinations, to enable quick data analysis. It also ensures consistent data quality, which is absolutely crucial for reliable business insights.<\/p>\n A data pipeline is a set of actions that ingest raw data from disparate sources and move the data to a destination for storage and analysis. We like to think of this transportation as a pipeline because data goes in at one end and comes out at another location (or several others). The volume and speed of the data are limited by the type of pipe you are using and pipes can leak – meaning you can lose data if you don’t take care of them.<\/p>\n The data engineers who create a pipeline are a critical service for any organisation. They create the architectures that allow the data to flow to the data scientists and business intelligence teams, who generate the insight that leads to business value.<\/p>\n A data pipeline is created for data analytics purposes and has:<\/b><\/p>\n Data sources<\/b> – These can be internal or external and may be structured (e.g., the result of a database call), semi-structured (e.g., a CSV file or a Google Sheets file), or unstructured (e.g., text documents or images).<\/p>\n Ingestion process<\/b> – <\/span>This is the means by which data is moved from the source into the pipeline (e.g., API call, secure file transfer).<\/p>\n Transformations<\/b> – <\/span>In most cases, data needs to be transformed from the input format of the raw data, to the one in which it is stored. There may be several transformations in a pipeline.<\/p>\n Data quality\/cleansing<\/b> – <\/span>Data is checked for quality at various points in the pipeline. Data quality will typically include at least validation of data types and format, as well as conforming with the master data.<\/p>\n Enrichment<\/b> –<\/span> Data items may be enriched by adding additional fields, such as reference data.<\/p>\n Storage<\/b> – <\/span>Data is stored at various points in the pipeline, usually at least the landing zone and a structured store (such as a data warehouse).<\/p>\n End users<\/b> – <\/span>more information on this is in the next section.<\/span><\/p>\n We believe that, as in any software development project, a pipeline will only be successful if you understand the needs of the users.<\/p>\n Not everyone uses data in the same way. For a data pipeline, the users are typically:<\/p>\n Business intelligence\/management information analysts<\/strong>, who need data to create reports;<\/p>\n Data scientists<\/strong> who need data to do an in-depth analysis of point problems or create algorithms for key business processes (we use ‘data scientist’ in the broadest sense, including credit risk analysts, website analytics experts, etc.)<\/p>\n Process owners<\/strong>, who need to monitor how their processes are performing and troubleshoot when there are problems.<\/p>\n Data users are skilled at visualising and telling stories with data, identifying patterns, or understanding significance in data. Often they have strong statistical or mathematical backgrounds. And, in most cases, they are accustomed to having data provided in a structured form – ideally denormalised – so that it is easy to understand the meaning of an individual row of data without the need to query separate tables or databases.<\/p>\n Every organisation would benefit from a place where they can collect and analyse data from different parts of the business. Historically, this has often been met by a data platform, a centralised data store where useful data is collected and made available to approved people.<\/p>\n But, whether they like it or not, most organisations are, in fact, a dynamic mesh of data connections which need to be continually maintained and updated. Following a single platform pattern often leads to a central data engineering team tasked with implementing data flows.<\/p>\n The complexities of meeting everyone’s needs and ensuring appropriate information governance, as well as a lack of self-service, often make it hard to ingest new data sources. This can then lead to backlog buildup, frustrated data users, and frustrated data engineers.<\/p>\n Thinking of these dataflows as a pipeline changes the mindset away from monolithic solutions, to a more decentralised way of thinking – understanding what pipes and data stores you need and implementing them the right way for that case whilst reusing where appropriate.<\/p>\n So now we have understood a little more about the data pipeline, what it is and how it works, we can start to understand the benefits and assess whether they align with your digital strategy.\u00a0 We cover these in the next blog article, \u2018What are the benefits of data pipelines?\u2019<\/p>\n For more information on the data pipeline in general, take a look at our Data Pipeline Playbook.<\/a>\u00a0 And if you\u2019d like us to share our experience of the data pipeline with you, get in touch using the form below.<\/p>\n <\/p>\n\n\t\t\t\t\t\tWhy is a data pipeline critical to your organisation?<\/b><\/h2>\n
So what is a data pipeline?<\/b><\/h2>\n
So, who uses a data pipeline?<\/b><\/h2>\n
Is a data pipeline a platform?<\/b><\/h2>\n