{"id":6806,"date":"2020-03-05T10:33:12","date_gmt":"2020-03-05T10:33:12","guid":{"rendered":"https:\/\/ee.yelkdev.site\/?p=6806"},"modified":"2024-03-28T13:25:03","modified_gmt":"2024-03-28T13:25:03","slug":"equal-experts-engineer-chaos-at-john-lewis-partners","status":"publish","type":"post","link":"https:\/\/www.equalexperts.com\/blog\/our-thinking\/equal-experts-engineer-chaos-at-john-lewis-partners\/","title":{"rendered":"Equal Experts engineer chaos at John Lewis & Partners"},"content":{"rendered":"

The complexity of modern, distributed IT systems means they operate on the edge of failure. We help clients manage this risk by exploring the business impact of IT failure modes, and then investing time in improving the resilience of the people, products, and processes within the system. One way we do this is by running <\/span>Chaos Days<\/span><\/a>, in which we deliberately inject faults into a system and explore their impact.\u00a0<\/span><\/p>\n

Equal Experts have been working with John Lewis & Partners since 2018, to help them improve their ability to deliver and operate online retail services. This includes replacing existing eCommerce applications with cloud-based microservices running on the <\/span>John Lewis & Partners Digital Platform (JLDP)<\/span><\/a>. This bespoke platform-as-a-service enables product teams to rapidly build and run services, using Continuous Delivery and Operability practices, including <\/span>You Build It, You Run It<\/span><\/i>.\u00a0<\/span><\/p>\n

We recently partnered with John Lewis & Partners to run their inaugural Chaos Day. We were delighted to share our expertise. The John Lewis & Partners engineers gained valuable learnings about how revenue impacting failures could occur, and how best to mitigate them. They also learned the requisite skills and knowledge to run their own Chaos Days independently.\u00a0<\/span><\/p>\n

Organised chaos<\/strong><\/p>\n

John Lewis & Partners has 25 product teams building 40 services on their platform, which is built and run by two platform teams. This has created around 100 microservices, internal and external components, and downstream dependencies.\u00a0<\/span><\/p>\n

At Equal Experts, our experience of Chaos Days is to start small and then iterate. Rather than explore the impact of chaos on all teams and services, our first Chaos Day at John Lewis & Partners had two teams participating: one platform and one product team. A test environment was selected as the target for all experiments. To forewarn the organisation of any unintentional impact, and whet appetites for future Chaos Days, we preceded Chaos Day 1 with a presentation to John Lewis & Partners IT personnel on our approach.<\/span><\/p>\n

Experiment selection and design<\/strong><\/p>\n

Two weeks before Chaos Day, we met with the most experienced members of the platform team, who would become the <\/span>agents of chaos<\/span><\/i>. The people in this role design and execute the experiments, which the rest of their team detect and respond to, as if they were real production incidents. We intentionally asked for the most experienced platform engineers in order to test response skills liquidity within the team.<\/span><\/p>\n

We gathered around a whiteboard, and one of the engineers sketched out their platform architecture. The group then jointly brainstormed possible experiments, considering:<\/span><\/p>\n

    \n
  1. What failure mode we would learn about, e.g. loss of connectivity, an instance being terminated, network slowdown<\/span><\/li>\n
  2. What impact we would expect, e.g. impact on customer transactions<\/span><\/li>\n
  3. What response we would expect, e.g. the service auto-heals, an alert is fired, or nobody notices!<\/span><\/li>\n
  4. How normal service would be resumed, e.g. if the resolution is unsuccessful, how would the injected fault be rolled back?<\/span><\/li>\n
  5. If this experiment were to run in isolation, e.g. would a monitoring fault limit learning from other experiments?<\/span><\/li>\n<\/ol>\n

    The agents of chaos came up with 40 ideas, which we reviewed and whittled down to 10. We did this based on risk and impact to John Lewis & Partners, i.e. which failures were most likely to happen <\/span>and<\/span><\/i> damage the business. Two weeks were then spent developing and testing the code and configuration changes for fault injection ahead of the Chaos Day itself. The agents of chaos had their own private Slack channel to aid collaboration without revealing to other team members what havoc was soon to be unleashed.\u00a0<\/span><\/p>\n

    An example of something that was developed in this period was code to randomly terminate Google Kubernetes Engine (GKE) pods. The platform runs microservices in GKE pods, which don\u2019t have a guaranteed lifetime. The code developed was then used in a Chaos Day experiment to learn more about how microservices respond to pod termination.<\/span><\/p>\n

    Chaos execution<\/strong><\/p>\n

    On the day itself at John Lewis & Partners, the agents of chaos locked themselves away in a meeting room, with plenty of snacks. They worked their way through their Trello board of chaos experiments. We monitored each experiment closely, to analyse impact and team response.<\/span><\/p>\n

    In one experiment, we simulated a Google Cloud SQL failure by stopping a database instance without warning. The owning team received alerts from the platform and reacted promptly, but they took longer than expected to diagnose and resolve the failure. There were some useful learnings about the owning team\u2019s ability to resolve issues themselves without platform team assistance.<\/span><\/p>\n

    In another experiment, we reduced some outbound proxy timeout settings. We wanted to learn how well services handled failing requests to external dependencies. Some failures were detected by teams and responded to, but other failures went unnoticed. The agents of chaos had to provide a tip-off before the owning team became aware of a remaining problem. Some useful learnings on awareness, ownership and alerting came out of this.<\/span><\/p>\n

    Chaos reviews<\/strong><\/p>\n

    As John Allspaw puts it:<\/span><\/p>\n

    People are the adaptable element of complex systems<\/span><\/i><\/a><\/p><\/blockquote>\n

    After the Chaos Day, we held a review with all the people impacted by, and involved in, the chaos experiments. First, we established a joint timeline of events. Then we all shared what we saw, thought, did, and didn\u2019t do, in response to the chaos. During this process, various improvements were identified and added to the appropriate team\u2019s backlog. These included:<\/span><\/p>\n