{"id":11593,"date":"2022-11-01T09:18:06","date_gmt":"2022-11-01T09:18:06","guid":{"rendered":"https:\/\/ee.yelkdev.site\/?p=11593"},"modified":"2024-12-09T15:01:14","modified_gmt":"2024-12-09T15:01:14","slug":"chaos-days-common-mistakes","status":"publish","type":"post","link":"https:\/\/www.equalexperts.com\/blog\/tech-focus\/chaos-days-common-mistakes\/","title":{"rendered":"5 Common Mistakes to Avoid at your Chaos Day"},"content":{"rendered":"

The concept of Chaos Days is gaining popularity. For starters, it\u2019s always interesting to cause chaos and break things.<\/p>\n

But more importantly, Chaos Days provide an opportunity to improve system resilience by identifying potential weaknesses and understanding how to fix them.<\/p>\n

But Chaos Days always involve some risk, and as more teams start using chaos engineering, there are a few common mistakes that are worth discussing.<\/p>\n

In this blog post, we\u2019ll consider five things that can go wrong during a Chaos Day – and how to ensure you avoid all of these pitfalls. If you want to find out more about how to run a Chaos Day, don\u2019t miss our blog post<\/a> , and download our Chaos Days Playbook<\/a> for detailed guidance on planning and executing your next Chaos Day.<\/p>\n

Chaos Day Problem 1: Wrong Timing<\/h2>\n

You can\u2019t over-estimate the importance of having enough time to plan the timing of your Chaos Day.<\/p>\n

Trying to run a Chaos Day with anything less than a few weeks\u2019 notice means you\u2019re unlikely to have enough time to pull together the right team members. You need to be able to identify the right team members, ensure they are all available on the chosen date, AND have capacity to take part in Chaos Day planning.<\/p>\n

While your chaos team is planning your experiments, don\u2019t forget everyone else. Make sure all team members receive \u2018save the date\u2019 invitations and be sure to share required pre-reading well in advance of the Chaos Day.<\/p>\n

Chaos Day Problem 2: Wrong People<\/h2>\n

We recommend having the most experienced engineers from each team to act as agents of chaos, but your broader Chaos Day team should include other stakeholders. For example, it\u2019s important to have people from DevOps, engineering and the wider business community involved. In general, the CIO or CTO should be responsible for the Chaos Day, to ensure it will not compromise wider IT strategy. Finally, it\u2019s important\u00a0 to consider who, from the wider business, needs to be informed and responsible for changes made during Chaos Day.<\/p>\n

Chaos Day Problem 3: Lack of Planning<\/h2>\n

For a Chaos Day to be successful, it should in many respects be \u2018just another day\u2019 – otherwise it\u2019s difficult to create a realistic experience for the response team.<\/p>\n

However, many of your engineers will be unfamiliar with how chaos engineering works, and so it\u2019s helpful to ensure that people are aware of the possibility of a Chaos Day, and how it will operate within your organisation. Create and share a \u2018playbook\u2019 for Chaos Days and share it with your team well in advance of the \u2018live\u2019 Chaos Day. Encourage people to think about what experiments might happen, and how they could be approached.<\/p>\n

Planning should also consider what you want to achieve with your Chaos Days. For example, your objectives might include building cross-team knowledge and communication, building or testing business continuity processes, or building a business case for specific technology or process investment.<\/p>\n

Chaos Days represent a significant investment from the business in terms of time, resources and people. Given the costs involved, it\u2019s important to consider whether Chaos Days are providing a return on investment, and how that will be measured and reported.<\/p>\n

Chaos Day Problem 4: Running the wrong experiments<\/h2>\n

During the brainstorming process, it\u2019s easy to focus on experiments that allow the engineering team to test the biggest potential failures, or the most likely. But this won\u2019t always be the right approach.<\/p>\n

When designing and selecting experiments, consider which are the most business critical services, today, and how that will change in the coming months. If a system is or will soon become critical to overall business success, then it\u2019s important to test its resilience during a Chaos Day, even if a less critical system might be more likely to fail.<\/p>\n

Second, consider how big the experiments you design should be. Think of Chaos Days like a bomb blast – how wide should the radius be, to ensure you can recover quickly enough to avoid impacting the business?<\/p>\n

Chaos engineering is still a new concept for many teams, so it is better to start small with designed failures that are easy to roll back, and help build confidence and experience in your team. Measure the impact of these experiments before attempting more ambitious, large-scale experiments.<\/p>\n

Chaos Day Problem 5: Failing to Learn from Chaos Engineering<\/h2>\n

During the excitement of a Chaos Day, it\u2019s important to remember to monitor, capture and observe all the failures and team responses. For starters then, you need to plan carefully what tools and metrics will be used to monitor and measure the failure and its response. If a system response time increases, will you understand how much it increased, and also why it increased?<\/p>\n

Your Chaos Day doesn\u2019t end when the experiments end. You must be able to identify:<\/p>\n