Last Friday (September 13th) Kasper Nissen and I visited Amazon in London for a Chaos Community Day. The day was hosted by Russ Miles from ChaosIQ and the agenda included speakers from several different industries; banking, streaming services, digital and print publishing to just name a few. Our expectations were quite high, and it’s safe to say we weren’t disappointed. The talks were high quality and they gave us very different perspectives on the routes and journeys people in the community are taking.
The day was kicked-off with an introduction by Russ where he talked about the purpose of a Community Day and what kind of community he would like the Chaos Community to be:
This came through in the talks, but also in the discussions in-between. The atmosphere was friendly and inclusive and there was respect for the many different approaches to Chaos Engineering.
The question is not if failures will happen — sooner or later failures will happen. In Production! It’s better to plan “Chaos” ahead of time and find windows where the impact is small and making sure the right people are observing and ready to learn and remediate.
"Chaos Engineering offers a dialogue with your system" and @lawouach follows up with these wise words: "By practicing Chaos Engineering, the engineer becomes more familiar with messages brought by the system" Good point! #chaosdayeu #chaosengineering pic.twitter.com/0BYYPzzEs9— 𝚔𝚊𝚜𝚙𝚎𝚛 𝚗𝚒𝚜𝚜𝚎𝚗 🏡 (@phennex) September 13, 2019
@Charity Majors said it well in her talk: “Chaos” is a fancy marketing term for running tests later in the software development lifecycle. Indeed, exploring and becoming aware of the known and unknown is something that has been practiced in decades by exploratory testers. The new thing is that we shift right and experiment directly in Production. To build confidence in a system’s capability to withstand turbulent and unexpected conditions. Some unexpected conditions could be:
Making such experiments in Production may sound like a terrible idea. In practice, it’s a very good approach to be able to build confidence. Test and Staging environments are often not provisioned and configured exactly like Production, and the traffic volumes and patterns are different. You can work on simulating Production in another environment, but this can quickly become a time sink for engineering time.
Chaos Engineering is by no means a new concept — it’s just new in the software industry where traditional approaches to managing failure in Production have been mainly reactive.
Some examples are:
These are all reactive and provide ways to remediate unexpected conditions after they have already happened.
Chaos Engineering is inherently proactive; learning about system weaknesses before they manifest, giving us a chance to remediate before disaster strikes.
Father of #chaosengineering @caseyrosenthal, as @russmiles introduced him, tells the story of why Netflix built #chaosmonkey 10 years ago! Back then the stability of @awscloud wasn't that good! Kind of fun to hear given we are in the aws office 😆 pic.twitter.com/27q7pE6HmG— 𝚔𝚊𝚜𝚙𝚎𝚛 𝚗𝚒𝚜𝚜𝚎𝚗 🏡 (@phennex) September 13, 2019
One of the early practitioners of Chaos Engineering was Netflix, who created Chaos Monkey to weed out availability issues proactively. I heard about Chaos Monkey in 2014 when attending the talk “I Don’t Test Often … But When I Do, I Test in Production” at GTAC (Google’s Test Automation Conference). This was the first time I encountered a real-world application of Chaos Engineering at scale.
Times are changing, and today zero-downtime-deploys, self-healing infrastructure, and resilient systems are rapidly becoming the norm. Today’s customers expect perfect software and 24/7/365 availability; this applies especially to businesses (like ours) that have traditionally relied on physical infrastructure and humans but are becoming completely digital. As one of the speakers (coming from a well-known streaming service) put it:
Readiness to handle failure (or the unknown) is feature zero.
It’s impressive how Chaos Engineering has evolved; today a wealth of companies in very different industries are practicing every day. Their journeys are quite different, but there does seem to be some common ground:
The Game Day concept was coined by Jesse Robbins, an ex-Amazon engineer who used his experience and training as a volunteer firefighter and from other industries and studies. Game Days have proved to be very successful and have been adopted by many companies and are today considered a best practice.
So is Chaos Engineering something we want to do at Lunar Way?
Absolutely! Like most companies on a similar journey, we have started experimenting in our test environments. This will give us the confidence to proceed further and maybe one day run experiments in Production. We’ll be sharing our experiences along the way.
Thanks to Russ for inviting us, Amazon for hosting, and the community for being awesome. We had a blast and will be very happy to join again next year.
Lunar Way is a fintech company motivated by rethinking the experience of banking, and the way people perceive money and spending in general. That is why we are using the most innovative and smart technology in order to create the banking solution for tomorrow directly in our app.
Read more on lunar.app