Before a new car reaches the showroom on its way to your hands, the car maker crashes it on purpose. Engineers send it into barriers at speed, with instrumented dummies strapped inside, under conditions designed to resemble the real road.
Nobody wants a crash, but nobody trusts airbags, crumple zones and seatbelts on paper and untested. The test is how a manufacturer earns real confidence that the safety systems work when they are needed.
Vaccines work on a similar principle. A small, controlled exposure prompts the body to build defences, so that when the real threat arrives, the response is already trained. The dose is deliberately limited, and the aim is to strengthen, not to harm.
Both ideas apply to modern software systems. As more systems run as distributed services in the cloud, the case for deliberately and carefully introducing failure grows stronger.
The Shift from Monoliths to Cloud Complexity
A monolithic application has a limited number of ways to fail. A cloud architecture built from dozens or hundreds of services is fundamentally different. Each service depends on others, on networks, on databases, on caches and on third-party providers.
Any link can slow down, return errors or disappear—and the combinations multiply faster than any team can enumerate or test. Many failures are only visible when the whole system is running under real conditions. There are simply too many moving parts that shift shape as the situation changes.
This is where a practice like Chaos Engineering becomes invaluable.
What is Chaos Engineering?
Chaos engineering is the discipline of running controlled experiments on a system to build confidence that it can withstand turbulent conditions in production. It follows the scientific method:
1. Define a Steady State:
Identify a measurable signal indicative of a normally behaving system, such as successful requests or stream starts.
2. Formulate a Hypothesis:
Hypothesise that the steady state will hold even when something goes wrong.
3. Inject Realistic Faults:
Introduce controlled failure into the environment.
4. Compare & Analyse:
Compare the experimental group against a control group.
Key Takeaway: If the steady state holds, confidence in system resilience grows. If it doesn’t, the team uncovers a critical weakness before customers ever experience an outage.
⚠️ Two Essential Safeguards
- Reflect Real-World Events: Experiments should mirror actual risks, such as database latency or server failures.
- Limit the Blast Radius: Start small in scope and widen the experiment only as confidence builds to minimise potential impact.
How Netflix Made Chaos Engineering an Art Form
Netflix is where the discipline was elevated into an industry benchmark. After a major database corruption incident in 2008, the company spent several years migrating to Amazon Web Services (AWS) and rebuilding itself around microservices.
Losing instances became routine in the cloud, requiring services to recover on their own. To prove they could, engineers built Chaos Monkey, a tool that randomly terminated instances in production. Released as open source in 2012, it evolved into a family of failure-simulation tools (the “Simian Army”), capable of testing the loss of an entire AWS region.
Today, Netflix’s Chaos Automation Platform runs controlled experiments with baseline and experimental groups, injecting application-layer latency and errors while monitoring customer impact. If impact looks excessive, it halts automatically. Other organisations—like Uber and Facebook—have emulated this clearly beneficial practice.
Why Chaos Engineering Belongs in SRE
Despite the name, the practice is the opposite of reckless. Chaos engineering sits comfortably within Site Reliability Engineering (SRE), which treats reliability as something to be designed, measured and managed systematically:
Protecting Service Level Objectives (SLOs):
SREs define clear steady states for experiments to protect.
Spending Error Budgets Deliberately:
Controlled experiments spend a small, agreed fraction of an error budget to prevent massive, unplanned outages.
Validating Disaster Recovery Plans:
Incident response protocols are continuously verified against live environments rather than remaining untested documentation.
Under the guidance of SREs—with clear hypotheses, limited blast radius and automated safeguards—chaos engineering aims to reduce chaos, not create it.
Elevate Your Organisation’s Reliability
Chaos engineering is just one of several core practices SRE teams use to keep modern systems dependable—alongside SLOs, error budgets, incident response and the continual reduction of toil.
Interested in embedding these practices into your engineering culture? Explore our SRE Foundation and SRE Practitioner courses to learn how to apply reliability frameworks in your own environment.
Check out our IBF and SSG funded courses! There is no better time to upskill than now!
IBF Funding
IBF Funding
