Chaos engineering

Chaos engineering is the practice of deliberately injecting failures into a running system under controlled conditions to test whether it survives them the way its architects believe it will. Rather than waiting for a real outage to reveal weak points, engineers cause small, bounded failures on their own schedule and observe the result. The discipline was popularised by Netflix's Chaos Monkey, a tool that randomly terminated production instances and thereby forced every service to tolerate the loss of any machine.

How chaos engineering works

Each experiment follows a scientific loop:

1. Define steady state — measurable indicators of normal behaviour, such as success rate or p99 latency;
2. Form a hypothesis — "if one database replica dies, checkout keeps serving 99.9% of requests";
3. Inject the fault — terminate instances, introduce network latency or packet loss, exhaust CPU or disk, block a dependency's endpoint;
4. Observe and conclude — compare metrics with steady state; any deviation is a discovered weakness to fix, and the experiment is re-run after the fix.

Safety discipline is what separates chaos engineering from recklessness: a limited blast radius (a fraction of traffic, one availability zone), an automatic abort switch when error budgets are threatened, and strong observability so impact is visible in real time. Common tooling includes Chaos Mesh and LitmusChaos for Kubernetes, Gremlin, and AWS Fault Injection Service.

Practical use cases

Teams use chaos experiments to validate the machinery of high availability: database failover that has never been exercised, load balancers draining unhealthy nodes, retry policies and circuit breakers that should prevent cascading failure, autoscaling under sudden load, and multi-region traffic shifting. "Game days" extend the practice to people — a simulated outage tests runbooks, paging chains and disaster-recovery plans as much as the technology.

The honest prerequisite check: if a system lacks monitoring, redundancy or tested backups, fix those first. Chaos engineering verifies resilience that is supposed to exist — it cannot conjure it.