System Resiliency Patterns and Chaos Engineering

2 posts · Nov 2018 – Mar 2019

What this trail covers

Grab has evolved its system resiliency by implementing chaos engineering practices. This involves intentionally introducing failures into services and components to test the resilience of dependent systems. The Experimentation Platform (ExP) is leveraged to orchestrate application-level chaos, injecting failures like latencies and memory leaks. A dedicated SDK and UI have been developed to simplify the creation and execution of chaos experiments, allowing engineers to define ideal system behavior.

All Grab trails

Mar 2019Designing Resilient Systems Beyond Retries (Part 3): Architecture Patterns and Chaos EngineeringThis post, the third in a series, focuses on architecture patterns (idempotency, asynchronous responses) and chaos engineering to improve system resiliency. It details the process of conducting chaos experiments, including defining steady states, using control and experiment groups, hypothesizing outcomes, introducing failures, and analyzing results. The importance of automation and minimizing the blast radius of experiments is emphasized. The post also highlights how idempotency keys and asynchronous responses can complement retry strategies.
Nov 2018Orchestrating Chaos Using Grab's Experimentation PlatformThis post details the implementation of chaos engineering at Grab by integrating it with the existing Experimentation Platform (ExP) and Grab-Kit. It introduces a Chaos SDK built on top of the ExP SDK, allowing for application-level chaos injection (errors, CPU load, memory leaks, latency, disk space, goroutine leaks, panics, throttling) directly within middleware. A dedicated UI is provided for creating and managing chaos experiments, defining ideal system behavior, control/treatment groups, and failure configurations. The post emphasizes the benefits of controlled experimentation for verifying system resilience and identifying potential issues before they cause outages.
Advertisement