We will discuss the second kind of scenarios in more depth in the next chapter when presenting the Active–Active Replication Pattern.
12.3
The Let-It-Crash Pattern
“Prefer a full component restart to internal failure handling.”
In chapter 7 we discussed “principled failure handling,” noting that the internal recovery mechanisms of each component are limited because they are not sufficiently separated from the failing parts—everything within a component can be affected by a failure. This is especially clear for hardware failures that take down the component as a whole, but it is also true for corrupted state that is the result of some programming error that is only observable in rare circumstances. For this reason it is necessary to delegate failure handling to a supervisor instead of attempting to solve it within the component itself.
This approach is also called crash-only software.5 The idea is that transient but rare
failures are often very costly to diagnose and fix, making it preferable to recover a working system by rebooting parts of it. This way of hierarchical restart-based failure handling allows to greatly simplify the failure model and at the same time leads to a more robust system that even has a chance to survive failures that were entirely unforeseen.
12.3.1 The Problem Setting
We will demonstrate this design philosophy on the example of the worker nodes that perform the bulk of the work in the batch service whose component hierarchy we developed in the previous two patterns. Each of these is presumably deployed on its own hardware—virtualized or not—that it does not share with other components; ide- ally there is no common failure mode between different worker nodes other than a computing center outage.
The problem that we are trying to solve is that the workers’ code may contain pro- gramming errors that rarely manifest, but when they do they will impede the ability to process batch jobs. Examples of this kind are very slow resource leaks that can go undetected for a long time but will eventually kill the machine; this could be open files, retained memory, background threads, etc. and the leak might not occur every time but could be caused by a rare coincidence of circumstances. Another example is a security vulnerability that allows the executed batch job to intentionally corrupt the state of the worker node in order to subvert its function and perform unauthorized actions within the service’s private network—such subversion often is not completely invisible and leads to spurious failures that should better not be papered over.
5 Both of the following articles are by George Candea and Armando Fox: “Recursive Restartability: Turning the
Reboot Sledgehammer into a Scalpel,” USENIX HotOS VIII, 2001 (http://dslab.epfl.ch/pubs/ recursive_restartability.pdf) and “Crash-Only Software,” USENIX HotOS IX, 2003 (https://www.usenix.org/ legacy/events/hotos03/tech/full_papers/candea/candea.pdf)
THE TASK
Your mission is to consider the components we have identified for the batch service and describe how a crash and restart would affect each of them and which implemen- tation constraints arise from the let-it-crash philosophy.
12.3.2 Applying the Pattern
The Let-It-Crash Pattern by itself is simple: whenever a component—for example a worker node—is detected to be faulty, no attempts are made to repair the damage. Instead of doctoring with its internal state we restart it completely, releasing all its resources and starting up again from scratch. If we obtained the worker nodes by ask- ing an infrastructure service to provision them then we can go back to the most basic state imaginable: we decommission the old worker node and provision an entirely new one. This way no corruption or accumulated failure condition can have survived in the fresh instance since we start out from a known good state again.
Applying this approach to the Client Interface nodes means that all currently active client connections will be severed for the failed node, leading to connection abort errors in the clients. Upon detecting such an abort condition the client should try to reconnect, which is our first conclusion. The second follows immediately when considering that the new connection should not be routed to the failed node: this usually means changing the load balancer configuration to remove the failed node. Then a new node needs to be brought online and added to the load balancer to restore the same processing capacity as before. With these measures we can confi- dently crash and restart a Client Interface node at any given point in time. We do not need to consider the internal communication because no other components depend on this one: the Client Interface only has dependencies and no dependents—the con- sequence of this is that any new client requests that a fresh node receives will just refer to the Storage or Scheduling components as the sources of truth; the Client Interface can be “stateless.”6
For the Storage component, a node failure means that the stored data are invalid or lost—the consequence of either possibility is that the data cannot be relied upon any longer, so these states are fundamentally equivalent. Since the purpose of the component is to store data permanently, we will have to distribute them as per the dis- cussion in section 2.3. We will cover data replication in the next chapter; for now it suffices to assume that there will be other storage nodes that hold copies of the data. After stopping the failed node we will therefore need to start a new one that synchro- nizes itself with the other replicas, taking on the share of the responsibility that the failed node had. If the new node uses the previous node’s permanent storage device then recovery can be sped up by synchronizing only those updates that occurred after
6 This word has become so widely (mis)used that it does not stand on its own any longer; the author’s view is
that a truly stateless service which does not contain any mutable internal state does not exist (it would not be a component with a purpose to exist) and a more meaningful interpretation is to equate statelessness with the absence of persistent mutable state.
35