INSALUD Subdirección General de Coordinación

We now investigate the effect of discontinued communication (e.g., due to a co- ordinator failure) in Multi-Ring Paxos. In this experiment we deploy two rings and a learner that subscribes to these rings. Each ring generates messages with the same constant rate of approximately 4000 messages per second in average. In steady state, the learner receives and delivers approximately 500 Mbps of data (see Figure 5.11). After 20 seconds we stop the coordinator of ring 1, bringing the receiving throughout from this ring at the learner to zero. Although mes- sages still arrive at the learner from ring 2, the learner buffers such messages as it cannot execute its deterministic merge procedure. The result is that the deliv- ery throughput at the learner drops to zero (right-most graph of Figure 5.11). Notice that after ring 1 stops, the incoming throughput from ring 2 decreases, as the learner does not acknowledge the delivery of messages from group 2 to the node that multicasts to ring 2 and this one slows down its sending rate.

Three seconds later the execution at ring 1 proceeds. We forced a restart after three seconds to emphasize the effects of the discontinuity of traffic. In reality, it takes much less time to detect the failure of a coordinator and replace it with an operational acceptor. When the coordinator of the first ring starts, it notices that no consensus instances were decided in the last intervals and proposes to skip multiple consensus instances. As a result, the learner delivers all messages it has enqueued, momentarily leading to a high peak in the delivery throughput. Then the execution proceeds as normal.

5.4.8

Conclusions from the experiments

The following conclusions can be drawn from our experiments:

• Performance of Multi-Ring Paxos scales as new rings are added to the en- semble. Therefore by using Multi-Ring Paxos the performance of the repli- cated service will always be determined by the number of requests the service is capable of processing rather than the number of requests the ordering layer can order.

• Although Multi-Ring Paxos uses a few configuration parameters to coor- dinate the rings, assigning proper values to these parameters does not impose extra processing costs on the system.

• In the presence of failures, given that the failed rings can be replaced in a reasonable time, learners are not subject to buffer overflow that could otherwise paralyze their delivery.

5.5

Conclusion

In this chapter we revisited the scalability of atomic broadcast protocols and pro- posed the Multi-Ring Paxos protocol that implements atomic multicast. While atomic broadcast induces a total order on the delivery of messages, atomic mul- ticast induces a partial order. Differently from previous atomic multicast algo- rithms, Multi-Ring Paxos exploits the abstraction of groups in a different way: in Multi-Ring Paxos, messages are addressed to a single group only, but pro- cesses can subscribe to multiple groups. In all atomic multicast algorithms we are aware of, messages are multicast to one or more groups, and often groups cannot intersect.

The results of our experiments are promising: by composing eight instances of In-memory Ring Paxos, for example, we can reach an aggregated throughput of more than 5 Gbps, eight times the throughput of a single Ring Paxos instance. Recoverable Ring Paxos has similar scalability, linear in the number of Ring Paxos instances.

Chapter 6

Replicating Parallel Applications with

State-Machine Replication

The advent of multi-core processors and their wide availability has revo- lutionized systems programming and application development strategies. To increase their capacity in serving clients and to benefit from the new hardware, service providers have to parallelize their services. Similarly to their sequen- tial predecessors, however, parallel services must also be continually available to the clients, despite failures. State-machine replication is a well-established strategy to make services fault tolerant. In this chapter, we will look at the capability of the state-machine approach in replicating parallel services. On the one hand, parallel applications require concurrent processing of requests to provide high performance and on the other hand, state-machine replication requires sequential processing of requests to preserve consistency. We will con- sider the possibility of uniting these two apparently incompatible models and will propose a scalable solution to achieve it.

6.1

Problem statement

Replicas in state-machine replication execute an ordered sequence of client re- quests sequentially and deterministically. Sequential execution of requests is an important means for ensuring strong consistency promised by state-machine replication. Multithreaded applications on the other hand, allow multiple threads to concurrently process client requests. Concurrent execution of requests is im- portant from a performance perspective, in particular when the servers have ac- cess to multi-core processors. Replicating parallel applications by state-machine replication is not straightforward due to their incompatible execution models.

The problem under study in this chapter is to modify state-machine replication to make its integration with parallel services a reality, while preserving the high performance of multithreaded applications and the strong consistency guaran- tees of state-machine replication.

It has been observed earlier that replicas in state-machine replication can relax the order among the independent commands and execute them concur- rently [68]. Two commands are independent if they access different variables or they only read the values of the common variables. Two commands are de-

pendent if they access at least one common variable, v, and at least one of the commands modifies the value of v. As an example, consider a service composed of three objects x, y, and z and assume commands C_x, C_y, C_z, C_{x y}, where the in- dices indicate the objects accessed and modified by the commands. Commands

C_x, Cy, and Czaccess disjoint objects. Thus, they are independent and can be ex-

ecuted in parallel at each replica. Command C_{x y} depends on commands C_x and

C_y and must be serialized with C_x and C_y. C_{x y} can be executed in parallel with

C_z, however. Several techniques have built on the command interdependencies to introduce parallelism in the execution of commands on replicas [69; 70]. In the next sections, we review these techniques and by identifying their main shortcomings we propose a novel and high performance approach to implement- ing parallelism in state-machine replication.

6.1.1

Outline

The rest of this chapter is organized as follows: In Section 6.2 we review parallel approaches to state-machine replication. In Section 6.3 we present our model, P- SMR, and discuss its algorithmic details. In Section 6.5 we empirically compare several techniques used in implementing parallel services. Finally in Section 6.6 we conclude this chapter.