(a) Termination Condition for Writes: Every write issued by a processor eventually completes (i.e., within a finite time) with respect to all processors.
(b) Return Value for Read Operations: A read operation by Pi returns a value that satisfies the following
conditions. Below, we assume the read and write operations are to the same address. If there is a write operation issued by Pithat has not yet completed with respect to Pibefore the read completes, then the value returned
by the read must be from the last such write that has been issued by Pi. Otherwise, the value returned by the
read must be from the latest write (from any processor) that has completed with respect to Pibefore the read
completes. If there are no writes that satisfy either of the above two categories, then the read must return the initial value of the location.
(c) Coherence Requirement: Writes to the same address complete in the same order with respect to every processor.
Conditions (a) and (c) above may be restricted to only a subset of the write operations.
The termination and coherence requirements are especially relevant to systems that replicate data through caching. The termination condition requires an issued write to eventually appear in the completion order with respect to every processor. Furthermore, our specification framework implicitly requires each issued write to appear at most once in the completion order with respect to a given processor. The above constraints imply that each issued write appears exactly once and only once in the completion order with respect to each processor. The coherence requirement further requires that writes to the same address complete in the same order with respect to every processor. Finally, the value condition constrains the value that a read must return. The remaining part of this section presents implementation techniques for satisfying the restrictions outlined by Condition 5.1 above. We begin by discussing implementation techniques for satisfying the value condition and coherence requirement. We next discuss the importance of supporting the coherence requirement and issues that arise if this condition is not supported for all writes. We finally describe techniques for satisfying the termination condition. The more subtle issues in correctly implementing a coherence protocol (e.g., transient cases, cross checking, maintaining point-to-point order, etc.) are described in Appendix K.
Supporting the Value Condition and Coherence Requirement
This section presents a few different techniques for supporting the value condition and coherence requirement. There are a number of subtle issues in a cache coherence protocol for correctly supporting the above two conditions that are covered in Appendix K.
The simplest way of supporting the coherence requirement is to serialize writes to the same location at the current owner. This serialization is used to ensure that writes to the same location complete in the same order with respect to every processor. In a typical invalidation-based protocol, the owner may be either the issuing processor’s cache hierarchy, the memory at the home node, or another processor’s cache; the read-exclusive or exclusive reply typically signals the issuing processor that its write is serialized. In a simple update-based protocol, the memory at the home node is the only possible owner; the update reply typically signals the serialization in this case.
One of the key aspects in supporting the value and coherence requirements is to correctly deal with outgoing and incoming requests at a processor that already has a pending write to that location. Consider an invalidation-based coherence protocol with a write buffer and a single level write-back cache per processor. We consider two different implementations below. In the first implementation, an exclusive or read-exclusive request is issued to the memory system and the pending write remains in the write buffer until the cache
obtains exclusive ownership for the line. Consider subsequent operations to this cache line by the issuing processor while the original write is pending. A subsequent write is simply queued in the write buffer; the write can be merged or coalesced with the pending write. Since an exclusive copy of the line has already been requested, no new requests are generated for the memory system. A subsequent read must return the value of the latest write to the same address from the write buffer if any such write exists. In case of a match, the system may support forwarding or may simply stall the read until the conflicting writes are retired from the write buffer. Requests from other processors destined to this cache are handled in the usual way. The value condition is satisfied because pending writes are only visible to the issuing processor; this is because the value of the write is not placed in the cache until after the write is serialized at the owner. Appendix K describes the message orderings between incoming invalidate requests and read or read-exclusive replies that may be important to maintain especially within the cache hierarchy.
Note that from the time the write miss request is issued by Pito the time the write is serialized with respect
to other writes to the same location, conflicting writes from other processors that are serialized and complete with respect Pi are actually not visible to Pi. Therefore, reads from Pimay never return the value of such
writes since they instead return the value of Pi’s outstanding write.
The second implementation differs from the implementation above by allowing the write to be retired and the write data to be written to the cache while the write is still pending (similar to the lockup-free implementation described in Section 5.2.3). This implementation is trickier since we have to ensure that the value of a pending write is visible to its own processor and yet disallow other processors from observing the value placed in the cache. Furthermore, the data placed in the cache must be appropriately merged with the reply data that returns in response to the write (in the case of a read-exclusive reply). Mechanisms for achieving the above functionality were discussed in the context of lockup-free caches in Section 5.2.3. For instance, while the line is pending, incoming read or read-exclusive requests are either delayed or turned back with a negative acknowledge reply. Similarly, incoming invalidate requests are serviced by changing the state of the line from pending clean to pending, but do not get rid of the values in the cache line corresponding to pending writes. Designs such as DASH [LLG+
90] use a hybrid scheme that is a merge of the two techniques discussed above. In DASH, a write updates the write through first level cache immediately (similar to the second technique),13but the update to the second level write back cache is delayed by the write buffer until
ownership is granted (as in the first technique).
Appendix M considers implementation issues for supporting the value condition and coherence require- ment in update-based designs.
Importance of Upholding the Coherence Requirement
While the system-centric models discussed in this thesis impose the coherence requirement on all writes, the programmer-centric models relax this condition by imposing it on only a subset of the write operations (i.e., competing writes for PL1 and PL2, and non-loop and non-sync writes for PL3). While it may seem beneficial to exploit the above relaxation, this section describes the several advantages of providing coherence for all writes in the context of hardware shared-memory implementations.
The primary reason for providing coherence is to simplify the implementation. Consider the sufficient
13Dealing with a write through cache is simpler than dealing with write back caches because it is alright to invalidate or replace the
A = 1; P1 Interconnection Network Processor Memory Memory A = 1 Processor Cache Memory Processor Cache Cache P2 A = 2; Home Node A: 0 1 2 A: 0 2 1 A: 0 1 2 A = 2 A = 1 A = 2
Figure 5.9: Updates without enforcing the coherence requirement.
conditions for satisfying the PL1 model shown in Figure 4.14 in the previous chapter, with the coherence requirement only imposed among competing writes. Nevertheless, it turns out that satisfying constraints such as the uniprocessor dependence condition (of the form W1 po
,!W2) or multiprocessor dependence chain (of
the form Wc co 0 ,! Rc
spo
,! W) is easier if we guarantee coherence among all writes. For example, for the
uniprocessor dependence condition, enforcing W1(i) xo
,! W2(i) for all i is much simpler if the coherence
requirement is upheld for the two writes. The same holds for the constraints imposed by the atomicity condition for read-modify-write operations (Condition 4.7). Finally, upholding the coherence requirement for all writes can be done quite efficiently in the context of hardware cache-coherent implementations. As we will see in Section 5.3.5, the simplest implementations for keeping track of outstanding invalidation or update requests inherently satisfy the coherence requirement for all writes by allowing only a single write to have outstanding coherence transactions.
The second reason for enforcing the coherence requirement for all writes is to provide a sensible semantics for programs that are not properly-labeled. Figure 5.9 illustrates an update-based coherence protocol which does not satisfy the coherence requirement for all writes. Assume location A is initialized to the value 0, P1 and P2 both maintain copies of this location, and the home for A resides on a third node. The example shows both P1 and P2 writing to location A. Each processor updates its own cache and sends update requests to other copies. If the coherence requirement is not upheld for the two writes, it is possible for the copy at P2 to transition from 0 to 2 to 1 while the copy at memory and at P1 transition from 0 to 1 to 2, allowing P1 and P2 to permanently observe different values for location A. The semantics is even more intractable if P2 replaces its cache copy, making the copy at memory visible to P2. In this scenario, the value of A visible to P2 transitions from 0 to 2 to 1 and back to 2, making it appear as if P2’s write occurred more than once. Without maintaining a complete copy of memory at each node, it is virtually impossible to alleviate the above behavior caused by replacements unless the coherence requirement is upheld for all writes.
Migrating a process to a different processor can present similar problems. Consider migrating the process at P2 to either P1 or the home node. Again, the process can observe the effect of its write more than once, with the value transitioning from 0 to 2 to 1 while the process is on P2 and then transitioning back to 2 after it is migrated. In fact, indefinitely migrating a process among different processors can lead to the process
observing the effect of a single write infinitely many times. Furthermore, even maintaining a full copy of memory does not solve the problems caused by migration. Therefore, supporting the coherence requirement on all writes is worthwhile unless designers are willing to limit an implementation to only executing properly- labeled programs. Even though this limitation is imposed by several software-supported shared-memory implementations (see Section 5.6), hardware implementations are often required to support a larger set of programming styles.
Supporting the Termination Condition
The termination condition requires a write that is issued to eventually complete with respect to all processors, thus ensuring all processors will eventually observe the effect of a given write.14 For some specifications
(e.g., RCsc, RCpc, and the sufficient conditions for PL1, PL2, and PL3), the termination condition is imposed on only a subset of the write operations (i.e., competing writes).15 A number of software-supported shared-memory implementations (see Section 5.6) benefit from this relaxation. However, most hardware cache-coherent implementations end up inherently ensuring the termination condition for all write operations. To satisfy the termination condition, an implementation must ensure that a write eventually affects all stale copies in the system. A designer must carefully consider places in a design where the servicing of a write is delayed either due to queueing or due to arbitration (e.g., if reads have priority over writes) to ensure that the write operations will eventually get serviced. Furthermore, the implementation cannot depend on the arrival of new operations to trigger the completion of previous write operations. For example, a merging write buffer that delays write operations in order to increase the window of opportunity for merging may need to periodically flush the write operations in the buffer. Similarly, incoming buffers within a cache hierarchy must ensure queued invalidate and update requests will eventually get serviced.
Appendix H describes a more aggressive form of the termination condition for the PL models, along with its implementation implications.
5.3.3
Supporting the Initiation and Uniprocessor Dependence Conditions
The initiation and uniprocessor dependence conditions interact closely to maintain the intuitive notion of uniprocessor data dependence; i.e., to ensure that the “effect” of a processor’s conflicting read and write operations are consistent with program order. A simple way to ensure this is to require conflicting operations from the same processor to complete in program order. Thus, given conflicting operations from Pi, R
po ,! W
implies Rinit(i) xo ,! W
init(i) and R(i) xo ,! W(i), W1 po ,! W2 implies W1 init(i) xo ,! W2
init(i) and
W1(j) xo ,! W2(j) for all j, 16and W po ,! R implies W init(i) xo ,! R
init(i) and W(i) xo
,! R(i). Note that the
initiation and uniprocessor dependence conditions do not impose any ordering constraints on two reads to the same location.
Except for the IBM-370 model, the other models discussed in this thesis relax the above constraints in the case of W po
,! R by only requiring W init(i)
xo ,! R
init(i) (and not W(i) xo
,! R(i)) to hold. This relaxation
14The termination condition implicitly depends on the uniprocessor correctness condition (Condition 4.1 in Chapter 4) which requires
every write in the program to be issued (i.e., the Winit(i) sub-operation for every write must appear in the execution order).
15For some asynchronous algorithms, enforcing that all writes will eventually become visible to all processors may be important for
convergence. This can be guaranteed by periodically flushing any writes whose completion has been delayed by the system.
16In an implementation that maintains the coherence requirement for all writes, it is sufficient to ensure W1(i) xo
,! W2(i) since the
coherence requirement will then ensure W1(j) xo
allows a processor to read the value of its own write before the write completes with respect to it. Even though it may seem that allowing a read R that is after a conflicting write W in program order to complete before the write may cause a possible violation of uniprocessor data dependence, the initiation condition still requires Winit(i)
xo ,!R
init(i) and along with the value condition, the read is guaranteed to return the value
of W (or a later conflicting write that is between W and R in program order). Therefore, the read still sees the effect of the last conflicting write that is before it in program order.
Techniques for satisfying the initiation and uniprocessor dependence conditions are well understood since similar conditions must be maintained in uniprocessors. A conservative way to satisfy the conditions is to delay issuing an operation until the previous conflicting operation in program order completes. Delaying an operation does not necessarily require the processor to be stalled; read and write buffers may be used for this purpose. Below, we discuss the implementation techniques for supporting the required orders in more detail. Given a conflicting read followed by a write, the system must disallow the read to return the value of the write by ensuring the read completes before the write. Processors with blocking reads inherently satisfy the above condition. A more efficient technique may be to delay the completion of the write without stalling the processor. For example, the write may be placed in the write buffer, or may be retired into a lockup-free cache, but the sending of the exclusive or read-exclusive request may be delayed until the read completes.17
In some protocols, it may be possible to issue the exclusive or read-exclusive request for the line before the read completes; this typically requires a guarantee that the read will complete without retries and that point-to-point order is maintained between the two requests. Such optimizations are more difficult to support in protocols that can force the read to retry (e.g., through a negative acknowledgement reply). For example, if the write is delayed in a write buffer, it is important for a read that is being retried to unconditionally bypass the write buffer (i.e., not forward the value of the write). A similar precaution applies to implementations that do an early retire of the write into the cache. Retrying the read from a lower level in the hierarchy can eliminate some of these issues. Finally, the presence of retries makes it difficult to correctly implement the optimization of issuing the exclusive request before the read completes since point-to-point order no longer inherently translates to the correct completion order.18
For two conflicting writes in program order, the simple approach is to delay the servicing of the second write until the first write completes with respect to the issuing processor. Again, this does not necessarily imply a processor stall since the second write may be delayed in a buffer or may be retired to the cache and merged when the reply for the first write arrives (as discussed in Section 5.2.3). Retiring the write to the cache is more complex for update-based schemes or write through invalidation-based schemes since the second write generates a new request to the memory system which may get out of order with respect to the first write’s request either if point-to-point order is not maintained or if the protocol can force the first write to be retried. For protocols that enforce the coherence requirement for all writes, ensuring that the writes complete in program order with respect to the issuing processor automatically ensures that the writes complete in program order with respect to other processors. As discussed in Section 5.3.2, this is one of the ways in which enforcing the coherence requirement on all writes simplifies the implementation.
Finally, in the case of a write followed by a conflicting read in program order, only the IBM-370 model
17In protocols that do an early retiring of the write values into the cache, an exclusive or read-exclusive request caused by any write
that lies in the same cache line as the read (i.e., even if the write and read do not conflict) may also need to be delayed until the read