load A, r1
add r2, ...
load B, r2
load A, r1
add r1, ...
load A,r1 load B,r2 Execute Fetch Instruction Lookahead Window add r2,... add r1,...(a)
(b)
(c)
add r2, ...
Figure 5.3: Example of out-of-order instruction issue.
of this type of overlap. The program segment in Figure 5.3(a) shows two load instructions each followed immediately by the use of the return value. Assume both load instructions are long latency cache misses. An in-order issue processor would issue the first load, stall at the use until the return value is back, issue the add, and then issue the second load. Even with non-blocking reads, no overlap is achieved between the two read operations. Figure 5.3(b) shows an abstract view of an out-of-order issue processor, modeled as a lookahead window of instructions where the processor can issue any ready instructions from the window. The processor starts by issuing the load to A. The add instruction that uses the return value of the load is not ready to be issued. However, the processor can proceed to issue the load to B while the load of A is outstanding. The add instructions are issued later when the relevant read data is returned to the processor. The effective overlap that is achieved is illustrated in Figure 5.3(c). The same type of overlap would be possible with an in-order issue processor (with non-blocking reads) if the original code is scheduled as in Figure 5.3(c), again illustrating that in-order issue processors are more sensitive to the initial instruction schedule.
To be effective in hiding large latencies, dynamic scheduling must be complemented with techniques such as register renaming [Kel75], dynamic branch prediction [LS84], and speculative execution. Register renaming alleviates write-after-read (WAR) and write-after-write (WAW) dependences that would otherwise delay the execution of future instructions. Dynamic branch prediction increases the lookahead capability of the processor while speculative execution allows the processor to execute instructions past unresolved branches.2 Appendix L provides a few examples of how speculative execution can help in exploiting a relaxed model.
The following sections describe how the various components in the memory system can exploit the overlap among memory operations that is exposed by the processor.
5.2.2
Read and Write Buffers
Memory operations are typically buffered after they are issued by the processor. The main purpose for buffering is to avoid processor stalls when the memory system cannot accept operations as fast as the processor issues them. This section briefly discusses buffering optimizations such as forwarding, bypassing, merging, and out-of-order (non-FIFO) service, that are enabled by relaxed models. Section 5.3.5 describes
how buffers can also play an important role in enforcing the appropriate order among memory operations as dictated by a memory model.
The processor issues two types of memory operations: a read operation along with an address, and a write operation along with an address and relevant data. In-order issue processors issue memory operations to the buffer in program order with valid address and data. Out-of-order issue processors may issue operations out of program order; however, some designs may issue place-holders for unready memory operations with unresolved addresses or values in order to allow the first level of buffering to keep track of program order among the operations. In the discussion below, we will assume two logical buffers, one for reads and another for writes. In an actual implementation, the two logical buffers may be physically merged or may be implemented as several separate buffers.
Consider the logical write buffer first. Assume write operations are issued to the write buffer in program order. A simple write buffer maintains first-in-first-out (FIFO) order in issuing the writes to the next level in the memory hierarchy. More aggressive buffers may allow non-FIFO issue to alleviate issue stalls. For example, the write at the head of the buffer may still be unresolved (i.e., given an out-of-order issue processor) or the next level in the memory hierarchy may temporarily not be able to accept operations to certain addresses (e.g., if there is already an outstanding operation to that line). In such cases, non-FIFO issue can improve the service rate for writes, thus reducing the chance of processor stalls due to buffer overflow.
Another common optimization for write buffers is write merging. The purpose of write merging is to coalesce writes to the same address or nearby addresses into a single write. In an aggressive implementation, each entry in the write buffer can maintain data for a set of consecutive addresses, typically corresponding to the line size for the next level in the memory hierarchy. When a write is issued to the write buffer, the write buffer first searches for an already existing entry for the given address range. In case such an entry already exists, the new data is merged in with the previous data for that entry. Otherwise, a new entry is allocated for the write. In this way, before an entry is retired, any writes issued to the buffer that match the entry are merged and coalesced into a single write with potentially more data. Of course, a side effect of merging and coalescing writes is that writes may be serviced in a different order relative the order they are issued to the buffer.
We now consider optimizations for reads. Read operations are more critical than write operations since the processor typically requires the return value before proceeding with subsequent computation. Therefore, it is beneficial to allow a read to be serviced before pending writes that have already been issued. This requires the read to bypass any writes in the write buffer. To ensure the correct value for the read, such bypassing is only allowed if the address of the read does not match any of the write addresses. This requires an associative match, but the match can be made conservative by comparing fewer address bits. The read is simply stalled if an address match is detected. An optimization in this case is to service the read right away by forwarding the value of the latest write in the buffer that matches its address; this latter match requires comparing all address bits. Both bypassing and forwarding can lead to the reordering of reads with respect to writes relative to the order in which they are issued by the processor. Finally, similar to writes, non-FIFO issue of reads from the buffer may also be beneficial.
The next section describes cache optimizations that exploit relaxed models. An aggressive cache design may reduce the benefits from optimizations such as bypassing and forwarding by increasing the service rate of buffers which increases the chances of an operation encountering an empty buffer; this effect is illustrated
by our simulation results in Section 6.2.3 of the next chapter. Finally, the optimizations discussed above (i.e., non-FIFO issue, merging, bypassing, and forwarding) may also be beneficial at the lower levels in the memory hierarchy.
5.2.3
Caches and Intermediate Buffers
To exploit overlap among memory operations requires a lockup-free cache that is capable of servicing memory operations while previous operations are still outstanding [Kro81]. A standard blocking cache services processor requests one at a time. Therefore, only a single processor request is serviced during a long latency cache miss. It is reasonably straightforward to extend a blocking cache design to allow requests that hit in the cache to be serviced while there is an outstanding miss. To achieve higher performance, it is important to also allow multiple outstanding misses. Lockup-free caches are inherently more complex than standard blocking caches. However, since they play a critical role as a latency hiding mechanism, their use is becoming more widespread both in uniprocessor and multiprocessor designs. The first part of this section describes various design issues for lockup-free caches. The second part considers the design of the intermediate buffers between caches, and issues such as deadlock that arise in handling multiple messages in the incoming and outgoing paths.
Lockup-free Cache Design
We begin by identifying some of the requirements for a lockup-free cache. A general lockup-free cache must support multiple outstanding read and write requests. For writes, the data written by the write must be buffered while the write request is outstanding and must be correctly merged with the reply data. For reads, there needs to be a mechanism to forward the relevant portion of the returning data reply to the requesting unit or destination register.
A key component in the design of a lockup-free cache is the mechanism that is used for tracking outstanding requests. Most designs track outstanding requests in transaction buffers that are external to the cache [Kro81, SD91, FJ94], often referred to as MSHRs (miss information/status holding registers). An alternative approach is to track the requests directly in the cache, as in the remote access cache for the Stanford DASH [LLG+
90, Len92]. Finally, some recent proposals advocate a hybrid approach [Lau94]. The main difference between these designs is in the limitations placed on the number and type of outstanding requests, and the hardware complexity of each design.
Below, we describe a straightforward implementation that keeps track of outstanding requests within the cache. For simplicity, we assume an invalidation-based coherence scheme using a simple write back cache with three states: invalid, clean, and dirty. To support multiple outstanding operations, an extra state is introduced per cache line to represent a pending (or transient) state. A cache line enters the pending state on a cache miss and remains in this state while the miss is outstanding. The line may not be replaced while it is in the pending state. Therefore, operations that conflict on the line must be delayed, allowing only a single outstanding request per cache line (or per set in a set-associate cache). While this restriction may cause some stalls in small caches, larger caches or set-associative caches are less likely to be affected by such conflicts.
As we will see below, there are numerous optimizations possible for accesses to a cache line that already has an outstanding request. The following is a list of such optimizations for operations to the same address
or line: merge reads with outstanding requests as appropriate, read data from a line with a write outstanding and forward the value appropriately, write to a line with a read outstanding, and merge multiple writes to the same line. Some designs may also allow simultaneous outstanding operations to different addresses even when there is a mapping conflict in the cache.
A write miss results in the write data to be written into the cache line and the state of the line to be changed to pending. To allow for correct merging with the reply data, each cache line is augmented with fine grain valid bits (corresponding to the smallest size write operation that is supported). The state of the valid bits are only considered while the line is in the pending state. The appropriate valid bits are set during a write miss as the write data is written to the cache. A future write to the pending line may be merged by simply setting the corresponding valid bits and writing its data into the line. Once the write data reply arrives, it is merged appropriately with the data present in the cache line according to the valid bits. A future read to the pending line may fetch its value if the corresponding valid bits for that data segment are all set. Otherwise, the read must be delayed. As an optimization, an extra pending state may be used to signify a line that was in clean mode before a write miss. We refer to this as the pending clean state. A read to a line in pending clean state may fetch its value immediately regardless of the state of the valid bits. The line can exit the pending or pending clean state as soon as the exclusive or read-exclusive reply returns, even in protocols that provide an early response (i.e., before all invalidation-ack replies are back).
A read miss results in the data to be requested and the state of the cache line to be changed to pending. To allow appropriate routing of the return value when the read reply arrives, the destination register or unit and the actual word address to be read from the line are also recorded in a separate read transaction buffer. The size of this buffer places a limit on the number of outstanding reads. A future read to the pending line can be merged with the pending read if there is sufficient space in the transaction buffer to record the information for the new read. A future write to the pending line is treated in the same way as when the state is set to pending due to a previous write miss (in fact, these two cases are indistinguishable based on the state); the appropriate valid bits are set and the write data is stored in the cache line. The read data reply is placed in the cache on arrival. In addition, for each address/register pair in the transaction buffer that corresponds to this line, the specified word is selected and appropriately routed to its destination. If any of the valid bits are set, the read data reply is properly merged with the write data in the line, a write miss request is generated to fetch the line in exclusive mode, the valid bits remain set, and the line remains in the pending state (or is changed to pending clean state if this state is supported) until the write reply arrives. Even though delaying the servicing of a write miss until the read reply returns is conservative, it greatly simplifies the design.
While the line is in the pending or pending clean state, incoming read and read-exclusive requests (i.e., originating from lower level caches or the external interface) destined for the line must be either delayed or sent back with a negative acknowledgement reply.3 Incoming invalidate requests may be readily serviced,
however. An invalidate request has no effect if the line is in the pending state. An invalidate to a line in pending clean state changes the state to pending to reflect the fact that blocks with clear valid bits contain potentially stale data (and should therefore not be forwarded to future reads).
Another issue in lockup-free cache designs is the out-of-order arrival of replies. In a standard blocking
3Implementations that support a dirty-shared caching state may have to service certain requests in order to avoid deadlock or livelock.
For example, consider P1 with a dirty-shared copy. Assume P1 and P2 both write to A, resulting in an exclusive request from P1 and a read-exclusive request from P2 to be sent to the home. If P2’s request gets to the home first, it will be forwarded to P1 and may need to be serviced by P1 to avoid deadlock or livelock.
cache, there is only a single request outstanding. Therefore, an incoming reply can be easily matched with the request. A simple way to solve the problem in lockup-free caches is to require the reply to carry an address. This allows replies to be easily matched with requests, but implies that the address must be carried back by the reply even outside the cache hierarchy. To reduce the amount of extra bits carried by the reply, the external interface can assign a unique tag to each outstanding request to be carried back by the reply. For example, supporting eight outstanding requests would require only a three bit tag.
In the lockup-free cache hierarchy design described above, only the first level cache needs to support the mechanisms required for dealing with pending requests and appropriately merging requests to a pending line. The first level cache essentially acts as a filter for requests to the lower level caches, guaranteeing that a lower level cache will not receive further requests for a given line while the line has a miss pending.4 Therefore,
while the lower level caches still need to allow multiple outstanding requests, they need not support extra cache states, valid bits, or read transaction buffers. Allowing multiple outstanding requests is a simple extension over a blocking cache in the context of a multiprocessor, since even blocking caches need to simultaneously service processor requests and requests from the external interface.
Compared to a standard blocking cache, the above lockup-free cache design incurs some obvious additional costs. There is a separate transaction buffer that keeps track of address/register pairs for pending read requests. Furthermore, replies must carry either a unique tag or the full address. Finally, each cache line has one or two (in case the pending clean state is provided) extra states in addition to the fine grain valid bits. There is a way to reduce the extra cost per cache line, however. The main observation is that the valid bits are only useful for lines with outstanding requests. Based on this observation, Laudon has proposed a hybrid design that maintains the valid bits for outstanding cache lines in an external transaction buffer, called the pending-write buffer, while still maintaining the write data in the cache [Lau94]. The pending-write buffer can be built as a small fully-associative cache, with the tag pointing to the cache line for which the pending-write entry is allocated. While Laudon’s design only maintains a single pending state (i.e., the optimization involving the pending clean state is not included), it is possible to extend the pending-write buffer to support the extra pending clean state and still maintain a single pending state per line within the cache. Of course, the size of the pending-write buffer places a limit on the number of outstanding write requests as compared to the original approach discussed above which theoretically allows a maximum of one outstanding request per cache line. However, the pending-write buffer can have more entries than a traditional MSHR since each entry is smaller given the write data is placed within the cache.
Overall, there are numerous possible techniques for supporting lockup-free caches with varying imple- mentation and performance trade-offs. While there have been some detailed studies of these trade-offs in the context of uniprocessors [FJ94] and multiprocessors [Lau94], we expect more studies to appear as lockup-free caches become more widely used. Finally, multi-ported and interleaved caches are also becoming more prominent as a way of satisfying the bandwidth requirements of multiple instruction issue processors. Like lockup-free caches, these designs depend on the reordering of operations allowed by relaxed models to