Criterios Generales de la Inversión

Benchmarking is the act of measuring some aspect of a system. A benchmark is a small program designed to stress one particular part of a supercomputer. The output of a benchmark is a set of metrics, typically including runtime, that can be used to infer things about the performance of the hardware being mea- sured. In high-performance computing, more complicated application bench- marks are often used. Production supercomputing applications are large, com- plex, and often commercially sensitive. A benchmark provides a way to mea- sure aspects of a prospective system without needing to release sensitive source code. Despite being smaller than production applications, the degree to which a benchmark represents a production application often means it will still be large. Figure 2.7 shows the range of benchmark categories and their corresponding simplicity and representativeness.

Micro-Benchmarks

A micro-benchmark is designed to time specific machine components. The most famous benchmark is LINPACK [115], and more specifically, the portable High-Performance Linpack (HPL) implementation [48, 134]. A simple linear algebra program first used in the 1970s, the LINPACK benchmark has become responsible for determining which machine is classified as the fastest in the world. Performing dense matrix-matrix operations is a good test of floating point performance, and the output of the LINPACK benchmark is the number of floating point operations performed per second on the system being bench- marked. Other low-level numerical benchmarks include Livermore Loops, a collection of small loops designed to characterise the kind of operations per- formed by key applications at Lawrence Livermore National Laboratory (LLNL) [109], and the HPC Challenge benchmark suite [101], a set of seven tests to measure numerical, memory, and network performance.

Another example of a micro-benchmark, SkaMPI, is used to examine the performance of the machine interconnect using the MPI library [140]. The benchmark can be configured to perform multiple repeated communications, with each transfer being timed. The output of the benchmark is a set of timings, typically grouped by message size. This data allows an engineer to examine network performance in the machine being benchmarked. Alternative bench- marks for measuring network performance include MPPTest [63] and the Intel MPI Benchmarking Utility [84].

Other machine components such as memory can also be benchmarked. The STREAM benchmark measures memory bandwidth by executing a variety of read and write operations [107]. These operations are arguably more im- portant that the famous FLOPs metric, since many modern codes are memory- rather than compute-bound, i.e. they are limited by how fast they can fetch data rather than how fast the data can be processed.

The problem with benchmarks like SkaMPI and LINPACK is that they very rarely mimic the behaviour found in real applications. Rather than in-

tensive sections using a small subset of the architectural capability, production applications contain a unique mix of memory access, computation, and com- munication. Whilst benchmarks that are cut-down (either in terms of code base or problem size) versions of production codes can be useful, the complexity in- herent in a production application can make it difficult to reason about its be- haviour and attribute different performance aspects to architectural capability. In the early 90s, the correlation between HPL and real application performance was good, but the regular memory access patterns exhibited by the dense lin- ear algebra kernels no longer correlate with the more irregular pattern seen in production applications.

The recently proposed High-Performance Conjugate Gradient (HPCG) benchmark aims to provide a more realistic mix of computation, memory ac- cess, and communication that resembles production applications [47]. Unlike the regular memory access pattern of HPL, HPCG contains a mix of more com- plex communication and computation patterns that reward investment in col- lective communication operations and the performance of the local memory system. Both these characteristics have been shown to affect the performance of real applications. The legacy of HPL, and its use as tool for ranking the fastest supercomputers in the world, looks set to maintain some influence on the HPC design space, but HPCG now provides an alternative ranking system that should more accurately reflect FLOPs achieved by real applications.

Application Benchmarks

The benchmarks discussed so far are small in both number of lines of code, and the scope of which parts of a machine they are designed to stress. Typ- ical scientific codes are large, and problems of interest can have runtimes of months. To make performance analysis via benchmarking feasible, application benchmarks must be smaller, and problem runtimes must be in the range of hours, rather than days or weeks. Application benchmarks aim to replicate the behaviour of a large parallel code but with fewer lines of code and less complex-

ity. Examples of application benchmarks include the NAS Parallel Benchmark Suite [12, 13], the ASC benchmarks [96], and codes such as Shamrock and Hy- dra from the United Kingdom Atomic Weapons Establishment [41, 70].

Application benchmarks fall short of being small enough to enable the rapid investigations necessary for future programming models and architec- tures. Hence in this thesis we consider mini-applications, a class of benchmark that is carefully designed to allow easy experimentation with programming languages, programming models and machine architectures.