Estudiar el principal producto de exportación ecuatoriana a los Países

Following the results analysis documented in Section 5.4.1, an additional set of experiments was conducted on Archer. The aim of these additional experi- ments was to examine the e↵ect of: the proprietary Cray non-blocking SHMEM operations; employing 4MB huge-pages; and applying the FTL optimisation (Section 5.3) to the CAF bu↵er-exchange based version. The same experimental methodology, previously outlined in Section 5.4, was followed and the results of the experiments are presented in Figure 5.6. As these experiments were conducted at a di↵erent time (di↵erent system loads) and using di↵erent node allocations from the batch system, compared to the first set of experiments, the performance results between the two sets of experiments will di↵er, particularly at scale. This thesis therefore only presents performance comparisons within each set of experimental results rather than between them.

FTL Optimisation Technique

It is evident from Figure 5.6 that the CAF bu↵er-exchange based version does indeed benefit significantly from the FTLoptimisation. The modified version delivers substantially superior performance at all the node configurations exam- ined, achieving 2.2⇥and 1.9⇥more iterations/sec during the 1,024 and 2,048 node experiments, respectively. Although significantly improved, its perfor- mance still does not quite match that of the equivalent OpenSHMEM-based version particularly at large node counts. In the 2,048 node experiment the OpenSHMEM bu↵er-exchange version achieved 184.23 iterations/sec compared to 138.01 for the CAF-basedFTLversion, an improvement of 1.33⇥. As in the initial set of experiments, the original OpenSHMEM version is not able to match the performance of the equivalent MPI implementation. It achieved 135.21 and 184.23 iterations/sec in the 1,024 and 2,048 node experiments respectively, compared to 153.92 and 209.65 for the MPI version.

Non-blocking SHMEM Operations

The use of the proprietary Cray non-blocking operations, however, delivers some further performance benefits for the OpenSHMEM-based versions, particularly at high node counts. The performance of the version which utilises these non-blocking operations is virtually identical to that of the original in the experiments_256 nodes. At 512 nodes and above, however, it starts to deliver significant performance advantages, achieving 206.66 iterations/sec in the 2,048 node experiment, compared to only 184.23 for the original version. In both the 1,024 and 2,048 node experiments it also delivered broadly equivalent perfor- mance to the MPI implementation, achieving 155.31 and 206.66 iterations/sec respectively, compared to 153.92 and 209.65 for the MPI version.

This demonstrates that the use of the proprietary non-blocking operations can deliver some significant performance improvements for this class of ap- plications, by reducing the overheads associated with inter-process message communication and enabling sequences of messages to be more rapidly injected into the network. The OpenSHMEM standard would therefore benefit from the standardisation of these constructs within future versions of the specification. Utilisation of Huge Memory Pages

The performance benefits observed from employing the larger 4MB huge mem- ory pages are even more significant. In the 2,048 node experiment the version which utilised these larger page sizes achieved 217.42 iterations/sec, a 1.2_⇥im- provement over the original OpenSHMEM version and an improvement of 7.78 iterations/sec over the equivalent MPI implementation. Interestingly, however, its performance was fractionally worse than the original OpenSHMEM version in all of the experiments below 1,024 nodes.

5.5

Summary

The research presented within this chapter examined the PGAS based program- ming models of OpenSHMEM and CAF as potential candidate technologies for delivering performance advantages, on current and future system architec- tures, for the explicit hydrodynamics applications which CloverLeaf represents. Related work in the field was documented together with the implementation of multiple CAF- and OpenSHMEM-based versions which were developed as part of this work. The performance of each programming model is evaluated and compared to an equivalent MPI-based implementation, at considerable scale (up to 2,048 nodes/49,152 cores) on two significantly di↵erent, whilst still state-of-the-art, system architectures from two leading vendors.

The recorded results demonstrate that the OpenSHMEM PGAS program- ming model can deliver portable performance across both the Cray and SGI system architectures. On the SGI ICE-X architecture it is able to match the performance of the MPI model, whilst delivering comparable—albeit surpris- ingly slightly slower—performance compared to MPI on the Cray XC30 system architecture. Use of the proprietary Cray non-blocking operations, however, enabled the performance of the SHMEM-based versions to match and sometimes exceed that of their MPI equivalents. Additionally, the library-based PGAS model of OpenSHMEM can be significantly more performant than equivalent language/compiler-based PGAS approaches such as CAF on the Cray XC30.

Applications based on either PGAS paradigm can also benefit, in terms of improved application performance, from the aggregation of data into communi- cation bu↵ers. This enables the required data to be collectively communicated to the remote processes, rather than moving it via strided memory operations. The performance of CAF-based applications was also shown to be sensitive to the selection of appropriate co-array data structures within the application, as this can have implications for how these data structures are accessed by remote memory operations.

This research also demonstrated that performance improvements can be achieved, for both OpenSHMEM- and CAF-based applications, by employing

point-to-point synchronisation mechanisms rather than global synchronisation primitives. Furthermore, the selection of implementation mechanisms for the

point-to-pointsynchronisation operations (shmemwaitorvolatile variables), and the choice of the remote memory operation ordering constructs (fence and

quiet), was shown to not significantly a↵ect the overall performance of this class of application. Similarly, the use of the proprietary Cray CAFpgas defer sync constructs and the optimisations to overlap communications and computation also do not significantly a↵ect overall application performance.

Portable Performance Through OpenCL

This chapter documents the work undertaken to assess the utility of OpenCL for delivering portable performance for hydrodynamics applications. In particular it examines the ability of OpenCL to express intra-node parallelism and im- plement a hybrid programming model which enables multiple novel processing architectures (e.g. GPGPUs) to be utilised for this class of application. Related work within this research arena is first discussed within Section 6.1. Following this the actual OpenCL implementation of CloverLeaf, produced as part of this research, is documented in Section 6.2 together with several optimisations which have been implemented within the codebase (Section 6.2.2). Results from both small (single processor) and large scale experiments are then analysed in Section 6.3. Finally, Section 6.4 summaries the findings of this research and concludes the chapter.

6.1

Related Work

Insufficient work has, to date, been undertaken to examine whether OpenCL is a viable alternative programming model for delivering intra-node parallelism on HPC system architectures, particularly for Lagrangian-Eulerian explicit hy- drodynamics applications. This includes examining whether OpenCL runtime systems are now able to automatically optimise a single source-code for di↵erent platforms in order to achieve portable performance for these hydrocodes, or whether device specific optimisations are still required.

A considerable body of work has, however, examined porting smoothed particle hydrodynamics (SPH) applications to GPU-based systems [68, 47, 164, 174]. These applications generally employ mesh-less, particle based numerical methods and are therefore significantly di↵erent to the hydrodynamics scheme simulated within CloverLeaf. Existing studies have also predominantly focused on utilising CUDA and have not sought to examine OpenCL as an alternative technology for delivering portable performance.

Bergenet al.developed an OpenCL version of a finite-volume hydrodynamics application which is similar to CloverLeaf [25]. They do not, however, present any performance results or compare the development, performance or porta- bility of the application to alternative approaches or across architectures. The GAMER library also provides similar functionality, however, it is implemented entirely in CUDA and therefore does not allow OpenCL to be evaluated as an

alternative approach [182]. Additionally, Brooket al. present their experiences porting two computational fluid dynamics (CFD) applications to an accelera- tor [32]. Whilst their Euler-based solver has similar properties to CloverLeaf, they focus exclusively on the Intel Xeon Phi architecture and employ only the OpenMP programming model.

Existing work has examined using OpenCL to deliver portable performance within other scientific domains. Pennycook presents details of the development of OpenCL implementations of the NAS LU benchmark [159] and a molecular dynamics application [160], which achieve portable performance across a range of current architectures. Similarly, Brownet al.describe work and performance results, for both OpenCL and CUDA, within the molecular dynamics domain which enables computational work to be dynamically distributed across both CPU and GPU architectures [34]. Du [60] and Weber [205] also provide direct analyses of OpenCL’s ability to deliver portable performance for applications targeting accelerator devices; however, both focus on di↵erent scientific domains. Additionally, Komatsu [116] and Fang [66] provide a detailed examination of the performance of CUDA and OpenCL, as well as the performance portability of both programming models. Van der Sanden also evaluates the performance portability of several image processing applications expressed in OpenCL [198]. The majority of existing work also focuses on accelerator devices; conse- quently there is considerable uncertainty regarding how to optimise OpenCL codebases for CPU devices. Several techniques for improving performance on CPU architectures are, however, presented in [112]. Lan et al. also document several techniques for improving the performance of GPU-focused OpenCL

kernels on CPUs [118]. Additionally, Seoet al.examine how optimised versions of the NAS parallel benchmarks should be expressed in OpenCL for both CPU and GPU architectures [177].

OpenACC [155] has recently emerged as a new, directive-based, program- ming model for porting applications to accelerator devices. Consequently in- sufficient work has thus far been conducted to assess the utility of OpenCL- based approaches relative to this model, however, Wienke et al. do provide one direct comparison [207]. Although little work exists which has examined using OpenCL to scale this class of application to the levels examined in this research, Levesque et al. used OpenACC at extreme scale within the S3D application [122].

Existing studies have examined utilising OpenCL together with MPI to deliver portable performance [163, 189]; however, these studies have generally focused on applications from di↵erent scientific domains. Additionally, Kim et al. propose a novel framework which enables OpenCL applications to be exe- cuted in a distributed manner [113].

Auto-tuning has also been recognised as a key technology for enabling scien- tific applications to be rapidly ported to, and achieve optimal performance on, new computational platforms. In [56] Dolbeau et al.examined using OpenCL as a target software layer for an OpenACC compiler, as well as employing an auto-tuning strategy to achieve optimal performance on a range of processor technologies. Rahman et al. developed an auto-tuning framework with the ability to optimise for both performance and energy efficiency, it supports a broad range of code optimisation techniques, and they demonstrate it on several commonly used scientific kernels [169]. The tuning of thread counts and loop tiling parameters was also shown to deliver significant performance improvements by improving cache utilisation by Jordan et al. [108]. Simi- larly, Kamil et al. examined applying an auto-tuning strategy to a range of stencil-based codes to achieve portable performance across several di↵erent processor architectures [111]. Additionally, Zhanget al.examined auto-tuning stencil computations on GPU architectures, although their work focused on the iterative Jacobi method and the CUDA programming model [210].