CHECK LIST DE AUDITORÍA DEL SGSSO

Different than inter-node collective communication, intra-node collective communi- cation focuses on collective communication on a shared memory multicore compute node. With increasing CPU cores and memory nodes integrated into compute nodes, intra-node collectives become equally important as inter-node collectives. Several optimizations have been used to maximize the throughput of intra-node collective communication.

One of the most recent of those efforts is due to Richard Graham et al., who proposed a shared memory-based fan-in/fan-out implementation for multicore MPI

collectives, implemented in the Open MPI SM collective component [27]. Their

optimization focuses on lightweight synchronization, reducing memory copy times, increasing parallelism by copying messages in a pipelined way, and controlling working set size to fit into caches by building a logical fixed degree tree. This shared memory based approach simply treats a multicore/many-core compute node as an SMP system and ignores other architecture characteristics, such as NUMA, memory hierarchy,

and core distance. The fixed degree tree is built following the logical ranks layout, which cannot always reflect architecture characteristics. With more heterogeneity in modern NUMA multi-core designs, it is hard to optimally tune a shared memory based implementation for different platforms.

Most state-of-the-art MPI libraries also select a Tuned -style collective component as a default intra-node collective component on a shared memory compute node. Although Tuned -style collective components were originally designed for single- processor clusters, they can still deliver good performance on SMP-style compute nodes with a careful tuning on shared memory links. However, tuning for an unknown platform is not a trivial task, even for expert users. Indeed, there are too many parameters such as pipeline size, thresholds, etc., and any wrong selection might ruin the overall performance of the Tuned collective. Another severe issue for Tuned collectives is, their logical collective topologies are built based on MPI ranks, which can’t reflect underneath multicore or many-core topologies. The mismatch between collective topologies and hardware topologies often causes performance penalties [43]. To further exacerbate this issue, intelligent process placements, used as a bridge between applications and MPI libraries, e.g., MPIPP [16] or TreeMatch [33], often break regular process-core binding patterns and schedule continuous processes (in the way of MPI ranks) to cores between a long physical distance. This irregular mapping leads into a further mismatch between collective topologies and underneath hardware topologies [43].

Another orthogonal work is toward improving the efficiency of shared memory inter-process communication. Copy-in/copy-out in a shared memory segment is still the most common approach for transferring data between processes on a shared memory platform. The copy-in/copy-out approach implies two memory copies to pass a single message, greatly wasting memory bandwidth and CPU cycles. More severely, when applying the copy-in/copy-out approach into rooted collective operations like Gather(v) or Scatter(v), the ‘copy-in’ or ‘copy-out’ operation in the root process is executed sequentially by N times (N is often the number of cores per compute node).

These sequential memory copies on the root process cause a serious scalability issue with more CPU cores integrated into compute nodes.

Several platform-specific efforts offered single-copy inter-process communication (IPC). For instance BIP-SMP implemented such an optimization for Myrinet based clusters [24]. This idea has spread into most vendor specific HPC stacks, such as Myricom MX, Qlogic IPath, and Bull MPI. Some lightweight kernels enabled an even bigger rework of the model on Cray platforms, thanks to the ability of the operating system to make all processes address spaces accessible to any of them. This unusual

feature enabled single-copy RMA-based communication (SMARTMAP [10]), which

greatly reduces memory copies needed by intra-node message passing, especially

for collectives. Recent Linux kernels support the remapping of other processes’

memory thanks to the XPMEM module [49] which enables similar optimizations

but is restricted to SGI or Cray machines.

Other efforts towards single-copy IPC on general purpose platforms is the kernel-

assisted approach. Lei Chai et al. introduced a kernel memory copy module

interface called LiMIC [34]. This kernel-based approach can reduce the number of necessary memory copies to one. Another similar kernel module called by CMA (cross memory attach) has been integrated into the latest Linux kernel 3.1. KNEM [6, 26] is another similar kernel module used in MPICH2 and Open MPI. KNEM offers additional features such as an asynchronous copy model, vectorial buffer support, and copy offload on dedicated hardware [59]. This approach has already proved to be valuable to increase point-to-point bandwidth between processes communicating

over shared memory compute nodes [13]. Stephanie Moreaud et al. [48] proved

collective communication can get free upgraded performance by using single-copy kernel-assisted memory copy instead of a shared memory copy-in/copy-out approach as underneath point-to-point communication. However, even this upgrading is still not enough to reach full potential performance. Collective algorithms need more control over the underlying memory copy mechanism.