Conexión de módulos de expansión externos

The performance of the hybrid MPI/MPI parallelization has been assessed using the d50 50000 dataset. The experiments were conducted on BGL partitions of 32, 128, and 512 processors. For the sake of completeness, Tables 6.5 and 6.6 provide the absolute execution times of these experiments.

# Total processors # Processors/Group 8 16 32 64 128 32 1,984s 963s

128 1,986s 964s 502s 291s 512 1,977s 963s 502s 291s 190s

Table 6.5: Runtimes for a single tree search on the d50 50000 dataset in multiple-groups configuration on the BGL system.

# Total Processors # Processors/Group 16 32 32 15,387s 15,008s 128 3,850s 4,006s 512 963s 1,005s

Table 6.6: Runtimes for 32 distinct tree searches on the d50 50000 dataset in multiple-groups configuration on the BGL system.

Figure 6.15 shows execution times for individual tree inferences using groups of 8, 16, 32, 64, and 128 processors. The blue line shows the times for a single master-worker group (see Table 6.4). The remaining three graphs depict execution times for multiple master-worker groups on the aforementioned BGL partitions which have been split into 4, 8, 16, 32, and 64 groups (where applicable) using MPI Comm split() as described in Section 5.3.3. So for example, the red graph shows the runtimes of the experiments on a total number of 128 processors. The datapoint at ”32 Processors/Group” depicts the average runtime of 4 groups with 32 processors each. Note that the experiments have been organized such that each group uses the same starting tree and hence yields identical results.

As expected, the runtimes observed for the multiple groups setup are slightly higher than the corresponding runtimes of a single master-worker group. This is due to the fact that on the BlueGene/L system messages are sent over the higher latency peer-to-peer network in case of the multi-group setups while a single group can utilize the faster specialized collective network (see Section 5.3.3). Note that on any other supercomputer or cluster that does not provide such a specialized collective network but utilizes a single network for all message types, no runtime differences should occur between the single-group and a multi-group setups.

Figure 6.16 depicts the total execution times for 32 distinct tree searches on 32, 128, and 512 processors that have been split into groups of 16 and 32 processors. So, for example, in case of 128 processors and 16 processors per group, 8 masters with their private set of 15 workers perform 32 distinct tree searches in parallel, four for each group. Again, identical starting trees have been used for the individual tree searches such that each experiment yielded 32 identical result trees. The plot shows that the total execution time decreases linearly with an increasing number of total processors used for computation. Furthermore, one can see that groups of 16

6.5. PERFORMANCE OF THE HYBRID MPI/MPI PARALLELIZATION 105 0 200 400 600 800 1000 1200 1400 1600 1800 2000 8 16 32 64 128 Runtime [s] # Processors/Group single group multiple groups, 512 processors multiple groups, 128 processors multiple groups, 32 processors

Figure 6.15: Execution times of multiple groups setup on the d50 50000 dataset.

0 2000 4000 6000 8000 10000 12000 14000 16000 32 128 512 Runtime [s] # Total Processors 32 processors/group 16 processors/group

processors perform slightly better than groups of 32 processors – as expected, given the absolute execution times for single groups in Table 6.4.

The experiments show that the total number of distinct groups does not influence the run- time of the individual tree searches. That is, the runtime per tree search depends solely on the number of processes (workers) per group, not on the number of groups. This also proves that communication between sub-masters and the super-master is infrequent enough to not influence the fine-grained parallelism within each group.

Chapter 7

Conclusion and Future Work

This final chapter provides the conclusion of the work and gives an outlook on potential direc- tions for future research.

7.1

Conclusion

The inference of large phylogenetic trees from molecular sequence data is still one of the grand challenges in bioinformatics. The steadily growing amount of sequence data that is available for phylogenetic analyses continuously poses new challenges for the developers of tree inference tools. As the input datasets for phylogenetic analyses are growing larger and larger, problems arise not only from the computational requirements of such analyses but also from their memory requirements. In fact, the memory requirements can easily exceed 100 GB and hence render a tree inference impossible on most workstations. Even on large HPC systems, that amount of memory is typically not available to a single process as they are composed from many smaller nodes with significantly less memory. Applications that aim at facilitating the inference of huge phylogenetic trees therefore have to meet two requirements: firstly, they need to distribute their computations over multiple processors in order to gain the required computational power. Secondly, they also need to distribute their data structures over multiple nodes for gathering the necessary amount of aggregate memory.

Inferring phylogenetic trees under the Maximum Likelhood criterion exhibits several poten- tial sources of parallelism at different levels of granularity: loop-level parallelism, inference-level parallelism, and embarrassing parallelism. Although all of them can be exploited for distribut- ing the computational workload, only the exploitation of the loop-level parallelism within the phylogenetic likelihood function allows for also distributing the data. Yet, exploiting such a fine- grained source of parallelism under a rather coarse-grained distributed memory programming paradigm like MPI – which is a necessity for acquiring huge amounts of aggregate memory – is not as straight forward as its exploitation under shared memory programming paradigms like OpenMP.

The need for parallel tools for phylogenetic analyses is not only driven by the molecular data flood, but also by the developments in computer architecture over recent years. Due to

the power wall, the performance of new computer systems can only be increased by increasing the degree of parallelism. This applies to computers of all scale – desktop computers as well as supercomputers. In the multi-core era, even the smallest notebooks have at least two cores, powerful workstations with less than sixteen cores have become rare, and supercomputers com- prise thousands of cores. Consequently, parallel tools are required even for comparatively small phylogenetic analyses and large-scale analyses require highly optimized tools that are able to scale up to thousands of processors.

Apart from the specific problems in phylogenetic inference, the new parallel computer architectures pose new challenges for all application domains. As the multi-core trend continues, microprocessors will become increasingly complex and heterogeneous, and the total number of cores will increase in computers of all scales. As a high degree of complexity and parallelism is hardly manageable manually by the user, new automatic tools are required that allow for exploiting new architectures efficiently – for end-users as well as for developers.

The work described in this thesis mainly focuses on parallelization concepts for phylogenetic tree inference under the Maximum Likelihood criterion. The concepts have been implemented in RAxML under the shared memory programming paradgim (OpenMP and PThreads) as well as under the distributed memory programming paradigm (MPI). However, as the concepts are generic, they could easily be adopted in any other ML-based tool. Due to similarities of Bayesian and ML-based phylogenetic tree inference, most of the concepts can also be applied to Bayesian programs. The performance of the parallel implementations has been assessed on eight different computer systems with four different datasets. All implementations show good speedup values and in some cases even super-linear speedups. The MPI parallelization overcomes memory requirements as a limiting factor for phylogenetic analyses and has proven to scale up to 2048 processors. It has recently been chosen to become part of the SPEC MPI2007 benchmark suite which is used to assess the performance of large-scale computer systems. Furthermore, it has been used to perform the most computationally intensive real-world phylogenetic analysis to date for which over 2.25 million CPU hours have been utilized.

Besides the parallelization concepts for phylogenetic tree inference, the work also covered problems that are due to the continuously increasing degree of parallelism of new computer ar- chitectures: the novel properties of recent multi-core architectures have been analyzed and their potential performance bottlenecks described. Additionally, the autopin tool has been developed that allows for performance-optimized and automated thread pinning on these architectures and hence helps to exploit their performance efficiently by by-passing bottlenecks.

Additionally, new scalable approaches for identifying performance bottlenecks in MPI ap- plications have been developed and implemented in the peformance analysis tool Periscope. They are based solely on summary information and therefore facilitate automatic performance analysis, even on large-scale distributed memory systems.