(BSC-CNS), is a Computation Mechanics (CM) code with two main features. Firstly, it is specially designed to run with the highest efficiency standards on large-scale supercomputing facilities. Secondly, it is capable of solving different physics problems, each one with its own modelling characteristics, in a cou- pled way. These two main features are intimately related, which means that any complex coupled problems solved by Alya will still be solved efficiently. Alya’s architecture is modular, grouping the different tasks into kernel, modules and services. The kernel, the core of Alya, contains all the facilities required to solve any set of discretized PDEs (e.g., the solver, the I/O, the coupling, the elements database, the geometrical information, etc.). Each module describes the physical description of a given problem (e.g, the discretized terms of the PDE, the meaning of the boundary and initial conditions, etc.). Finally, the services contain the tool boxes providing several independent procedures to be called by modules and kernel.
The parallelization of Alya is a service inside the code is based on a mesh partitioning technique performed by METIS (see the reference: METIS (2014) for more details) and uses a Master-Slave strategy. The main input data of METIS are the element graph and the weight of the vertices of the graph. The number of Gauss integration points is used to control the load balance of hybrid meshes related to the former aspect. The element graph is a cru- cial point in the application of the HERMESH method and will be discussed later on. To construct this element graph needed by METIS, two strategies are possible. On the one hand, an adjacent element to an element e can be considered as all the elements sharing a node with e. However, this graph can be quite memory- and time-consuming. On the other hand, the other strategy consists in taking as adjacent elements to e only the elements sharing a face with e. The latter option requires much less memory. As a reference for valu- ing the difference in memory consuming, for a regular hexahedra mesh the face connectivity criterion gives 6 neighbours while the node connectivity criterion gives 26 neighbours. The former strategy will be referred to as by-faces and the latter as by-nodes.
Based on the Master-Slave strategy, the Master is in charge of reading the mesh, of performing the partitioning and the output. The slaves are in charge of the construction of the local right-hand side (bi) and the local matrices
(Ai) and of the solution of the resulting system in parallel. In other words,
each process will be in charge of each subdomain, which are the slaves. In the assembling tasks, no communication is needed between the slaves, and the scalability only depends on the load balancing. In the iterative solvers, the scalability depends on the size of the interfaces, which METIS minimizes, and the communication scheduling. All the details on the code parallelization can
be found in Houzeaux, Vázquez, Aubry, & Cela (2009).
Figure 2.10 is a schematic flowchart for the execution of a simulation using Alya. The tasks that the master process is responsible for are shown on the left side of the same figure with a grey background. As we have explained, it performs the first steps of the execution, namely reading the file and parti- tioning the mesh. Afterwards, the master sends the corresponding subdomain information to each worker process or slaves; then the master and the slaves enter the time and linearization loops, represented as one single loop. Along
Figure 2.10: Parallel flowchart. Master (grey) and n slaves (white).
with the execution of the iterative solvers carried out by the slaves, two types of communications are required to exchange interface information with their neighbours using the MPI functions:
• Point-to-point communications via MPI_SendRecv, which are used when sparse matrix-vector products are carried out.
• Global communications via MPI_AllReduce, which are used to compute residual norms and scalar products.
All solvers need both these types of communication, but, when using com- plex solvers like the DCG, additional operations may be required, such as the MPI_AllGatherv functions explained in öhner, Mut, Cebral, Aubry, & Houzeaux (2011). In the current implementation of Alya, the solution obtained in parallel is, up to round-off errors, the same as the sequential one all the way through the computation. This is because the mesh partition is only used for
distributing work without, in any way, altering the actual sequential algorithm. This would not be the case if one took into account more complex solvers, like the primal/dual Schur complement solvers, or more complex preconditioners, like linelet Soto, Löhner, & Camelli (2003) or block LU.
The code has proven to scale well on the main European supercomputers, as shown in Figure 2.11. As implemented in this work, the HERMESH method
Figure 2.11: Parallel performance of Alya.
can be viewed as part of the preprocess and the mesh partitioning is carried out after the mesh gluing. Therefore, the parallel performance of the code is automatically inherited by the current implementation.
Chapter
3
Proposed Method : HERMESH
3.1
Motivation
As has been presented in the first chapter we propose to devise a mesh compos- ing method with certain characteristics. In particular, the three requirements of the method were: implicit, versatile and parallel. These are essential condi- tions in order to be able to solve real problems in different contexts as they are solved in the department where HERMESH has been developed. So the method has to be able to address problems with non-matching grids whether they be disjoint with (as can be observed in Figure 3.1-a) or without a gap (as can be observed in Figure 3.1-b). Also, the method has to be able to solve the mesh coupling with a certain overlapping between independent meshes (as we can see in Figure 3.1-c) or even Chimera-type problems (illustrated in 3.1-d).
The scenarios represented in Figure 3.1 a,b and c belongs to the applica- tion referred to in the first chapter as mesh gluing. The other scenario, Figure 3.1-d, corresponds to the Chimera-type problem. The particular details of the proposed method for both cases will be explained in the following chapters, 4 and 5. In this chapter, the principles and general properties of the HERMESH method will be described, as will certain details related to the implementation. In order to fulfil the three main requirements described before (implicit, parallel and versatile), we have come a long way, and it is basically divided in these three main points:
• Making implicit the explicit conditions transmissions which was described in Houzeaux (2003).
Figure 3.1: Possible HERMESH scenarios
• Stabilizing the transmission conditions in the variational multiscale frame- work.
• The HERMESH method.
These points will be described in this chapter in order to gain a better under- standing of proposed method.