Active Axes and Internal Points
The decomposition of a domain will depend on the size and shape of the domain to be decom- posed. The dimensionality of the mesh must be considered to ensure that decomposition only occurs along axes that are in use. Crucial to this process is the concept of an active axis. An
axis is considered to be active if its maximum (xmax,ymaxorzmax) is greater than or equal to 3.
The reason that three points is significant with respect to the length of an axis is due to the
way points at the extremes of the axis are used. InQUI, points at the extreme of the axis would
be updated by a boundary conditions device, such asneum3dordirich3d. The remaining points,
i.e. those contained by the default space, would be operated on by theeuleranddiff3ddevices.
Because the stencil of the diffusion device will cause it to reference immediate neighbours, it is important that the diffusion device does not operate on points at the extremes of the medium, lest it refer to non-existent points beyond the mesh’s edges. For this reason, we make the distinction between internal and boundary points on an axis. Boundary points have coordinates of 0 and
imax−1. Internal points range from 1≤i≤imax−2. We identify an axis as active if there
are three or more points as this allows for at least one internal point, with a boundary point at either end. Figure 3.3a shows points defined by the user and Figure 3.3b shows internal and boundary points.
To avoid ambiguity regarding boundary allocation, the MPI implementation adds a condition tostate()that if the user specifies exactly 2 points along any axis, an error will be raised and Beatboxwill quit.
(a) The mesh. (b) Mesh with boundary points identified.
3
0 1
4 5
2
6 7 8
ix1 0 2 iy 0 1 2(c) Mesh with overlaid supergrid.
0
1
2
3
4
5
6
7
8
(d) Subdomains with halos.
3
0 1
4 5
2
6 7 8
(e) Subdomains united.
3.2. DOMAIN DECOMPOSITION 75
local_ymax
local_ymin
local_xmin
local_xmax
Figure 3.4: Limits of the local subdomain.
Subdomain Bounds
The MPI implementation aims, wherever possible, to maintain the illusion for devices and users that the program is running sequentially. Although local subdomains contain only a subset of the points in the simulation medium, the values held at those points are accessed by the same coordinates as would be used in the sequential version. To achieve this, each process must know the area of the mesh represented by its local subdomain.
To define the area on which each process will operate, the MPI implementation adds local
minima and maxima for each process. These determine the range of coordinates of internal
points for that subdomain. For example, internal points in the local subdomain are those with
coordinates local imin≤i < local imax, as illustrated in Figure 3.4.
The values of local minima and maxima are arrived at by dividing internal points of the
medium between processes, as illustrated in Figure 3.3c. Pseudocode for obtaining subdomain
dimensions on thexaxis is given inGetSubdomainDimensions. Code for the process is given
GetSubdomainDimensions()
Compute approximate slice size
1 x slice←(xmax−(ONE×2))/nx)
Remaining points will be shared amongst the first processes on the axis
2 x remainder ←(xmax−(ONE×2))mod(nx)
3 if ix <x remainder
4 then
5 local xmin ←ONE+(ix×x slice) +ix
6 else
7 local xmin ←ONE+(ix×x slice) +x remainder
8 if ix <x remainder
9 then
10 local xmax ←ONE+((ix+1)×x slice) + (ix+1)
11 else
12 local xmax ←ONE+((ix+1)×x slice) +x remainder
Repeat fory andz axes. . .
After decomposition, the union of internal points from all subdomains will be equivalent to the default space. As shown in Figure 3.3d, around each subdomain is a single layer of points,
referred to as ahalo, which performs two roles. Where halo points lie at the ends of an axis, they
will act as the boundary points of the simulation medium. Halo points on internal edges of a subdomain are used to hold local copies of points from neighbouring processes. The remainder of the halo points will be used to hold copies of values from neighbouring subdomains, as described in Section 3.3. Placing points outside the default space in the halos in this way allows for more consistent behaviour when adapting devices to run on decomposed subdomains. This is discussed in more detail in Section 3.2.3.
AccessingNewin Parallel
Accessing data in the simulation medium requires mapping the coordinates of the point being
accessed to the corresponding array index inNew. As inQUI, this mapping is performed by the
ind()function macro (state.h). The MPI implementation maintains the coordinate space of the
simulation medium by providing an alternative version ofind(), conditionally compiled, to take
account of the subdomain’s position in the mesh. Pseudocode for the parallelisedind() macro
is given in theParallelIndalgorithm, below.
ParallelInd(x,y,z,v)
1 return((x+ONE)−local xmin)×vmax zmax ymax+
((y+TWO)−local ymin×vmax zmax+
((z+TRI)−local zmin)×vmax+v
Figure 3.3e illustrates how the medium can be reconstructed from its subdomains. Overlap- ping points, where dots are contained in boxes, will have the same coordinate values in both
subdomains, although every point will beinternal to only one process. As described in the Sec-
tion 3.3, halo points on the internal edges of subdomains (boxes) will hold a copy of the values from their corresponding internal point (blue dot).
3.2. DOMAIN DECOMPOSITION 77