One problem often encountered in numerical computations is that of resolution. For example, in the case of a fluid falling into a black hole, there is very little going on far from the black hole. However, the fluid may develop very fine features near the black hole which would need to be resolved accurately. If a sufficiently high resolution to resolve these were to be used over the whole domain, there would be a lot of unnecessary computation far from the black hole, and the overall evolution would proceed very slowly.
There are two approaches to mesh refinement, depending on whether it can be predicted in advance where the most accuracy will be required.
Adaptive Mesh Refinement (AMR) uses a hierarchical set of grids, where one relatively coarse grid is used to cover the complete domain, and levels of successively finer grids are used in regions where high resolution is required. See Figure 4.7 for an example. The finer grids used are repositioned every few timesteps, so that they adapt to the evolving solution. This decreases the amount of computation used in regions of low activity and smooth solution, as we know that the numerical methods we use have a higher order of accuracy on smooth solutions than near discontinuities. Further, since the grids can be evolved independently of each other, apart from some interaction at the boundaries and when interpolating or extrapolating, the computation can be efficiently parallelised, giving different grids to different processors of a multi-processor computer, although Overture in fact splits individual grids over different processors instead.
The most widely used approach for AMR was developed by Berger and Oliger [19], which refines in both space and time, thus maintaining the effective CFL number of the finite volume schemes being used, and therefore maintains accuracy by minimising diffusion.
In theory, using AMR with a certain effective resolution (the resolution of the finest grid) should give the same results as evolving the entire system at that finest resolution. In practice this may not quite be achieved, but it is possible to come very close to the ideal.
The second approach is Fixed Mesh Refinement (FMR), which is a special case of AMR where the grid structure is not changed throughout the simulation. In this case, the grid structure can be carefully specified beforehand, and then no overhead regarding grid generation need be incurred. However, since in Overture we have sufficient routines to enable us to implement AMR, we do not consider this simplification again.
Level 0 Level 1 Level 2
Figure 4.7: Example AMR grid setup. The finer lines correspond to refined grids, with a refinement ratio of two. Levels 0 and 2 consist of one rectangular grid each, and Level 1 consists of two separate rectangular grids.
4.9.1 Grid creation
In order to be able to use AMR, we must create an appropriate grid structure. This is done as follows:
Error flagging
The first step is to flag points on the coarsest grid where we expect large numerical errors to occur.
One way of doing this is to evolve the coarse grid and an even coarser version of the grid forward by one time-step and to compare the solutions. Where the solutions differ by more than a certain threshold, we flag a grid point on the original coarse grid.
An alternative, which we follow here, is to create some criterion, based on the evolution variables and their derivatives, which tends to pick out places where large errors in the numerical method are likely to occur. For example, we could refine where the density gradient exceeds a certain threshold:
ρ−2γijρ,iρ,j > ǫtol, (4.60) which would tend to pick out regions where there are steep gradients in the fluid density. It has the added advantage that it is scale invariant, i.e. if we double the density everywhere, then the refinement occurs in exactly the same places.
Another possible refinement criterion is to consider the divergence of the momentum
γijSi,j<−ǫtol. (4.61)
(a)Possible grid refinement with an ef- ficiency of 0.28
(b)Possible grid refinement with an ef- ficiency of 0.47
(c)Possible grid refinement with an ef- ficiency of 0.73
Figure 4.8: Series of possible grid structures to refine a grid given the marked points as being flagged.
as suggested by Ruffert [93] since “in shocks matter is decelerated and thus a high negative value of the divergence [of the velocity] appears.”
Once the initial flagging is complete, we extend the flagging regions slightly so as to include nearby regions. This means that when a shock moves, it will still be inside the flagged region, thus retaining the accuracy of the method.
Boxing flagged points
Once we have flagged points, we must create grids which contain them. This must be done with some care; if we simply cover all the flagged points with a single grid, we are likely to cover too much space and to waste computational resources. On the other hand, if we cover only points that have been flagged, we run the risk of creating too many small grids with a lot of computational overhead. We therefore specify an efficiency (we choose 0.7), which dictates the minimum proportion of each grid that should be filled with flagged points. An example with various efficiencies is shown in Figure 4.8.
Once the grids have been decided upon, we also extend them by a few cells in all directions so that the features we are seeking to refine upon do not immediately move outside the refined region. With this, we can avoid redefining the refined grid structure every time-step and only redefine it every four steps, for example.
Child grid creation
The grids are created as described above, with an appropriate refinement factor. This is often taken to be a power of two, although this is not necessary, and any integer ratio is in fact permissible. However, we use a refinement ratio of two throughout this work. Once the first set of grids has been created, we repeat
the process on the new grid level to obtain another refined level, up to some user-defined maximum.
In this way, each refined level is entirely contained within the level one stage coarser than it. In some implementations, it is further required that a refined grid must be contained entirely within a single grid on the next coarser level, although this is not a necessary part of the AMR algorithm, and is not followed by Overture itself, for example. We therefore have the concept of a hierarchical grid structure, which can be coded efficiently within modern programming languages.
When we change the grid structure, we need to find solution data on the newly created grids. This is done by interpolation or copying from the most refined data available. For example, if the grid structure for the third refined level is changed, then the data for those grids can be obtained from data on the second refined level and any solution data previously obtained on the third refinement level. This can always be done due to the parent-child relationship described above.
With the exception of specifying and evaluating the refinement criterion, all the above grid creation steps can be carried out by Overture.
4.9.2 Sub-cycling
Once the grid structure has been determined, we need to find a way of evolving the system consistently so that it retains the accuracy of the other numerical methods used, and does not introduce any new errors. We could simply evolve the grid using the time-step found from the finest grid. However, this will give rise to more diffusive results than necessary on the coarsest grid.
We therefore use a scheme in which all grids are evolved with the time-stepping appropriate to their resolution in a process called sub-cycling. This proceeds as follows for a grid with a single level of refinement:
• Start with all grid levels at timet.
• Impose physical boundary conditions on Level 0
• Store the solution on Level 0 in a temporary array
• Advance the solution on Level 0 by one time-step ∆t
• Advance the solution on Level 1 by one time-step ∆tas follows:
– Impose physical boundary conditions on Level 1
– Advance the solution on Level 1 by a half time-step ∆t/2
– Interpolate non-physical boundary conditions on Level 1 grids from the two stored solutions on Level 0 at timestandt+ ∆t.
– Impose physical boundary conditions on Level 1
– Advance the solution on Level 1 by a half time-step ∆t/2
• Project data on Level 1 back onto any relevant cells on Level 0.
• All levels now have the most accurate data available for timet+ ∆t.
This can be easily extended to multiple levels of refinement. In practice, the sub-cycling algorithm is implemented as a recursive procedure.
In order to maintain flux conservation of our scheme, we should apply a fix-up for the fluxes at cell- faces on grid cells that border a refined grid. However, we have not found this loss of conservation to be a problem, and we postpone applying this correction for future work.
4.9.3 Overlapping grids
With regard to overlapping grids and AMR, there is no increase in complexity once both are imple- mented individually within Overture. AMR grids are defined as refinements of the main base grids of the overlapping grid setup. We have to be somewhat careful as to when we interpolate between grids, and not to mix data from different times. If, however, we only ever interpolate between grids on the same refinement level at once (something Overture lets us do quite simply), then this is not a problem.
4.9.4 Refinement criterion
One other consideration that must be made is that of the refinement criterion. In CFD, this may be related in some way to local truncation error, which requires the use of shadow grids to calculate. Also, in CFD we usually only refine where shocks, or at least steep gradients, occur. In NR, it would also be useful to refine in the presence of strong gravitational fields, which could nevertheless be smooth, since we expect highly dynamic flows here.
When considering the refinement criterion, there is also the issue of coordinate system. If we refine a spherically symmetric black hole then, even if it is represented in a Cartesian coordinate system, we would wish that the refinement be spherically symmetric. In order to achieve this, our refinement criterion must be a scalar comparison, i.e. independent of the coordinate system used. We use separate criteria for the metric and the fluid, respectively:
(3)R=−2∂kVk+DkrsΓr sk −DkrrDkss> ǫtol, and γij∂iρ∂jρ
ρ2 > ǫtolin three dimensions, orγijSi,j<−ǫtolin two dimensions.
(4.62)
We note that these are derived from 3-vectors in the space-like hypersurface, not from 4-vectors in the full space-time, and are therefore not gauge independent.