|
p4est
2.8.7
p4est is a software library for parallel adaptive mesh refinement.
|
An overview of the ghost layer functionalities.
Mesh-based computations may require access to neighboring elements. An example of such a computation using a ghost layer can be found in steps/p4est_step3.c (2D) and steps/p8est_step3.c (3D) – a simple parallel advection solver. For parallel computations, we we need to ensure that neighboring mesh elements, which are not local to the current process, become accessible locally. The need of process-neighboring mesh elements leads to the definition of the ghost (a.k.a. halo) layer: The ghost layer is the set of all remote, i.e. non-local, mesh elements that are adjacent to the process-local mesh elements. This data structure is ideally suited to establish the communication pattern between mesh-adjacent parallel processes. The ghost layer contains all information required to determine all sender-receiver pairs without additional MPI communication. The pattern is necessarily symmetric: Each sender of a pair is also a receiver and vice versa.
p4est provides the functions p4est_ghost_new (2D), p8est_ghost_new (3D) to construct a ghost layer. A ghost layer can be constructed from a valid input_p4est (cf. p4est_t (2D), p8est_t (3D) and p4est_is_valid (2D), p8est_is_valid (3D)). The forest does not need to be 2:1 balanced.
The second and last required parameter to construct a ghost layer is a btype (cf. p4est_connect_type_t (2D), p8est_connect_type_t (3D)) that specifies if the ghost layer should collect face adjacency, face and edge adjacency in 3D, or full adjacency including corner neighbors.
Ghost layer construction in 2D:
ghost = p4est_ghost_new (input_p4est, btype)
The returned ghost layer object is explained in the following section.
The ghost layer object (cf. p4est_ghost_t (2D), p8est_ghost_t (3D)) can be queried and searched without accessing the original input_p4est. It is read-only immutable and must be destroyed when no longer needed (cf. p4est_ghost_destroy (2D), p8est_ghost_destroy (3D)).
The ghost layer is a public struct with documented entries, most of them arrays. The 2D and 3D declarations are structurally identical: Linear tree storage is dimension independent, and ghost elements are ordered ascending just as mesh elements. We may index into the element storage by window start and offset indices. We may also use binary search in the linear order to find ghost elements.
The elements of the ghost layer start with the mpisize and num_trees. They store the number of MPI ranks and the number of trees of the ghost layer. Both elements are as in the p4est that was used to construct the ghost layer.
The array ghosts stores the ghost elements that form the ghost layer of the process-local elements. The quadrants in the ghosts array store their tree index and the local number in the owner's numbering in piggy3. The ghosts array is windowed by the two arrays tree_offsets and proc_offsets. Therefore, the MPI rank and the tree number of each element in ghosts can be deduced. Naturally, some trees or processes may have zero ghost elements for a given process, in which case the window has length zero.
The mirrors array stores all local quadrants that are ghosts to remote processes, sometimes called the (inside) parallel boundary elements. ghosts can be considered as the outside parallel boundary elements. One mirror quadrant may be ghost to more than one remote process. Thus, the indexing structure is slightly less direct than for ghosts: We have an array mirror_proc_mirrors that contains one set of indices into the mirrors for each remote process. The array mirror_proc_offsets indexes into these sets, which vary in length by the remote process. As for ghosts, we have for mirrors also an array with tree offsets, namely the array mirror_tree_offsets.
For further elements of the ghost data structure see also p4est_ghost_t and p8est_ghost_t and for parallel ghost data exchange functions see the respective functions in p4est_ghost.h (2D) and p8est_ghost.h (3D).
Besides the already mentioned parallel advection solver in 2D and 3D, one can also find a simple synthetic usage examples of ghost in timings/timings2.c (2D) and timings/timings3.c (3D).
