Accurate numerical solution of time-dependent, high-dimensional partial differential equations (PDEs) usually requires efficient numerical techniques and massive-scale parallel computing. In this thesis, we implement and evaluate discretization schemes suited for PDEs of higher dimensionality, focusing on high order of accuracy and low computational cost. Spatial discretization is particularly challenging in higher dimensions. The memory requirements for uniform grids quickly grow out of reach even on large-scale parallel computers. We utilize high-order discretization schemes and implement adaptive mesh refinement on structured hyperrectangular domains in order to reduce the required number of grid points and computational work. We allow for anisotropic (non-uniform) refinement by recursive bisection and show how to construct, manage and load balance such grids efficiently. In our numerical examples, we use finite difference schemes to discretize the PDEs. In the adaptive case we show how a stable discretization can be constructed using SBP-SAT operators. However, our adaptive mesh framework is general and other methods of discretization are viable. For integration in time, we implement exponential integrators based on the Lanczos/Arnoldi iterative schemes for eigenvalue approximations. Using adaptive time stepping and a truncated Magnus expansion, we attain high levels of accuracy in the solution at low computational cost. We further investigate alternative implementations of the Lanczos algorithm with reduced communication costs. As an example application problem, we have considered the time-dependent Schrödinger equation (TDSE). We present solvers and results for the solution of the TDSE on equidistant as well as adaptively refined Cartesian grids. / eSSENCE

L'influence des transferts radiatifs sur les écoulements de convection naturelle en cavités habitables est étudié numériquement en régimes turbulents. L'étude considère des approches DNS et LES pour le problème de convection et une méthode des ordonnées discrètes (MOD) pour la résolution du problème radiatif combinée au modèle de gaz réel SLW. La configuration étudiée est basée sur une cavité différentiellement chauffée expérimentale en air située à l'institut PPRIME, de rapport d'aspect vertical 4, pour des nombres de Rayleigh allant de 1,5x109 à 1,2x1011. La première partie de l'étude se focalise sur les techniques de parallélisations hybrides MPI + OpenMP de la MOD. Les méthodes développées montrent une amélioration des performances de 13 à 1600% pour des niveaux d'hybridations élevés par rapport à la méthode classique de front d'onde. Puis, une étude du couplage convection-rayonnement surfacique est réalisée au travers d'une étude de sensibilité de l'écoulement vis-à-vis des émissivités de parois pour différentes valeurs du nombre de Rayleigh. Ensuite, le rayonnement volumique du gaz est ajouté, et son impact est évalué au travers d'une variation du taux d'humidité relative du mélange air sec/vapeur d'eau. Les résultats obtenus sont comparés aux cas d'une cavité convectivement adiabatique (i.e. flux convectif nul aux parois passives). Les transferts radiatifs ont pour effet de diminuer la stratification thermique centrale et d’augmenter la dynamique générale de l'écoulement. L'émissivité des parois passives pilote principalement la localisation de la transition laminaire-turbulente aux parois actives et la stratification centrale, tandis que le rayonnement de gaz ne semble impacter que les couches limites des parois horizontales. / The influence of radiative transfer on natural convection flows in cavities is studied numerically in turbulent regimes. The study considers DNS and LES approaches for the convection problem and discrete ordinate method (MOD) to solve the radiative problem combined with the SLW real gas model. The studied configuration is based on an experimental differentially heated cavity in air located at the Pprime Institut with a vertical aspect ratio of 4, for Rayleigh numbers ranging from 1,5x109 to 1,2x1011. The first part of the study focuses on hybrid MPI + OpenMP parallelization of the DOM. The methods developed show performance improvements of 13 to 1600% compared to the classical wavefront method. Then, a study of convection-wall radiation coupling is achieved through a flow sensitivity study to walls emissivities for different values of the Rayleigh number. Then, the gas radiation is added, and its impact is measured through a variation of the relative humidity of the dry air + steam. The results are compared to the case of a convectively adiabatic cavity (i.e. zero convective flux at the passive walls). Radiative transfers have the effect of reducing the central thermal stratification and increase the overall dynamics of the flow. The emissivity of the passive walls drives the location of the laminar-turbulent transition on the active walls and the central thermal stratification, while the gas radiation seems to impact the boundary layers of the horizontal walls.

Dense fluid-particulate systems are widely encountered in the pharmaceutical, energy, environmental and chemical processing industries. Prediction of the heat transfer characteristics of these systems is challenging. Use of a high fidelity Discrete Element Method (DEM) for particle scale simulations coupled to Computational Fluid Dynamics (CFD) requires large simulation times and limits application to small particulate systems.  The overall goal of this research is to develop and implement parallelization techniques which can be applied to large systems with O(105- 106) particles to investigate particle scale heat transfer in rotary kiln and fluidized bed environments. The strongly coupled CFD and DEM calculations are parallelized using the OpenMP paradigm which provides the flexibility needed for the multimodal parallelism encountered in fluid-particulate systems. The fluid calculation is parallelized using domain decomposition, whereas N-body decomposition is used for DEM. It is shown that OpenMP-CFD with the first touch policy, appropriate thread affinity and careful tuning scales as well as MPI up to 256 processors on a shared memory SGI Altix. To implement DEM in the OpenMP framework, ghost particle transfers between grid blocks, which consume a substantial amount of time in DEM, are eliminated by a suitable global mapping of the multi-block data structure. The global mapping together with enforcing perfect particle load balance across OpenMP threads results in computational times between 2-5 times faster than an equivalent MPI implementation. Heat transfer studies are conducted in a rotary kiln as well as in a fluidized bed equipped with a single horizontal tube heat exchanger. Two cases, one with mono-disperse 2 mm particles rotating at 20 RPM and another with a poly-disperse distribution ranging from 1-2.8 mm and rotating at 1 RPM are investigated. It is shown that heat transfer to the mono-disperse 2 mm particles is dominated by convective heat transfer from the thermal boundary layer that forms on the heated surface of the kiln. In the second case, during the first 24 seconds, the heat transfer to the particles is dominated by conduction to the larger particles that settle at the bottom of the kiln. The results compare reasonably well with experiments. In the fluidized bed, the highly energetic transitional flow and thermal field in the vicinity of the tube surface and the limits placed on the grid size by the volume-averaged nature of the governing equations result in gross under prediction of the heat transfer coefficient at the tube surface. It is shown that the inclusion of a subgrid stress model and the application of a LES wall function (WMLES) at the tube surface improves the prediction to within ± 20% of the experimental measurements. / Ph. D.

Understanding, predicting, and controlling electromagnetic field interactions on and between complex RF platforms requires high fidelity computational electromagnetic (CEM) simulation. The primary CEM tool within NASA is GEMINI, an integral equation based method-of-moments (MoM) code for frequency domain electromagnetic modeling. However, GEMINI is currently limited in the size and complexity of problems that can be effectively handled. To extend GEMINI’S CEM capabilities beyond those currently available, primary research is devoted to integrating the MFDlib library developed at the University of Kentucky with GEMINI for efficient filling, factorization, and solution of large electromagnetic problems formulated using integral equation methods. A secondary research project involves the hybrid parallelization of GEMINI for the efficient speedup of the impedance matrix filling process. This thesis discusses the research, development, and testing of the secondary research project on the High Performance Computing DLX Linux supercomputer cluster. Initial testing of GEMINI’s existing MPI parallelization establishes the benchmark for speedup and reveals performance issues subsequently solved by the NASA CEM Lab. Implementation of hybrid parallelization incorporates GEMINI’s existing course level MPI parallelization with Open MP fine level parallel threading. Simple and nested Open MP threading are compared. Final testing documents the improvements realized by hybrid parallelization.

Submitted by Fabio Sobreira Campos da Costa (fabio.sobreira@ufpe.br) on 2016-07-28T12:46:53Z No. of bitstreams: 2 license_rdf: 1232 bytes, checksum: 66e71c371cc565284e70f40736c94386 (MD5) tese_diogotc_final.pdf: 7303783 bytes, checksum: f9959e8bb63b91d247de9903c2484d35 (MD5) / Made available in DSpace on 2016-07-28T12:46:53Z (GMT). No. of bitstreams: 2 license_rdf: 1232 bytes, checksum: 66e71c371cc565284e70f40736c94386 (MD5) tese_diogotc_final.pdf: 7303783 bytes, checksum: f9959e8bb63b91d247de9903c2484d35 (MD5) Previous issue date: 2016-01-29 / Esta tese apresenta uma metodologia de paralelização híbrida aplicada ao Método dos Elementos Discretos (DEM - Discrete Element Method) que combina MPI e OpenMP com o intuito de melhoria de desempenho computacional. A metodologia utiliza estratégias de decomposição de domínio visando a distribuição do cálculo de modelos de larga escala em um cluster. A técnica proposta também particiona a carga de trabalho de cada subdomínio entre threads. Este procedimento adicional visa obter maiores desempenhos computacionais através do ajuste de utilização de mecanismos de troca de mensagens entre processos e paralelização por threads. O objetivo principal da técnica é reduzir os elevados tempos de comunicação entre processos em ambientes computacionais de memória compartilhada tais como os processadores modernos. A divisão de trabalho por threads emprega a curva de preenchimento de espaço de Hilbert (HSFC) visando a melhoria de localidade dos dados e evitando custos computacionais (overheads) resultantes de ordenações constantes para o vetor de partículas. As simulações numéricas apresentadas permitem avaliar os métodos de decomposição de domínio, técnicas de particionamento, mecanismos de controle de acesso à memória, dentre outros. Algoritmos distintos de particionamento e diferentes estratégias de solução paralela são abordados para ambientes computacionais de memória distribuída, compartilhada ou para um modelo híbrido que envolve os dois ambientes. A metodologia desenvolvida e a ferramenta computacional utilizada nas implementações realizadas, o software DEMOOP, fornecem recursos que podem ser aplicados em diversos problemas de engenharia envolvendo modelos de partículas em larga escala. Nesta tese alguns destes problemas são abordados, em especial aqueles relacionados com fluxo de partículas em rampas, em funis de descarga e em cenários reais de deslizamento de terra. Os resultados mostram que as estratégias de execução híbridas atingem, em geral, melhores desempenhos computacionais que aqueles que se baseiam unicamente em troca de mensagens. A técnica de paralelização híbrida desenvolvida também obtém um bom controle de balanço de carga entre threads. Os estudos de caso apresentados apresentam boa escalabilidade e eficiências paralelas. O método proposto permite uma execução configurável de modelos numéricos do DEM e introduz uma estratégia combinada que melhora localidade dos dados e um balanceamento de carga iterativo. / This thesis introduces a methodology of hybrid parallelization applied to the Discrete Element Method (DEM) that combines MPI and OpenMP to improve computational performance. The methodology uses domain decomposition strategies to distribute the computation of large-scale models in a cluster. It also partitions the workload of each subdomain among threads. This additional procedure aims to reach higher computational performance by adjusting the usage of message passing artifacts and threads. The main objective is to reduce the expensive communications between processes in computer resources of shared memory such as modern processors. The work division by threads employs Hilbert Space Filling Curves (HSFC) in order to improve data-locality and to avoid the overhead caused by the dynamical sorting of the particles array. Presented numerical simulations allow to evaluate several domain decomposition schemes, partitioning methods, mechanisms of memory access control, among others. The work investigate distinct schemes of parallel solution for both distributed and shared memory environments. The method and the computational tool employed, the software DEMOOP, provide applied resources for several engineering problems involving large scale particle models. Some of these problems are presented on this thesis, such as the particle flows that happen on inclined ramps, discharge hoppers and real scenarios of landslides. The results shows that the hybrid executions reach better computational performance than those based on message passing only, including a good control of load balancing among threads. Case studies present good scalability and parallel efficiencies. The proposed approach allows a configurable execution of numerical models and introduces a combined scheme that improves data-locality and an iterative workload balancing.

Ce travail concerne l’étude d’écoulements incompressibles soumis à la rotation avec un solveur haute précision dans des géométries semi-complexes. La technique numérique mise en œuvre combine des schémas compacts, une méthode de projection multi domaine directe et un traitement efficace de la singularité à l’axe basé sur des conditions de parité dans l’espace de Fourier. Le solveur a été parallélisé avec une approche hybride MPI-OpenMP pour réduire les temps de calcul. Dans un premier temps, les précisions spatiales et temporelles de la méthode numérique et la scalabilité du solveur ont été vérifiées. La capacité du solveur à traiter des écoulements plus complexes a été évaluée en considérant des écoulements de type éclatement tourbillonnaire et un écoulement turbulent en conduite cylindrique. Dans un second temps, plusieurs écoulements typiques des machines tournantes ont été étudiés. Le premier écoulement est un écoulement turbulent incompressible isotherme dans un étage simplifié d’un compresseur haute pression d’une turbine à gaz. Les simulations menées ont mises en évidence l’effet de la rotation sur l’écoulement, notamment sur les instabilités se développant le long des parois et sur les différentes structures cohérentes. Le second cas traité est un écoulement turbulent de jet impactant un disque en rotation avec un fort confinement et transfert thermique. Une attention particulière a été portée sur les champs hydrodynamiques et thermique le long du rotor. Enfin, une étude préliminaire d’un jet turbulent impactant un disque fixe d’épaisseur non nulle dans une configuration moins contrainte avec prise en compte du couplage conduction-convection a été réalisée. / This work deals with the study of rotating incompressible flows with a high accurate solver in semi complex geometries. The numerical method used in this work combines compact schemes, a direct multidomain projection method and an efficient axis treatment based on parity conditions in Fourier space. The use of cylindrical coordinates introduces this mathematical singularity. In order to reduce the calculation time, the solver was parallelized with an hybrid MPI-OpenMP parallelization. First, the spatial and temporal accuracies of the numerical method and the scalability of the solver were checked. Then, the capability of the algorithm to deal with more complex flows was verified. Vortex breakdown flows and turbulent pipe flow were studied. In the second step, typical flows of turbomachineries and rotating systems were considered. The first flow was an incompressible isothermal turbulent flow in a high pressure compressor of gas turbine. The different simulations highlighted the rotation effects on the flows, especially on the instabilities appearing along the walls and the coherent structures. The second considered flow was a turbulent impinging jet on a rotating disk with heat transfer in a small aspect ratio cavity. The hydrodynamic fields and heat transfer near the rotor were analyzed in detail. Finally, a preliminary investigation of an impinging jet on a non-rotating disk in a larger aspect ratio cavity with a coupling between conduction and convection transfer was carried out.