HOMME on the IBM BlueGene/L

Principal Contributors

Collaborators & Participants

Project Overview

NCAR researchers, funded in part by the Department of Energy’s Climate Change Prediction Program, have built a scalable and efficient spectral-element-based atmospheric dynamical core using the Computational Science Section’s High Order Method Modeling Environment (HOMME). In order for this to be a useful tool for atmospheric scientists it is necessary to couple this core to physics packages employed by the community.

The physics of cloud formation is generally simulated rather crudely using phenomenological parameterizations. The dream of modelers is the direct numerical simulation of cloud processes on a global scale. Unfortunately, this requires an increase in computational power of approximately six orders of magnitude over what is currently available. A promising alternative to improve the simulation of cloud processes in climate models is a compromise technique called Cloud Resolving Convective Parameterization (CRCP, also known as Super-Parameterization). The cost of this approach is two to three orders of magnitude more expensive than traditional parameteriz ation techniques. However, with the advent of BlueGene/L this is now tractable. We have built a super-parameterization package and work is underway to couple this to HOMME. The result will be an atmospheric model capable of exploiting BG/L’s scalability and computational power to realize practical and scientifically useful integration rates for super-parameterized climate simulation.

First Step Moist Held-Suarez

The moist Held-Suarez test of Galewski, Sobel and Held (2004) extends the standard (dry) Held-Suarez (1994) test of the hydrostatic primitive equations by introducing a moisture tracer and simplified physics. It is the next logical test for a dynamical core beyond dry dynamics. Moisture is injected into the system at a constant rate from the surface according to a prescribed zonal profile, is advected as a passive tracer by the model, and precipitated from the system when the saturation point is exceeded. (Visit the IBM booth to view the full movie which displays the water vapor mixing ratio and temperature field evolution during a 1200 day Held-Suarez integration. Small scale flow features such as breaking baroclinic waves are clearly evident with transport of moisture from the tropics to mid-latitudes.)


The model is written in F90 and has three components: dynamics, physics and a physics/dynamics coupler. The dynamics has been run on the BG/L systems at Watson and Rochester on up to 7776 processors using one processor per node and only using one of the floating-point pipelines. The peak performance expected from a Blue Gene/L processor for the runs is then 1.4 Gflops/s. The sustained Mflops per second per processor is shown below for meshes with 1536 (16×16x6), 1944 (18×18x6), and 7776 (36×36x6) elements on the sphere and with either 20 or 40 layers in the vertical direction. The total amount of work (total flop count to solve the system) was kept constant while the processor count was increased (strong scaling), until, in the 1944 and 7776 mesh runs, there was only one element per processor. The flat part of the graph shows the scaling region for each problem size, which would be the region where one would make the most effective use of the BG/L processor. The average sustained performance in the scaling region for the Dry Held-Suarez code is ~200-250 MF/s/processor (out to 7776 processors) while for the Moist Held-Suarez code it is ~300-400 MF/s/processor (out to 1944 processors) out of a peak of 1400 MF/s/processor.

More information about this project can be found at:

Related Papers and Files

  1. G. Bhanot, J. M. Dennis, J. Edwards, W. Grabowski, M. Gupta, K. Jordan, R. D. Loft, J. Sexton, A. St-Cyr, S. J. Thomas, H. M. Tufo, T. Voran, R. Walkup, and A. A. Wyszogrodski. "Early experiences with the 360TF IBM Blue Gene/L platform". International Journal of Computational Methods (IJCM), 5(2):237-253, 2008.
  2. J.M. Dennis, M. Levy, R.D. Nair, H.M. Tufo, and T. Voran, "Towards an Efficient and Scalable Discontinuous Galerkin Atmospheric Model", proceedings of the 19th IEEE International Parallel & Distributed Processing Symposium, 2005.
  3. G. Almasi, G. Bhanot, D. Chen, M. Eleftheriou, B. Fitch, A. Gara, R. Germain, M. Gupta, M.C. Pitman, A. Rayshubskiy, J. Sexton, F. Suits, P. Vranas, B. Walkup, T.J.C. Ward, Y. Zhestkov, A. Curioni, W. Curioni, C. Archer, J.E. Moreira, R. Loft, H.M. Tufo, T. Voran, K. Riley, "Early Experience with Scientific Applications on the BlueGene/L Supercomputer", Euro-Par 2005, Lisboa, Portugal, September 2005.