Direct Numerical Simulation for Homogeneous Charge Compression Auto-Ignition Using Scalable Parallel I/O

Philippe Pébay
Sandia Combustion Research Facility

Direct Numerical Simulation (DNS) is a mature and productive research tool in combustion science used to provide high-fidelity, computer-based observations of the micro-physics found in turbulent, reacting flows. DNS is also a unique tool for the development and validation of reduced model descriptions used in macro-scale simulations of engineering-level systems. Because of its high demand for computational power, current state-of-the-art (terascale) DNS remains limited to small computational domains (i.e. small Reynolds numbers) and to simplified problems corresponding to adiabatic, non-sooting, gaseous flames in simple geometries.

S3D is an existing Sandia, Livermore (principal investigator: Dr. Jackie Chen) DNS compressible Navier-Stokes solver coupled with an integrator for detailed chemistry. It is based on high-order finite differencing, high-order explicit time integration, and conventional structured meshing. Figure 1 shows the example of the scalar dissipation rate in a turbulent jet flame as recently computed by S3D on Sandia-Livermore’s latest 128-node research cluster, Catalyst (dual 3.0 GHz Xeon processor nodes, 10 Gbps InfiniBand and gigabit Ethernet interconnect fabrics). The development of S3D has been supported by the U.S Department of Energy (DOE), Office of Sciences, Basic Energy Sciences BES) program. DNS databases generated by S3D are currently being used by the ASCI Alliance program to test and validate turbulent combustion sub models for large-eddy simulations of accidental fires and explosions (C-SAFE, University of Utah).

The first production run performed on Catalyst is targeted at better understanding of inhomogeneous auto-ignition at constant volume. A better understanding of the ignition system will enable better control strategies for a mode of combustion being considered for compression ignition automotive engines, known as homogeneous charge compression ignition combustion (HCCI). More precisely, the objective of this particular numerical simulation was to understand the influence of temperature inhomogeneities on the evolution of the different ignition modes of combustion, namely deflagration, spontaneous ignition front propagation and detonation. Using DNS with complex H2/air chemistry, this study focused on the effect of the initial temperature distribution on the subsequent formation and development of ignition kernels, and the overall rate of heat-release.

A typical time-step in a 2D HCCI production-run on Catalyst takes about 3 seconds to compute, and generates a 600 MB restart and visualization file. Such a time-step represents roughly 5 nanoseconds of physical combustion. Ideally, one would like to save data generated from all time-steps in order to provide accurate analysis in real-time and/or post processing, particularly for transient, intermittent events such as ignition. Unfortunately, it takes several seconds to write the 600MB of data concurrently from the 128 compute nodes using the currently deployed NFS I/O subsystem, thus, the next compute time-step is placed on hold, thereby causing significant slowdown to the simulation run that typically consists of 1,000,000 or more time-steps. As such, production runs today are constrained to save data at predetermined intervals, causing the following adverse effects:

• A potential loss of critical data (e.g., extrema, sharp quantitative changes, etc.);
• More complicated data analysis, because post processing files are not computed on the fly, but need to be generated prior to analysis;
• Difficulties in real-time code-steering (e.g. IO frequency, time-step, data sub-setting of interesting regions).

Therefore, a scalable parallel I/O subsystem is critical in high-performance, clustered computing environments in order to improve high-impact science applications such as HCCI. We will demonstrate the acceleration of a HCCI simulation-run using the Panasas Direct Flow parallel I/O subsystem located at the SC04 StorCloud booth. The 48 compute nodes at the ASC booth use InfiniBand as its messaging fabric, and Gigabit Ethernet as its I/O infrastructure. The Panasas Direct Flow at the StorCloud booth will be configured to support 20 Gbps of aggregate I/O bandwidth to the ASC booth.

Team Members:
Philippe Pébay
Jacqueline. H. Chen
Helen Y. Chen
E. R. Hawkes

Industry Partners:
Panasas
Cisco
Appro
Force 10 Networks