Article Preview
TopIntroduction
This Paper investigates application performance on the Tri-lab Linux Capacity Cluster (TLCC) using a variety of applications and compute kernels. This is motivated by the size of the NNSA’s Advanced Simulation and Computing (ASC) TLCC program, the initial procurement of which consisted of 21 “Scalable Units” (SUs) spread out over eight clusters: three for Lawrence Livermore (8 SU, 2 SU, and 1 SU systems), two for Los Alamos (two 4 SU systems), and three for Sandia (three 2 SU systems, one of which will be housed at Sandia Livermore). The systems were designed around a single hardware design point, called a “Scalable Unit” (SU) to minimize costs. Multiple clusters of varying sizes are in service based on the scalable unit module.
Each SU consists of 144 four-socket, quad-core AMD Opteron (Barcelona) nodes, using 4x DDR InfiniBand as the high speed interconnect. Each node has 32 GB of 667 MHz DDR2 memory resulting in 8GB per processor socket. The systems also use a common software stack, based on the ASC Program Tripod Operating System Software (TOSS), a Tri-Lab packaging of the CHAOS/SLURM environment that has been in production on systems for several years (https://computing.llnl.gov/linux/projects.html#chaos). The basic elements of TOSS consist of Red Hat Enterprise Linux (RHEL5U2), the OpenFabrics Enterprise Distribution InfiniBand stack, MVAPICH and Open-MPI, and the MOAB/SLURM resource manager.
A common set of FORTRAN, C and C++ development tools are also provided. The site-specific software components include the parallel file system as well as the RAS and system monitoring software.
This paper examines application performance on TLCC and contrasts them with Red Storm/Cray XT4 performance. Comparisons of TLCC performance, to the partition of Red Storm with quad-core single socket nodes, are useful because of the processor and memory DIMM similarities. This is fortuitous as it provides a unique opportunity to evaluate the impact of node memory architecture, node interconnect architecture, and the operating system on performance. However differences in MPI libraries and possible choices of compiler must also be considered. While processor speed for both is 2.2 GHz, the Red Storm quad-core nodes have 800 MHz DDR2 DIMMs, a higher memory speed than on the TLCC with 667 MHz DIMMs. However, a smaller development system called Red Storm Qualification system (RSQUAL) with 667 MHz memory DIMMs has been employed for a few closer comparisons to TLCC.
This effort led to the following findings that are of interest in extracting the best performance from systems like TLCC, with multi-socket NUMA nodes. It further provides analysis of possible limitations to scaling of such clusters through the comparisons to Red Storm.