Application Performance on the Tri-Lab Linux Capacity Cluster - TLCC

Application Performance on the Tri-Lab Linux Capacity Cluster - TLCC

Mahesh Rajan (Sandia National Laboratory, USA), Douglas Doerfler (Sandia National Laboratory, USA), Courtenay T. Vaughan (Sandia National Laboratory, USA), Marcus Epperson (Sandia National Laboratory, USA) and Jeff Ogden (Sandia National Laboratory, USA)
Copyright: © 2010 |Pages: 17
DOI: 10.4018/jdst.2010040102


In a recent acquisition by DOE/NNSA several large capacity computing clusters called TLCC have been installed at the DOE labs: SNL, LANL and LLNL. TLCC architecture with ccNUMA, multi-socket, multi-core nodes, and InfiniBand interconnect, is representative of the trend in HPC architectures. This paper examines application performance on TLCC contrasting them with Red Storm/Cray XT4. TLCC and Red Storm share similar AMD processors and memory DIMMs. Red Storm however has single socket nodes and custom interconnect. Micro-benchmarks and performance analysis tools help understand the causes for the observed performance differences. Control of processor and memory affinity on TLCC with the numactl utility is shown to result in significant performance gains and is essential to attenuate the detrimental impact of OS interference and cache-coherency overhead. While previous studies have investigated impact of affinity control mostly in the context of small SMP systems, the focus of this paper is on highly parallel MPI applications.
Article Preview


This Paper investigates application performance on the Tri-lab Linux Capacity Cluster (TLCC) using a variety of applications and compute kernels. This is motivated by the size of the NNSA’s Advanced Simulation and Computing (ASC) TLCC program, the initial procurement of which consisted of 21 “Scalable Units” (SUs) spread out over eight clusters: three for Lawrence Livermore (8 SU, 2 SU, and 1 SU systems), two for Los Alamos (two 4 SU systems), and three for Sandia (three 2 SU systems, one of which will be housed at Sandia Livermore). The systems were designed around a single hardware design point, called a “Scalable Unit” (SU) to minimize costs. Multiple clusters of varying sizes are in service based on the scalable unit module.

Each SU consists of 144 four-socket, quad-core AMD Opteron (Barcelona) nodes, using 4x DDR InfiniBand as the high speed interconnect. Each node has 32 GB of 667 MHz DDR2 memory resulting in 8GB per processor socket. The systems also use a common software stack, based on the ASC Program Tripod Operating System Software (TOSS), a Tri-Lab packaging of the CHAOS/SLURM environment that has been in production on systems for several years ( The basic elements of TOSS consist of Red Hat Enterprise Linux (RHEL5U2), the OpenFabrics Enterprise Distribution InfiniBand stack, MVAPICH and Open-MPI, and the MOAB/SLURM resource manager.

A common set of FORTRAN, C and C++ development tools are also provided. The site-specific software components include the parallel file system as well as the RAS and system monitoring software.

This paper examines application performance on TLCC and contrasts them with Red Storm/Cray XT4 performance. Comparisons of TLCC performance, to the partition of Red Storm with quad-core single socket nodes, are useful because of the processor and memory DIMM similarities. This is fortuitous as it provides a unique opportunity to evaluate the impact of node memory architecture, node interconnect architecture, and the operating system on performance. However differences in MPI libraries and possible choices of compiler must also be considered. While processor speed for both is 2.2 GHz, the Red Storm quad-core nodes have 800 MHz DDR2 DIMMs, a higher memory speed than on the TLCC with 667 MHz DIMMs. However, a smaller development system called Red Storm Qualification system (RSQUAL) with 667 MHz memory DIMMs has been employed for a few closer comparisons to TLCC.

This effort led to the following findings that are of interest in extracting the best performance from systems like TLCC, with multi-socket NUMA nodes. It further provides analysis of possible limitations to scaling of such clusters through the comparisons to Red Storm.

Complete Article List

Search this Journal:
Open Access Articles
Volume 10: 4 Issues (2019): Forthcoming, Available for Pre-Order
Volume 9: 4 Issues (2018): 2 Released, 2 Forthcoming
Volume 8: 4 Issues (2017)
Volume 7: 4 Issues (2016)
Volume 6: 4 Issues (2015)
Volume 5: 4 Issues (2014)
Volume 4: 4 Issues (2013)
Volume 3: 4 Issues (2012)
Volume 2: 4 Issues (2011)
Volume 1: 4 Issues (2010)
View Complete Journal Contents Listing