ARSC HPC Users' Newsletter 400, February 6, 2009



Cray XT5 Performance Workshop Scheduled

ARSC is pleased to host John Levesque and Luiz DeRose of Cray for a three-day hands-on workshop on Tuesday, April 7 through Thursday, April 9. John and Luiz will be going over node architecture and optimization, message passing optimization and tools and techniques for understanding application performance in the mornings. Afternoon sessions will be hands-on, with users applying principles learned in the morning to their own codes. John and Luiz will also be available on Friday morning for further consultation with users.

This is a great opportunity for users or potential users of Pingo and other XT5 systems to bring their applications and sit down with these two performance experts to better understand and improve the performance of their code for the XT5 architecture. To register, see the ARSC training page at

or contact Tom Logan ( for more information.

Spring Training Lineup

In addition to the Cray Optimization and Performance Workshop in April, this spring ARSC will offer a new set of training classes in support of Pingo including Introduction to Pingo, Understanding Memory Caches, and Introduction to CrayPat Performance Tools as well as other topics.

The entire spring training list:

    20-Feb  2-4pm   Intro to Pingo                                  Taught by: Tom Logan 
    6-Mar   2-4pm   Introduction to Paraview Visualization Package  Taught by: Patrick Webb
    27-Mar  2-4pm   Understanding Memory Caches                     Taught by: Ed Kornkven  
    3-Apr   2-4pm   Introduction to CrayPat Performance Tools       Taught by: Don Bahls
    4/7-4/9 9am-5pm Cray Optimization and Performance Training      Taught by: John Levesque and Luiz DeRose (Cray)
    24-Apr  2-5pm   C interface to NetCDF                           Taught by: Anton Kulchitsky

Please note that the Cray Training (4/7-4/9) and the C interface to NetCDF classes both will have hands-on portions, and thus, will require preregistration through the ARSC website.

Registration and additional information, including abstracts, will be available soon at

For more information, contact Tom Logan at .


Working with Long Term Storage

[ By Don Bahls ]

When copying files from long term storage (i.e. $ARCHIVE_HOME), it's a good idea to pre-stage files using the "batch_stage" command. The "batch_stage" command brings files online in an orderly fashion and can significantly reduce the time it take to copy files from $ARCHIVE_HOME. This is especially true if the files were created at around the same time.

Here are a few examples:

1) Stage all files in a directory (without traversing subdirectories)

   % batch_stage $ARCHIVE/input/*
   % cp $ARCHIVE/input/* $WORKDIR/input

2) Stage files in a directory and also subdirectories recursively

   % find $ARCHIVE/input 
 batch_stage -i 
   % cp -r $ARCHIVE/input/* $WORKDIR/input

3) The "batch_stage" command can be used PBS jobs using the "data" queue.

   #PBS -q data
   #PBS -j oe
   #PBS -l walltime=1:00:00
   #PBS -l select=1:ncpus=1
   # create a directory to store input files.
   mkdir -p $WORKDIR/input 
   # bring data files back online and transfer to $WORKDIR
   batch_stage $ARCHIVE/input/*
   cp $ARCHIVE/input/* $WORKDIR/input
   # submit a compute job which uses input files.
   qsub compute.pbs 
Please note that "batch_stage" is intended for optimizing retrieval of a "reasonable" number of files from tape storage to disk for immediate use. Attempting to stage an entire large $ARCHIVE unnecessarily, for example, could cause problems. For a list of other best practices for ARSC long term storage, see:


$PET_HOME software

The $PET_HOME directory on pingo and midnight contains a number of popular scientific libraries. Here's a current list of packages on each system:

   Package Name         Version
   -----------------    --------
   ARPACK               96
   CMake                2.6.1
   FFTW-2               2.1.5
   FFTW-3               3.1.3
   GotoBLAS             1.26
   GSL                  1.12
   HDF5                 1.8.1
   MatlabMPI            1.2
   netCDF               4
   Octave               3.0.3
   PAPI                 3.6.2
   PETSc                2.3.3
   ScaLAPACK            1.8.0
   Scalasca             1
   SPRNG                3
   SuperLU              3
   TAU                  2.18
   Trilinos             9.0.1
   Valgrind             3.4.0

Recent changes to packages in $PET_HOME are listed in the PET_HOME news item:

Pingo: Link no longer available

Midnight: Link no longer available


Newsletter News: Don Bahls Turning Over Reins to Craig Stephenson

[ By Ed Kornkven ]

This issue will be the last edited by my colleague Don Bahls. The good news is that ARSC Consultant Craig Stephenson will be stepping in to serve with me as co-editor. Don joined Tom Baring as co-editor way back on issue #301 so this issue marks his centennial. Don has been a major contributor of articles as well as editor, has taught me the editing ropes and has been an all-around catalyst for the publication. Thankfully, he plans to continue sending those articles and will still be around to keep Craig and me on track. Thanks Don. To Craig: Welcome! We will give a little more introduction to Craig next time.


Quick-Tip Q & A

A:[[ For some reason the "make clean" rule in my Makefile is no longer
  [[ working.  It's a really simple Makefile. I'm baffled as to why it 
  [[ suddenly quit working.
  [[ Here it is:
  [[ bash-3.2$ more Makefile
  [[ CXX=CC
  [[ %.o : %.cpp
  [[        $(CXX) $< -c
  [[ a.out: main.o lirp.o
  [[        $(CXX) main.o lirp.o
  [[ clean:
  [[        rm -f a.out main.o lirp.o
  [[ When I run "make clean", make reports:
  [[ bash-3.2$ make clean
  [[ make: `clean' is up to date.
  [[ An "ls" shows that it didn't work!
  [[ -bash-3.2$ ls
  [[ Makefile        clean           lirp.cpp        lirp.o          
  [[ main.cpp        main.o          a.out
  [[ What's going on here?

# Thanks to Jay Cliburn, Sean Ziegeler, Jed Brown, Chris Petrich
# Bracy Elton for helping out this Makefile.  They recognized the
# problem and provided solutions.  We'll use Sean's words to summarize
# their responses...

The presence of a file named "clean" is causing make to consider
it a target to create.  If you don't need the "clean" file, delete
it.  If you do need the file for something, you can add the following
line to the Makefile to prevent it from looking at the file:

.PHONY: clean

# ...Bracy's example of the final Makefile...


%.o : %.cpp
    $(CXX) $< -c

a.out: main.o lirp.o
    $(CXX) main.o lirp.o

.PHONY: clean

    rm -f a.out main.o lirp.o

# ... and Chris' pointer to documentation


# Finally, Alec Bennett found this method that works with GNU make.

Not the most elegant solution, but the -B flag (or --always-make) will
force an unconditional make of the target, and cause make to ignore the
file check for "clean":

make -B clean

Q: I am writing a script, call it A, that calls another script B.
I want to maintain both scripts in the same directory but I need
to be able to call A from anywhere.  The problem I'm having is that
A needs to be able to refer to the directory it is stored at in
order to find B.  I tried `pwd` as the path to B but that gives me
the directory I was in when I called A, not the directory that A
(and B) are stored in.  Is it possible to do this with a shell
script?  I guess I could use Perl if Perl has a way of doing it.
Python?  Help!

[[ Answers, Questions, and Tips Graciously Accepted ]]

Current Editors:
Ed Kornkven ARSC HPC Specialist ph: 907-450-8669
Kate Hedstrom ARSC Oceanographic Specialist ph: 907-450-8678
Arctic Region Supercomputing Center
University of Alaska Fairbanks
PO Box 756020
Fairbanks AK 99775-6020
E-mail Subscriptions: Archives:
    Back issues of the ASCII e-mail edition of the ARSC T3D/T3E/HPC Users' Newsletter are available by request. Please contact the editors.
Back to Top