ARSC HPC Users' Newsletter 335, March 3, 2006

X1: Capturing missing stdout

[ This article is inspired by the PET Consolidated X1 FAQ ]

The Cray X1 buffers stdout, which can lead to missing information in your output files when the program crashes before the buffer is flushed. You could add explicit flush statements to your code to ensure stdout is written, however the assign command provides a much more elegant way accomplishing this task. Since stderr is not buffered, we can simply direct stdout (unit 6) to stderr to ensure that output is written. NOTE: there could be considerable performance impact doing this, so this technique should only be used for debugging purposes.

First, set the file to store the assign command to:

sh/ksh/bash syntax:

  export FILENV=./myassign

csh/tcsh syntax:

  setenv FILENV ./myassign

Next, use the assign command to direct stdout to stderr:

  assign -D stderr u:6

Finally, run the executable as you normally would.


  aprun -n 4 ./a.out

If you redirect stdout to a file, be sure to redirect stderr as well:

e.g. (sh/ksh/bash):

  aprun -n 4 ./a.out 2> output

or (csh/tcsh):

  aprun -n 4 ./a.out >& output

To remove an assign statement from the file, use the following:

  assign -R

Here's a PBS script which ties everything together.

  #PBS -q default
  #PBS -l mppe=4
  #PBS -l walltime=30:00
  #PBS -j oe
  #PBS -S /bin/ksh 
  export FILENV=./assign.debug
  assign -D stderr u:6
  aprun -n 4 ./a.out

Since the environment variable FILENV is set within the script, there is no need to remove the assign statement stored in the file unless you are using the same file to store other assign statements.

NOTE: this trick doesn't work with C based codes.

The PET X1 Consolidated FAQ is available at:

To get to the Consolidated X1 FAQ:

  1. Select the "CE" link under the "Cross-Cutting Areas" heading (in the "Menu" frame on the left side of the window), then
  2. Then the "Papers/Pubs" link (in the "CE" frame), and finally
  3. the "Cray X1 Frequently Asked Questions" link (in the list of papers).

ARSC also maintains a direct link to the FAQ from the klondike how to page:

Three-part GIS sessions March 27th ,29th, and 31st

Title:    Workshop of Online 3D Terrain Visualization and GIS tools
Location: WRRB 010 Conference Room
Dates:    Monday,  March 27
          Wednesday, March 29
          Friday, March 31
Time:     noon - 3:00 pm (lunch provided)


This workshop is intended to create earth science, earth science education/outreach, and earth science data access applications using online 3D terrain visualization and GIS tools such as EarthSLOT, Google Earth, World Wind and GeoFusion. Your own Windows laptop computer is required (the needs of participants needing Macintosh access will be covered, but development will occur on PCs for consistency). The format of the workshop will be tailored to the nature of the participants' projects.

To attend, please send an email to matt.nolan (at) that outlines the nature of your project, the types of data you hope to use, the target audience, what you would like to accomplish in the workshop, and how you envision using these tools in the future. We will then work with you before the workshop to refine your ideas, ensure your data are in easily ingestible formats, ensure that your computer has all of the necessary software, and suggest some advance reading if appropriate.

The workshop is limited to 10 participants. If the number of applicants exceed 10, preference will be given to graduate students with a clear vision of what they want to accomplish and who have all the necessary data in hand to finish a pilot project by the end of the workshop. Additional workshops will be held as demand warrants, and it is hope that the participants of this first workshop will serve as the nucleus of a UAF community built around use and advancement of these tools.

Hardware Upgrades

In the last few months, ARSC has added new hardware to both iceflyer and nelchina. These additions have increased the available computation resources of both systems by 2 times or more.

Iceflyer Upgrades

4- p575 nodes have been added to iceflyer. Each of these nodes has 8- 1.9 Ghz Power5 processors and 16 GB of RAM. Currently the Power5 nodes are running a variety of operating systems including: AIX 5.2, AIX 5.3, and SuSE Linux Enterprise Server (SLES) 9.2.

Both AIX 5.3 and SLES 9.2 can exploit the SMT (simultaneous multithreading) support available within the Power5 architecture. This functionality allows the operating system to schedule two threads per processor. Preliminary tests done by ARSC staff show that SMT reduces run time for some OpenMP applications over 10% when doubling the number of threads (i.e. setting OMP_NUM_THREADS=16 rather than 8).

These nodes are now open to all iceflyer users via the "p5" class.

For more information on the power5 nodes iceflyer users should see:

news power5

Nelchina Upgrades

During the past two weeks ARSC and Cray staff have been busy adding three more chassis to the Cray XD1, nelchina. Each of the new chassis has 6 nodes with 2 dual core AMD Opteron processors per node.

Once the upgrade is complete, nelchina will have 108 PEs in the following configuration.

  6-  nodes with 2 AMD Opteron 250 Processors (Single Core) with FPGAs
  12- nodes with 2 AMD Opteron 250 Processors (Single Core)
  18- nodes with 2 AMD Opteron 280 Processors (Dual Core)

The upgrade also includes a change to the fat tree network topology which improves performance for all-to-one, one-to-all, or all-to-all communications by providing a direct link between each node.

Users interested in using nelchina should contact the ARSC help desk at

Quick-Tip Q & A

A:[[ I want stdout and stderr from a command to go to a file AND to my 
  [[ screen.  These commands, 
  [[   CSH:
  [[     ./prog >&  prog.out
  [[   KSH:
  [[     ./prog > prog.out 2>&1
  [[ would merge stdout and stderr to one file.  But, as I said, I'll
  [[ like to see it on the screen at the same time.

We got a ton of responses to this, but, as professors lament at finals
time, several people didn't read the question too carefully. The real
problem here was dealing with stderr (not just stdout and "tee").

Thanks to all: Chris Young, Brad Chamberlain, Harper Simmons, Steve
Finn, Rich Griswold, Greg Newby, Kurt Carlson, Hank Happ, Lorin
Hochstein, and Martin Luthi.

# From Brad Chamberlain, here's a csh solution: 

Try the following (CSH format):

& tee prog.out

"tee" is a program that feeds its standard input to a filename and to
stdout. Thus, what we're doing here is piping the standard (
) and error
(&) output of prog into tee, which is piping it to prog.out and to stdout.

# From Kurt Carlson, ksh: 

    ./prog 2>&1 
 tee prog.out 

# Lorin Hochstein reminds us of another possibility, tail -f.  A problem
# with this, tail -f doesn't terminate with the program, which could be
# misleading.

  Terminal 1:
    touch prog.out
    tail -f prog.out

  Terminal 2:
    ./prog >& prog.out

The "tail" programs allows a user to view the file on Terminal 1 as data
is being appended. (When the file is done, Ctl-C is required to
terminate the tail program.)

Q: On some systems, when I use vi/vim to open a file and scroll down to
   a particular line and then exit, vi/vim remembers the position within 
   the file. The next time I open the file it returns me to that 

   I can't get this feature working everywhere. Do you know how one 
   enables this?

[[ Answers, Questions, and Tips Graciously Accepted ]]

Current Editors:
Ed Kornkven ARSC HPC Specialist ph: 907-450-8669
Kate Hedstrom ARSC Oceanographic Specialist ph: 907-450-8678
Arctic Region Supercomputing Center
University of Alaska Fairbanks
PO Box 756020
Fairbanks AK 99775-6020
E-mail Subscriptions: Archives:
    Back issues of the ASCII e-mail edition of the ARSC T3D/T3E/HPC Users' Newsletter are available by request. Please contact the editors.
Back to Top