ARSC T3E Users' Newsletter 172, June 30, 1999

Yukon Default Programming Environment Switched

On June 29th, both yukon and chilkoot's default programming environments were switched from PE 3.0 to 3.2. You may still access the old PE by executing the command:

module switch PrgEnv PrgEnv.old

To help you identify changes since PE 3.0, the section headings and selected text of the release notes for both PE 3.1 and PE 3.2 are included below. The latter is available in its entirety at:

Programming Environment Releases Overview For PE 3.1

Chapter 3. New Features

  • 3.1 Cray C/C++ Features
    • 3.1.1 New C++ Language Features
    • 3.1.2 C Comments
    • 3.1.3 Multibyte Characters Allowed in Source
    • 3.1.4 New Data Types for C and C++
    • long long and unsigned long long
    • long double complex
    • 3.1.5 Bit Matrix Multiply Intrinsic Functions for UNICOS/mk Systems
    • 3.1.6 Vector Versions of Math Library Functions Added to libmfastv on UNICOS/mk Systems
    • 3.1.7 float Math Library Functions on UNICOS Systems
    • 3.1.8 float complex Math Library Functions on UNICOS Systems
    • 3.1.9 long double Math Library Functions on UNICOS/mk Systems
    • 3.1.10 long double complex Math Library Functions on UNICOS/mk and UNICOS Systems
    • 3.1.11 long long and unsigned long long Math Library Functions on UNICOS and UNICOS/mk Systems
  • 3.2 CF90 Features
    • 3.2.1 Bit Matrix Multiply Intrinsics for UNICOS/mk Systems
    • 3.2.2 Vector Intrinsics Added to libmfastv on UNICOS/mk Systems
    • 3.2.3 Co-array Support on UNICOS/mk Systems
    • 3.2.4 Directives for Array Syntax Statements
    • 3.2.5 New Compiler Option for Nondynamic (Post Mortem) Debug
  • 3.3 CrayTools Features
    • 3.3.1 Wrapper Routines for PAT
    • 3.3.2 GUI for PAT
    • 3.3.3 Use of PAT with Vampir
    • 3.3.4 New debugview Options
    • 3.3.5 Common Block Printing Change for debugview
    • 3.3.7 Tool Support Updated

Chapter 4. Compatibilities and Differences

  • 4.1 perfdmp Utility Incompatibility
  • 4.2 MPP Apprentice Tool No Longer Available for UNICOS Systems
  • 4.3 atchop Utility Retired

Programming Environment Releases Overview For PE 3.2

Chapter 3. New Features

This chapter lists the new features in CF90, Cray C/C++, and CrayTools for this release. The Programming Environment 3.2 releases add compiler support for Fortran 95. The features described in this chapter are supported on CRAY T3E systems running UNICOS/mk release 2.0.4 and later and Cray PVP systems running UNICOS release 10.0 and later.

  • 3.1 CF90 Features
    • 3.1.1 FFIO Layers
    • 3.1.2 NAMELIST and List-Directed I/O Enhancements
    • 3.1.3 Edit Descriptor Changes
    • 3.1.4 Global Semantics
    • 3.1.5 Nested WHERE Construct Support
    • 3.1.6 FORALL Construct Support
    • 3.1.7 Loopmark Enhancements and Optimization Messages
    • 3.1.8 New Inlining Capability
    • 3.1.9 Default Initialization for Fortran 95 Default initialization can now be declared for each component in a derived type definition.
    • 3.1.10 Enhanced CEILING and FLOOR Intrinsics for Fortran 95
    • 3.1.11 NULL() Intrinsic
    • 3.1.12 CPU_TIME Intrinsic
    • 3.1.13 Generic Name on END INTERFACE for Fortran 95
    • 3.1.14 Obsolescent Feature Detection for Fortran 95
    • 3.1.15 Additional Run-Time Check Enhancements
    • 3.1.16 -M Option for Log Warning Suppression
    • 3.1.17 ftpp Replaced with Integrated Preprocessor
    • 3.1.18 Support for FORM="SYSTEM"
    • 3.1.19 Fortran PXF Routines
    • 3.1.20 Array Reshaping Option (Performance Feature) The compiler command line option, -O reshape [=array [,array,...]], makes the rightmost dimension of the specified array the leftmost dimension, both in declarations and references. Remaining dimensions are shifted one position to the right. The -O reshape option, with an array unspecified, performs the transformation on all arrays with a constant rightmost dimension extent of less than 16. A message is printed for each array that is transformed.
    • 3.1.21 Option to Create .mod File (Performance Feature)
  • 3.2 Cray C/C++ Features
    • 3.2.1 #pragma nounroll Directive
    • 3.2.2 Loopmark Enhancements and Optimization Messages
  • 3.3 CrayTools Features
    • 3.3.1 Support for long long and unsigned long long Types
    • 3.3.2 TotalView Line Mode Enhancements
    • 3.3.3 alias Command in TotalView

Chapter 4. Compatibilities and Differences

This chapter provides information on compatibilities and differences that you should consider when writing new code and porting code written for Programming Environment releases prior to 3.2.

  • 4.1 !MIC$ MAXCPUS Directive Removed
  • 4.2 gpp Preprocessor Retired The Fortran preprocessor, gpp, has been retired. Use the following compiler command instead: f90 -eP ...
  • 4.3 !DIR$ INTEGER Directive Removed
  • 4.4 -i46 and -i64 Options Removed

Getting Started with PAT 2.0

ARSC made programming environment 3.2 the default on June 29th. With this change, PAT 2.0 replaced 1.0 as the default version of PAT at ARSC.

PAT (Performance Analysis Tool) is a low-overhead, easy-to-use programming tool which reports on basic aspects of performance, such as:

  • load balance across processors
  • time spent in subroutines and functions
  • hardware counter performance information
PAT 2.0 and 1.0 are not compatible. Here's a brief tutorial to PAT 2.0, with some changes noted (also see "man pat"). STEP 1:

First re-link your program with the PAT run-time library and specify the pat.cld loader directive file:

  cc  *.o -l pat pat.cld -o a.out
  CC  *.o -l pat pat.cld -o a.out
  f90 *.o -l pat pat.cld -o a.out
You don't need to re-compile, but you could:

  cc  prog.c  -l pat pat.cld -o a.out
  CC  prog.CC -l pat pat.cld -o a.out
  f90 prog.f  -l pat pat.cld -o a.out

## # No change in version 2.0 ##


Run as usual:

  mpprun -n7 ./a.out
## # No change in version 2.0 ##


The program run produces a .pif ("Program Information File") which is given the name of your executable with the extension "[.N].pif" (where "N" is added and incremented as necessary to prevent overwriting existing files).

Now run PAT, giving it the "-L" flag, the names of the executable and .pif files, and other flags according to the output desired. For instance:

  pat -L -m -p -T a.out a.out.pif  
The flag, "-L," specifies "line-mode," circumventing the default X Windows System GUI. "-m" requests performance-counter information, "-p," profile information, and "-T," timing information.

## # Two changes since 1.0: # # 1) In PAT 1.0, executables produce pdf ("Program Data Files") not # .pif files. The format of pdf and .pif files differ: you # can't use pat 2.0 to look at pdf files. # # 2) In PAT 1.0, line-mode is the only interface. # # # The flags, -m, -p, -T, and others are unchanged. ##


yukon$ cc -l pat pat.cld mc.c circle.c -o circle.pat

yukon$ mpprun -n10 ./circle.pat 2000 3 1 
  [0] 2000 2272.239196: avg:  1.136119598
  [1] 4000 4514.527705: avg:  1.128631926
  [2] 6000 6746.326122: avg:  1.124387687

yukon$ ls -1 *.pif

yukon$ pat -V                                
  PAT: version

yukon$ pat -L -T circle.pat circle.pat.1.pif
  Elapsed Time                  1.002 sec   10 PEs
  User    Time (ave)            0.996 sec   99%
  System  Time (ave)            0.004 sec    0%

yukon$ pat -L -p circle.pat circle.pat.1.pif

  Profile Information:

  Entity Name        Percent        90% Conf.
                    User Time       Interval  
  next                 42%            1.1
  drand48              40%            1.0
  simulate             15%            0.8

yukon$ pat -L -m circle.pat circle.pat.1.pif

                   Performance counters for FpOps

      Values given are in MILLIONS.
  PE(id)      cycles  operations   ops/sec    dcache  misses/sec
   0(   0)    2649.43    581.74       98.82    0.04     0.01
   1(   0)    2687.46    581.73       97.42    0.29     0.05
   2(   0)    2688.23    581.71       97.39    0.31     0.05
   3(   0)    2688.29    581.74       97.39    0.31     0.05
   4(   0)    2688.05    581.75       97.40    0.30     0.05
   5(   0)    2687.43    581.93       97.45    0.30     0.05
   6(   0)    2687.42    581.81       97.43    0.30     0.05
   7(   0)    2687.30    581.69       97.42    0.29     0.05
   8(   0)    2688.01    581.72       97.40    0.31     0.05
   9(   0)    2686.97    581.76       97.44    0.29     0.05

  1. PAT 2.0 can't read pdf files produced by PAT 1.0.
  2. Don't be fooled by the wording of the man page, which states:
         a.outfile.pif       Specifies the name of the PAT information 
                               [ ... ]  
                             This parameter is optional.
    This parameter is not optional if you want to examine results. If you leave it off, yet ask to see results, PAT halts (but with a helpful message):

    yukon$ pat -L -m circle.pat

    performance counter display option not available without pat data file

    If you leave it off and don't ask for results, then PAT enters a different mode which lets you instrument your code for tracing.
  3. Here are the two notes about PAT 2.0 I was able to find in the document:

    Programming Environment Releases Overview RO-5212 3.0.2

    which is available on-line at:

    3.3.1 Pthread Support in PAT

    The Performance Analysis Tool (PAT) now supports the pthread library and can be used to check performance of threads on all PEs.

    Note: Due to the implementation of pthread support in PAT, pdf files created with older versions of PAT will not work with this release of PAT. The files must be recreated.

    3.3.2 Binary Event Tracing for MPI Routines in PAT

    The Performance Analysis Tool (PAT) now allows the insertion of trace calls into a user binary file. The user can provide entry and exit trace routines in the form of a .o or .a include file in the link or by using the generic trace routines provided by libpat.a.

  4. If needed, ARSC users may still access PAT 1.0 by entering the command:

    module switch PrgEnv PrgEnv.old

    Note that this capability will be removed on the next PE upgrade.
In future issues of this newsletter, we'll cover additional features of PAT including the X Windows interface and tracing.

ARSC Lecture: Linux Clusters, Supercomputing, and Education

Dr. Don Morton of the University of Montana and ARSC will give a presentation on linux clusters, supercomputing, and education, titled:
The Integrative Role of Free Unix Clusters and Supercomputers in HPC Research and Education Activities

The talk will be Monday July 12, 1:30-2:30 pm in Butrovich Building, room 109, and is open to all.


Over the course of the 1990's technology advances have made it possible for increasing numbers of researchers to use supercomputers for solving their large-scale computational problems. At the same time, Free Unix operating systems have increased the power of personal computers and workstations by providing familiar environments to those familiar with the "Unix" way of doing things. The two seemingly disparate computing environments actually have much in common, and occupy unique niches in a symbiotic relationship.

The talk begins by presenting experiences of the author in using and building Linux clusters for scientific and parallel computing activities, coupled with research activities at the Arctic Region Supercomputing Center.

Through a description of research and education activities, we explore the thesis that workstation clusters (focusing on Linux) are often the ideal environment for training and code development activities in high performance computing, while supercomputing systems remain the environment of choice for large-scale number-crunching. Finally, we present suggestions on how you, too, can build your own supercomputer.

Speaker Biography:

Don Morton received his PhD in Computer Science from Louisiana State University in 1994, with research interests in scientific computing, particularly finite element methods as applied to fluid flow problems.

Don is currently a Professor of Computer Science at The University of Montana where he makes heavy use of Linux clusters for research and education.

Don is also an ARSC research affiliate, has worked here every summer since 1994, and has been a regular contributor to this newsletter. Here's a list of his past articles:

  • Issue 90 Porting Heterogeneous Distributed Codes to the T3D
  • Issue 91 pxfgetarg() - Handling Command Line Arguments in T3D Programs
  • Issue 96 18:1 Speedup Demonstrated: Free for Using System Libraries
  • Issue 98 Porting T3D PVM Codes to Network PVM
  • Issue 100 Don Morton Drives the Alcan -- Again
  • Issue 128 Case Study : Streams and 450MHz Clock
  • Issue 146 MPI Communicators for Coupled T3E Codes


Quick-Tip Q & A

A:{{ Sometimes I need to duplicate entire "branches" of an existing
     directory tree.  For instance, I might need this structure:


    As you might guess, it's really annoying to type: 

      mkdir ./v12
      mkdir ./v12/19990201

    Is there an easier way?  }}

    Thanks go to four readers.  Three responded with this answer:

       mkdir -p ./v12/19990201/weather/Burmuda/restart/2301
       mkdir -p ./v12/19990201/weather/Triangle/restart/2303

    And the fourth with this:

        mkdir -p ./v12/19990201/weather/Burmuda/restart/2301   \

    Note that the corresponding "rmdir" option is also "-p".  "rmdir
    -p" recursively removes subdirectories, stopping if it encounters
    one that is not empty.  Removing the directories created above
    takes two steps:

        yukon$ rmdir -p v12/19990201/weather/Burmuda/restart/2301

          cmd-3192 rmdir: v12/19990201/weather/Burmuda/restart/2301 --
             'v12/19990201/weather' was not removed.  The directory is
             not empty.

        yukon$ rmdir -p v12/19990201/weather/Triangle/restart/2303

Q: I use "vi" a bit, and sometimes issue shell commands from within
   it. For instance,

     !1000j sort

   sorts the next 1000 lines of the file.  I also issue "ex" commands.
   For instance:

     :%s/^  \*\* /\<H1\>/c

   replaces "  ** " at the start of lines with "<H1>", and asks for
   confirmation. The problem is that such commands are easy to mistype,
   and that once issued, they're gone.  You can hit "u" to "undo" the
   action of the command, but I'd like "command line editing."

   Is there a trick in vi to retrieve a "!" or ":" command, edit it,
   and re-execute it?

[ Answers, questions, and tips graciously accepted. ]

Current Editors:
Ed Kornkven ARSC HPC Specialist ph: 907-450-8669
Kate Hedstrom ARSC Oceanographic Specialist ph: 907-450-8678
Arctic Region Supercomputing Center
University of Alaska Fairbanks
PO Box 756020
Fairbanks AK 99775-6020
E-mail Subscriptions: Archives:
    Back issues of the ASCII e-mail edition of the ARSC T3D/T3E/HPC Users' Newsletter are available by request. Please contact the editors.
Back to Top