ARSC T3E Users' Newsletter 161, January 5, 1999

ARSC T3E Downtime for Upgrade

Yukon and Chilkoot's hard drive capacities were increased to over 500GB each over the weekend of Jan 30-31.

Yukon will be down again from 6am, Saturday, Feb. 13 through 8am, Monday, Feb 15. It will come back up with 272 PEs! Certainly a nice upgrade.

MPI Message Tags On Yukon

[ Many thanks to Brad Chamberlain of the University of Washington CS department for this contribution. ]

This article details the use of MPI message tags in C on yukon using Cray's implementation of MPI as well as the MPICH version, based on one user's experience and problems and a bit of advice from the comp.parallel.mpi news group.

MPI's various Send/Recv calls take as one of their arguments a user-defined integer tag, used to distinguish messages from one another. In one model, these tags might be chosen explicitly by the user to represent different types of messages that their program might utilize. Another method of generating tags is to start with a small tag (such as 0) and cycle through them in an automated manner.

The range of possible tags varies from implementation to implementation of MPI. The MPI standard requires that all implementations support tags in the range 0..32767, however any particular implementation may use a larger upper bound if it desires.

The way to determine an implementation's upper bound in C is to use the MPI function MPI_Attr_Get() . The short description of MPI_Attr_Get() in the MPI 1.1 standard is as follows:

int MPI_Attr_get(MPI_Comm comm,int keyval,void *attribute_val,

                         int *flag)

Retrieves attribute value by key. The call is erroneous if there is no key with value keyval. On the other hand, the call is correct if the key value exists, but no attribute is attached on comm for that key; in such case, the call returns flag = false. In particular MPI_KEYVAL_INVALID is an erroneous key value.

The name of the attribute that describes the tag upper bound is called MPI_TAG_UB . Thus, impatient programmers, such as the author, that want to quickly determine this value, might make the following call:

{ /* INCORRECT code to find maximum tag value */

  int tagub,flag;



It turns out that this call is incorrect. MPI attributes are actually returned by giving the user a pointer to the value (presumably to get around the problem of different attributes having different types in C). As a result, code to correctly find the bound would look like this:

{ /* code to find maximum tag value */

  int *tagub,flag,maxtag;


  if (flag) {

    maxtag = *tagub;

  } else {

    maxtag = 32767;



Using this code on the T3E, we find that the upper bounds are as follows for the two MPI implementations:

                  Cray MPT            MPICH


         65,534          1,073,741,823

(Editor's note: Both Fortran and C have the same limits on the T3E. )

When exceeding these bounds (in this case using an MPI_Irecv call), each implementation halts with a message as follows:

Cray MPT:


        -MPI- ERROR FATAL: world rank 0, comm 0

          In call MPI_Irecv, [-496, class MPI_ERR_TAG],

          [Invalid tag argument] bad tag argument



        0 - MPI_IRECV : Invalid message tag 1073741825

        [0]  Aborting program !

        [0] Aborting program!

Thus, the result is that MPICH supports a higher tag limit (for those users that need it) and provides clearer error messages, in that it gives the illegal tag number (for those users that exceed it). Although this last point seems like a minor one, debugging an MPI program after it's cycled through 65,534 tags is no small task, and the extra information is appreciated.

Co-Array Fortran Web Site: ""

Dr. Alan Wallcraft of NRL maintains an important web site for CAF users, . It contains:

  • Articles
    • A short introduction to Co-Array Fortran
    • The complete language definition
    • I/O in Co-Array Fortran
    • Why not just use MPI?
    • A subset easy to translate into OpenMP Fortran
    • A Subset Co-Array Fortran to OpenMP Fortran translator
  • Discussion of implementations
  • Sample programs
  • Links to earlier papers
  • A request for feedback and links to new CAF papers/programs

Here are some comments concerning this site, from Alan:

In particular it includes a list of the parts of the language missing from CF90 3.1 and a link to a module that provides the missing intrinsics. This CF90 stuff is the most important for T3E users.

Note that I also now include an implementation of sync_images, because a) the original CF90 version was broken, and b) all my sync intrinsics have debugging and deadlock detection options. Any routine that starts with USE CAF_INTRINSICS will get the module version rather than the intrinsic.

From AAAREAD.ME in Co-Array_cf90.tar.gz:

A module providing the Co-Array Fortran intrinsics missing from CF90 3.1 for the T3E. Also includes a generic Fortran 90 wall time subroutine and a few example Co-Array Fortran programs.

From caf_intrinsics_t3e.f90:

module caf_intrinsics

      implicit none


!     Co-Array Fortran intrinsics missing from cf90 3.1 on Cray T3E.


!     sync_file,      sync_memory,

!     sync_all,       sync_team,

!     start_critical, end_critical.


!     Also includes a reference implementation of sync_images.

!     This may be slower than the intrinsic version, but has

!     debugging and deadlock detection capabilities.


!     Call caf_init once on each image at the start of the program.

!     Add  use caf_intrinsics  anywhere these "intrinsics" are called.


!     Environment variables:

!       caf_sync_debug       is set to turn on debugging (default off).

!       caf_deadlock_seconds controls deadlock detection (default none).


!     Call caf_reset to change debugging or deadlock detection status.


!     Alan J. Wallcraft,  NRL,  October 1998.

!     With modifications by John Reid,  November 1998.



Also, wtime.f90 is my attempt at a portable wall clock timer based on F90's SYSTEM_CLOCK. It has worked with all compilers I have tried it on so far, although at least one version of NAG F90 had a timing granularity of 1 second.

T3E/O2000 Scalability Paper

"PEMCS" is the "Journal of Performance Evaluation and Modelling for Computer Systems." PEMCS has several papers on line, including this new posting:

Comparing the Scalability of the Cray T3E-600 and the Cray Origin 2000 Using SHMEM Routines, by Glenn R. Luecke, Bruno Raffin and James J. Coyle, Iowa Sate University, Ames, Iowa USA

Check out...

HPF User Group meeting

>         HUG'99: The 3rd Annual HPF User Group meeting 
> The Third Annual High Performance Fortran User Group (`HUG') meeting
> will be held on May 24-25,1999 at the Renaissance Wailea Resort Maui,
> Hawaii. This meeting follows the first two meetings in the series held
> in 1997 in Santa Fe, New Mexico, USA, and in 1998 in Porto, Portugal.
> It provides an opportunity for users of HPF to meet each other, share
> ideas and experiences, and obtain up-to-date information on current HPF
> implementations and future plans.
> The HUG meeting will include:
>     * invited presentations by HPF users & technology providers
>     * contributed talks
>     * panel discussions
> There may also be a poster session and demonstrations.
> The HPF User's Group meeting will be followed by an HPF Forum Meeting
> on May 26 at the same location.
> Further information about this event will be available on the HUG'99 Web
> page at:

> Enquiries may be sent to

Quick-Tip Q & A

A:{{ Imagine... You've saved all issues of the T3E Newsletter as separate
     files in a single directory.  Can you generate a list of all
     articles, by issue ? }}

Thanks to one reader for this solution:

  cat ./* 
 egrep '(^ARSC T3E Users)
(^    \* )'

In case you don't use egrep, it's a powerful cousin of grep which lets
you use regular expressions in your matching criteria.  The above egrep
command uses some common syntax:

  The "
" character OR's the alternatives which it separates. For
    egrep '(this)
(that)' would match all lines containing "this" or "that". 
  The "^" character matches the start of a line. For instance, the above
  command would only match "ARSC T3E Users" if the "A" appeared in column 1.
  The "\" character indicates that the next character is to be taken
  literally (the "\" itself is not matched). This is often called
  "escaping" the next character.  In the above command, "*" must be
  escaped because it is also a character with special meaning to egrep.

A couple of other important special characters:

  The "$" character matches the end of a line.
  The "*" character indicates that 0 or more occurrences of the
  previous character must be matched.  
  The "." character matches 1 occurrence of anything.

Here's another solution which grabs the article headings instead of the
table of contents by matching the two leading/tailing *'s:

  cat ./* 
 egrep '(^ARSC T3E Users)
(^ *\*\* .* \*\* *$)'

The second alternative says to match all lines that:

  begin with 0 or more spaces which are followed by
    2 *'s,
      a space,
        0 or more of anything,
          2 *'s,
            0 or more spaces,
              and the end of the line.

Either of these two solutions could accidentally extract lines that
aren't actually titles, and the first extracts them from the table of
contents while the second extracts them from the body of the

Q: Which is better at retrieving multiple migrated files?
    yukon%  dmget A B C
    yukon%  dmget A
    yukon%  dmget B
    yukon%  dmget C

[ Answers, questions, and tips graciously accepted. ]

Current Editors:
Ed Kornkven ARSC HPC Specialist ph: 907-450-8669
Kate Hedstrom ARSC Oceanographic Specialist ph: 907-450-8678
Arctic Region Supercomputing Center
University of Alaska Fairbanks
PO Box 756020
Fairbanks AK 99775-6020
E-mail Subscriptions: Archives:
    Back issues of the ASCII e-mail edition of the ARSC T3D/T3E/HPC Users' Newsletter are available by request. Please contact the editors.
Back to Top