ARSC T3E Users' Newsletter 144, June 12, 1998

MPICH for the T3E

A preliminary release of MPICH for the T3E is now available.

MPICH is a portable, public implementation of the MPI standard, written jointly by researchers at Argonne National Labs and Mississippi State University, and has been ported to a wide variety of computing systems. The Cray T3D port was first written by Ron Brightwell of MSU, and later modified to conform to MPICH's second generation Abstract Device Interface (ADI-2) by Shane Hebert, also of MSU.

The T3E port was implemented by Shane Hebert, who is making a preliminary release available for final user testing.

Shane writes:

Download:

ftp://aurora.cs.msstate.edu/pub/t3e/mpich-t3e.tar.gz

You should be able to 'configure -arch=cray_t3e -device=t3e' and it should configure. Then you can 'make' and get all the libraries, mpirun, and such. Currently, mpirun only uses mpprun and can, therefore, only submit interactive jobs. Try it out and let me know if there are any bugs, at: <>. There is a paper in that same directory called t3e-preprint.ps that you may want to read as well.

For more on MPICH, visit the MPICH home page:

http://www-c.mcs.anl.gov/mpi/mpich/

MPICH at ARSC

ARSC will not make a supported installation of MPICH for the T3E until its official release. For now, it is installed in a temporary location. If you are eager to give it a try, please do so, and let us know what you discover.

On yukon, MPICH is temporarily installed in:
     /u1/uaf/tbtester/pkg/mpich/

The path to the MPI include files is:
     /u1/uaf/tbtester/pkg/mpich/include

The path to the libraries is:
     /u1/uaf/tbtester/pkg/mpich/lib/cray_t3e/t3e/

The following makefile shows how to compile and link using these resources.

######################################################################
# Generic makefile for MPI codes.  Can use to switch between the
# Cray MPT (message passing toolkit) version of MPI and the MPICH
# version.
# 
# Usage:
# 
# On the command line, define SRC as the list of source files, and make
# the object for the desired language/MPI pair.  For example:
# 
# make SRC="main.c procs.c" mpich11c
# make SRC="main.c procs.c" mptc
# make SRC="prog.f subs.f" mpich11f
# make SRC="prog.f subs.f" mptf
# 
# 
# 
# Note: to use the MPT version, make sure the mpt module is loaded.
# Check with "module list."  If you don't see "mpt," then load it
# with "module load mpt."
# 


# Define paths to ARSC preliminary installation of MPICH 1.1.0 
MPICH_11_INC_PATH=/u1/uaf/tbtester/pkg/mpich/include
MPICH_11_LIB_PATH=/u1/uaf/tbtester/pkg/mpich/lib/cray_t3e/t3e/


FC=f90
CC=cc


mpich11f:
       $(FC) $(SRC) -I$(MPICH_11_INC_PATH) -L$(MPICH_11_LIB_PATH) -lfmpi -lmpi

mptf:
       $(FC) $(SRC) 



mpich11c:
       $(CC) $(SRC) -I$(MPICH_11_INC_PATH) -L$(MPICH_11_LIB_PATH) -lmpi 

mptc:
       $(CC) $(SRC) 

######################################################################

"TARGET=cray-t3e" May Look Like a Good Idea...

Don't do this:

export TARGET=cray-t3e [korn or Bourne shell]

or this:

setenv TARGET cray-t3e [C shell]

and then expect f90 or CC to compile correctly for your local T3E host. The TARGET variable should be used for cross-compiling (e.g., using a Y-MP to compile for a T3D), not for local compiling.

To compile for the local host, don't set TARGET at all, execute "unset TARGET", or set TARGET to "target":

export TARGET=target [korn or Bourne shell]

or:

setenv TARGET target [C shell],

then compile.

The problem is that the compilers check the TARGET environment variable to get information about the target host for which they're compiling. However, the supplied target, "cray-t3e" is for a generic (128 MB) T3E, not necessarily equivalent to the local host, and (ARSC users) definitely not equivalent to yukon (256 MB).

This came up recently, when a user's code failed to compile because, according to the error messages, yukon only had 134,217,728 bytes (128 MB) of memory per PE while the code needed more.

The problem was in this portion of the makefile:

#------------------------------------------------------------------
#  compiler/loader options
#------------------------------------------------------------------
 Comp = env TARGET=cray-t3e f90 -c -O3,aggress,unroll2
 Load = env TARGET=cray-t3e f90
 F77 = env TARGET=cray-t3e f90 -c -O3,aggress,unroll2
 Cc = env TARGET=cray-t3e cc -c
#------------------------------------------------------------------
#------------------------------------------------------------------

The makefile's incorrect TARGET settings told f90 to compile for the generic, 128MB T3E. Ooops!

Here's a session to help you recognize the error messages you'll probably see ("exceedlim.f" simply allocates 160MB and quits):

yukon$ 
yukon$ export TARGET=cray-t3e   
yukon$ f90 exceedlim.f                                                        

        real  a( N , M)
              ^         
  cf90-614 f90: WARNING $MAIN, File = exceedlim.f, Line = 3, Column = 8 
    The storage size needed for "A" exceeds 16777216 words, the maximum 
  storage size available.
  
  cf90: Cray CF90 Version 3.0.0.1 (f18p45m332a39) Thu Jun 11, 1998  14:32:41
  cf90: COMPILE TIME 0.239443 SECONDS
  cf90: MAXIMUM FIELD LENGTH 1583776 DECIMAL WORDS
  cf90: 14 SOURCE LINES
  cf90: 0 ERRORS, 1 WARNINGS, 0 OTHER MESSAGES, 0 ANSI
  cf90: CODE: 30 WORDS, DATA: 20000099 WORDS
  cld-422 cld: WARNING 
    The zone `stack', which has a base address of 0x200000000 bytes and 
    a size of 160366080 bytes, exceeds the address space of 134217728 
    bytes provided by the target platform `CRAY T3E'.
  cld-423 cld: WARNING 
    The static address space size of 161533732 bytes exceeds the memory 
    size of 134217728 bytes provided by the target platform `CRAY T3E'.
  cld-431 cld: WARNING 
    The resulting output file `a.out' is not executable because of previous 
    WARNING messages.

yukon$ unset TARGET
yukon$ f90 exceedlim.f
yukon$ 


The "target" UNICOS command shows what's going on:

yukon$ export TARGET=target
yukon$ target
Primary machine type is:  CRAY-T3E
   memsize  = 33554432
yukon$ 
yukon$ export TARGET=cray-t3e
yukon$ target
Primary machine type is:  CRAY-T3E
   memsize  = 16777216
yukon$ 
yukon$ unset TARGET
yukon$ target
Primary machine type is:  CRAY-T3E
   memsize  = 33554432

NOTE: This article is basically a T3E update of:

"The 'TARGET' Environment Variable"

which we ran exactly two years ago today (+ or - 5 days) in the T3D Newsletter. See issue #90:

http://www.arsc.edu/support/news/T3Dnews/T3Dnews90.shtml

HPF Users: Help Requested!

A PhD student contacted us. He has built an HPF simulator and needs data from actual HPF users for his assessment.

His survey form is on-line at:

http://www.brunel.ac.uk/~cspgvvv/questionnaire.html

Here is the actual request:

> 
> Dear Sir, 
> 
> I am a PhD student at Brunel university and am looking for some people
> whose work is related to High Performance Fortran (HPF). 
> 
> My research is related to the role of simulation in the performance
> prediction of data parallel programs. As an example of how simulation
> might be used I have built a HPF simulator. This simulator provides
> information on performance that is accessible during the design stage of 
> a program. 
> 
> In order to assess the actual (or potential) use of such a
> simulator I am currently attempting to create a representative user
> group. Each user is sent a series of questions about simulation and
> performance prediction with regard to their actual experience in 
> developing parallel codes. Each will be subsequently contacted with the
> results of the survey.
> 
> I would be very grateful if you could spent a small amount of time to
> take part in this exercise. I have included the questionnaire in this
> mail. I will also send on the results of the survey when complete. I have
> to mention that your responses will be treated in confidence. 
> 
> Please feel free to distribute this questionnaire to anyone who is a user
> of HPF or Parallel Fortran.
> 
> 
> Many thanks, 
>   Vassilios Vassiliou
> 
>   E-mail: 
>   Web: http://www.brunel.ac.uk/~cspgvvv
> 

A: {{ The following snippet of Fortran code fails on the "write" because
     it would output 40*7, or 280 characters per line.  It turns out 
     that the maximum allowed is 267 per line, and that there is a
     corresponding restriction on "reads."

        open (unit=5,file="test.out")
        do j=1,5
          write (5,100) (out(j,i),i=1,40)
 100      format (40(i6,' '))
        enddo
        close (unit=5)

    [ ... ]

    What if you *really* needed to read or write lines in excess of
    267 characters?  What could you do?  }}


    # 
    # Many thanks to the reader who sent this response:
    #

    Just change the open to open (unit=5,file="test.out",recl=1024)
    where "1024" is any large enough record length in bytes (I
    typically use 4096, since then I almost never have to worry about
    extending it again).

    This example is a little confusing because by default stdin is
    connected to unit 5.  When you issued the open with a filename, the
    connection to stdin gets lost.  On MPPs stdin isn't very useful
    (and I never use it), but I still avoid unit 5 because someone
    reading the program but not seeing the open will assume stdin (in
    this case they should realize that a "write" isn't going to
    stdin).

    The default record length (without a recl) is machine specific.
    There is no standard way (so far as I know) of changing the record
    length for stdin or stdout (unit 5 and 6 without an open).  It is
    possible to issue an OPEN on an already open file (just leave out
    FILE=), but RECL isn't one of the keywords that is allowed by the
    Fortran 90 standard in such cases (it might be OK with Cray F90).
    An example is:

          OPEN(UNIT=6, DELIM='QUOTE', IOSTAT=IOS)

    which changes the delimiter for character strings on stdout from
    the default 'NONE' to a quotation mark.


Q: [[ This question submitted by a reader.  Thanks! ]] 

  Since you mentioned the totalview debugger in the last newsletter, I
  have a combination factoid/question:

  In C, you don't need to specify the size of arrays at compile time
  (ie.  pointers are basically arrays).  So you could have a code
  fragment:

       double* x;
       double* y;

       for(i=0;i<SIZE;i++) {
               y[i] = alpha*x[i] + y[i];
       }

  ie. the usual _AXPY loop.  However, since SIZE is variable (say
  changeable at run-time), the totalview debugger won't display this as
  an array by default - ie. you don't get the nice scrollable display
  of all SIZE components.

  How can you view C arrays in totalview?

[ Answers, questions, and tips graciously accepted. ]

Ed Kornkven	ARSC HPC Specialist	ph: 907-450-8669
Craig Stephenson	ARSC User Consultant	ph: 907-450-8653
Arctic Region Supercomputing Center
University of Alaska Fairbanks
PO Box 756020
Fairbanks AK 99775-6020

Send comments and questions to the current editors using this Contact Form.

Subscribe to (or unsubscribe from) the e-mail edition of the ARSC HPC Users' Newsletter.

The ASCII e-mail edition of the ARSC T3D/T3E/HPC Users' Newsletter is archived and available by anonymous FTP to: ftp://ftp.arsc.edu/pub/publications/newsletters/

Arctic Region Supercomputing Center
PO Box 756020, Fairbanks, AK 99775 | voice: 907-450-8600 | email:

home | search | about | support | news | science | resources