ARSC T3E Users' Newsletter 159, January 8, 1999

Kerberos/SecurID At ARSC

The only access to ARSC systems will soon be via Kerberos/SecurID. This change will take place by next Tuesday, Jan 12, at 10pm at the earliest, and by next Friday at the latest.

We have begun a series of e-mail bulletins to all ARSC users regarding this. They're all available online at:

http://www.arsc.edu/support/news/bulletins/Krb5.html

and the most recent is posted as "news transition" on all ARSC systems.

We appreciate your patience as we proceed with this vital project.

Creating Your Own Libraries

[ Many thanks to Dr. Jeff Orrey of Ojo Solutions for this contribution. Jeff's contact info is given at the end of this article. ]

Do you find yourself frequently reusing code fragments to perform tasks that you do on a regular basis? If so, those code fragments are perfect candidates for a library. Creating you own library of routines makes your codes cleaner and more compact, and it encourages you to standardize certain operations that your codes frequently perform.

Creating and using a library on the ARSC Cray T3E is no different that doing so on other Unix machines. First, the source code for the library must be compiled into object code. Then the object files are linked together into an archive using ar(1). The archive is then passed to ranlib(1), which prepares the archive for linking. All of these steps are typically performed in the Makefile for the library. Once the library is prepared for linking, the routines in the library may be called from code which links the library.

Following is a Makefile example for a library called libexample.a:

--------------------------------------------------------------------------
#
# Makefile for generating example library libexample.a
#
LIB= libexample.a

AR= /bin/ar
ARFLAGS= -r
LORDER=`lorder
TSORT=| tsort`
RANLIB= /opt/ctl/bin/ranlib
INCLUDES= -I${HOME}/include
INSTALLDIR= ${HOME}/lib

CLEAN=

cflags= -g -I${HOME}/include

OBJS= libexample.o

${LIB} : ${OBJS}
       ${AR} ${ARFLAGS} $@ ${LORDER} ${OBJS} ${TSORT}
       ${RANLIB} $@

.c.o:
       ${CC} ${CFLAGS} ${COFLAGS} ${INCLUDES} ${DEFINES} -c $*.c

install:
       cp ${LIB} ${INSTALLDIR}
--------------------------------------------------------------------------

In this case, libexample.c contains header files, a #define, and a routine which reads and prints a data file containing geographic header information and data values stored in a grid:

--------------------------------------------------------------------------
/*
 *     libexample.c
 */

#include 

#include "structure.h"

#define DIE(str) error_fatal(__LINE__,__FILE__,str)

void
error_fatal( line, file, msg )
int line;
char *file, *msg;
{
  fprintf (stderr,
    "FATAL ERROR in file: %s line: %d  message:\n*****\"%s\"\n",
    file, line, msg);

  fflush (stderr);

  globalexit (1);

  /* kill (pid0, SIGINT);
  exit(1); */
}

read_vgrid( g, filename )
VGrid *g;
char *filename;
{
       FILE *ifile;
       int i, err, nxyz;

       ifile = fopen( filename, "r" );
       if ( ifile == NULL ) DIE(  "Can't open input vgrid file!\n" );
       
       err = fread ( &(g->lat_origin), sizeof(double), 1, ifile );
       if (err != 1) DIE(  "Can't read parameter: lat_origin\n");

       err = fread ( &(g->lon_origin), sizeof(double), 1, ifile );
       if (err != 1) DIE(  "Can't read parameter: lon_origin\n");

       err = fread ( &(g->depth_origin), sizeof(double), 1, ifile );
       if (err != 1) DIE(  "Can't read parameter: depth_origin\n");

       err = fread ( &(g->angle), sizeof(double), 1, ifile );
       if (err != 1) DIE(  "Can't read parameter: angle\n");

       err = fread ( &(g->phase), 1, 8, ifile );
       if (err != 8) DIE(  "Can't read parameter: phase\n");

       err = fread ( &(g->nvel), sizeof(int), 1, ifile );
       if (err != 1) DIE(  "Can't read parameter: nvol\n");

       g->vel = (double *)malloc( g->nvel * sizeof(double) );
       if ( g->vel == NULL ) DIE(  "Malloc Error!\n" );
       err = fread ( g->vel, sizeof(double), g->nvel, ifile );
       if (err != g->nvel) DIE(  "Can't read velocities\n");

       err = fread ( &(g->nx), sizeof(int), 1, ifile );
       if (err != 1) DIE(  "Can't read parameter: nx\n");

       err = fread ( &(g->ny), sizeof(int), 1, ifile );
       if (err != 1) DIE(  "Can't read parameter: ny\n");

       err = fread ( &(g->nz), sizeof(int), 1, ifile );
       if (err != 1) DIE(  "Can't read parameter: nz\n");

       err = fread ( &(g->dg), sizeof(double), 1, ifile );
       if (err != 1) DIE(  "Can't read parameter: dg\n");

       nxyz = g->nx * g->ny * g->nz;
       g->vol = (unsigned int *)malloc( nxyz * sizeof(unsigned int) );
       if ( g->vol == NULL ) DIE(  "Malloc Error!\n" );
       err = fread ( g->vol, sizeof(unsigned int), nxyz, ifile );
       if (err != nxyz ) DIE(  "Can't read volume\n");

       fclose(ifile);

       printf( "read_vgrid: %s\n", filename );
       printf( "lat_origin: %f\n", g->lat_origin );
       printf( "lon_origin: %f\n", g->lon_origin );
       printf( "depth_origin: %f\n", g->depth_origin );
       printf( "angle: %f\n", g->angle );
       printf( "phase: %s\n", g->phase );
       printf( "nx: %d\n", g->nx );
       printf( "ny: %d\n", g->ny );
       printf( "nz: %d\n", g->nz );
       printf( "dg: %f\n", g->dg );
       printf( "nvel: %d\n", g->nvel );
       for ( i = 0; i < g->nvel; i++ ) 
               printf( "Velocity[%d]:\t%f\n", i, g->vel[i] );

}

--------------------------------------------------------------------------

Once libexample.c is compiled, it can be used by code which requires reading of data using the utility read_vgrid(), above. Following is a Makefile template, for a routine called mycode.c which incorporates the library libexample.a:

--------------------------------------------------------------------------
#
#      Makefile template for a routine called mycode.c which incorporates
#      the library libexample.a
#
BIN= mycode

INCLUDES= -I$(HOME)/include

MAN1= mycode.1

MYLIBDIR = -L$(HOME)/lib
MYLIBS  = -lexample
LDLIBS = $(MYLIBDIR) $(MYLIBS)

INSTALLDIR = $(HOME)/bin

CFLAGS= -O

F90 = f90

OBJS= mycode.o

$(BIN) : $(OBJS)
        $(F90) $(CFLAGS) $(INCLUDES) -o $@ $(OBJS) $(LDFLAGS) $(LDLIBS)

.f.o:
        $(F90) $(FOFLAGS) -c $*.f

.F.o:
        $(F90) $(FOFLAGS) -c $*.F

.c.o:
        $(CC) $(CFLAGS) $(COFLAGS) $(INCLUDES) $(DEFINES) -c $*.c

clean:
        rm -f $(BIN) $(OBJS)

install:
        cp $(BIN) $(INSTALLDIR)

--------------------------------------------------------------------------

Jeff Orrey may be reached at:

  Jeffrey Orrey
  Ojo Solutions
  2888 Bluff, Suite 400
  Boulder, CO 80301
  303-938-1038
  303-938-1338 ( fax )
  jeffo@ojosol.com

Specifying Number Of Processors

How do you tell Unicos/mk the number of processors (PEs) to use when it runs your application? It's a basic, and, for new users, a sometimes confusing, question. Here's a quick review:

Creating Malleable Executables

You should (probably) compile your application to be "malleable" so you may specify the number of PEs when you launch the executable. To create a malleable executable, omit the "-X" compiler option. E.g.:

  f90 prog.f
  cc prog.c 

Running Malleable Executables Interactively

At ARSC, you may run interactive parallel jobs on up to 8 PEs. Assuming "a.out" was compiled to be malleable, you may launch it using the "mpprun" command, providing the "-n" option to specify the number of PEs. E.g.:

  mpprun -n 5 ./a.out

(Remember, at ARSC you must specify an explicit path to the executable, even if it is the current default directory--otherwise known as ".".) The above command would tell Unicos/mk to launch "a.out" on 5 application PEs.

So far so good...

Running Malleable Executables In Batch

The batch scheduler on the T3E is "NQS." To use NQS, you write a script and submit it using the "qsub" command.

All qsub scripts have two parts: commands for NQS and command for the shell. With regard to the number of PEs, first you request them from NQS and then you request them from mpprun. All this is documented in:

  http://www.arsc.edu/support/howtos/usingnqs.html

Here's example #1 from that document:

############################################################
#QSUB -q mpp                 # Pipe queue, "mpp" required
#QSUB -l mpp_p=6             # Reserve 6 application PEs
#QSUB -l mpp_t=1:30:10       # Max overall MPP job time: 1 hour, 30 min, 10 sec.
#QSUB -l p_mpp_t=30:00       # Max per-process MPP time: 30 min.
#QSUB -lT 10:00              # Max overall time on front-end: 10 min.

cd /u1/uaf/morris/progs
date
mpprun -n6 ./myprog data1    # Run on 6 PEs with first data set
mpprun -n6 ./myprog data2
mpprun -n6 ./myprog data3
date
############################################################

Note: six PEs are requested of both NQS (#QSUB -l mpp_p=6) and the system (mpprun -n6).

Creating Non-Malleable Executables

You may hard-wire the number of processors required into an executable the "-X" compiler option. E.g.:

  f90 -X 6 prog.f
  cc -X 6 prog.c

We discourage you from doing this (or needing to do this)! If your program can only run on a fixed number of PEs, then it won't scale up to bigger machines and it won't squeeze into available partitions on the current machine.

Running Non-Malleable Executables

Whether running interactively or in batch, if your executable is non-malleable, you must NOT use the "-n" option to mpprun--even if you specify the same number of PEs for which the executable was compiled.

Either of the following commands will run the executable:

  ./a.out
  mpprun ./a.out

Testing For Malleability

To determine if your executable is malleable or not, and if not, how many PEs it will run on, use the "file" command:

  file a.out

More Information

See:

  http://www.arsc.edu/support/howtos/usingt3e.shtml,

past issues of this newsletter, man pages, etc.

A: {{ Will C correctly cast a pointer reference value?  For instance,
      given this declaration:

         unsigned temp = *uint;

      will "temp" be set as expected even if "uint" is an int pointer? }}

# 
# Thanks to Shawn Houston of ARSC for both question and answer.  Here's
# the answer as paraphrased from the ANSI C Standard:
# 
#   A pointer to an object may be converted to a pointer to a different
#   object type.  The resulting pointer might not be valid if it is
#   improperly aligned for the type pointed to.  It is guaranteed,
#   however, that a pointer to an object of a given alignment may be
#   converted to a pointer to an object of the same or less strict
#   alignment and back again. The result shall compare equal to the
#   original pointer.
# 
# Here's a curious little test:
# 

  /*------------------------------------------------------------------*/
  /* badcast.c */
  
  unsigned badcast(unsigned* uint)
  {
    unsigned temp = *uint;
  
    return temp;
  }
  
  
  int main(int argc, char* argv[])
  {
    int i;
  
    for(i = -4; i < 5; ++i)
    {
      printf("%d ?= %u : %s\n", 
        i, badcast(&i),  i == badcast(&i) ? "YES" : "NO");
    }
  
    return 0;
  }
  /*------------------------------------------------------------------*/

# 
# The compiler issues warnings about "incompatible pointer type assignment,"
# but, at the default error levels, creates an executable. Here's a run:
# 

  yukon$ ./a.out
  -4 ?= 18446744073709551612 : YES
  -3 ?= 18446744073709551613 : YES
  -2 ?= 18446744073709551614 : YES
  -1 ?= 18446744073709551615 : YES
  0 ?= 0 : YES
  1 ?= 1 : YES
  2 ?= 2 : YES
  3 ?= 3 : YES
  4 ?= 4 : YES



Q: To SecurID Card veterans:  any advice?  For instance, how do you
   carry your card (without destroying it) so it's not at work when you
   need it at home or vice versa?

[ Answers, questions, and tips graciously accepted. ]

Thomas J. Baring	ARSC Web Specialist	ph: 907-450-8619
Donald Bahls	ARSC User Consultant	ph: 907-450-8674
Arctic Region Supercomputing Center
University of Alaska Fairbanks
PO Box 756020
Fairbanks AK 99775-6020

Send comments and questions to the current editors using this Contact Form.

To subscribe to the e-mail edition of the ARSC HPC Users' Newsletter, send this message to: "majordomo@arsc.edu":
    subscribe hpc_users
    end
For help with majordomo, send this:
    help
    end
Leave the "subject" line of your message blank.

The ASCII e-mail edition of the ARSC T3D/T3E/HPC Users' Newsletter is archived and available by anonymous FTP to: ftp://ftp.arsc.edu/pub/publications/newsletters/

Arctic Region Supercomputing Center
PO Box 756020, Fairbanks, AK 99775 | voice: 907-450-8600 | email:

home | search | about | support | news | science | resources