ARSC HPC Users' Newsletter 277, September 14, 2003

Contents

• Access Grid Events: Genomics & Bioinformatics and MPI
• Documentation on ARSC IBM Regatta
• Cray assign To Make Files Memory Resident
• Quick Tip

Access Grid Events: Genomics & Bioinformatics and MPI

2003 Virtual Conference on Genomics & Bioinformatics

  Sept 16-19, 2003
  6am-2:15pm   Alaska Daylight Time (ADT)
    **  NOTE: ARSC will join this conference at 8am each day, by default.  
    **  If you want to show up at 6am, contact Paul Mercer (info below)
    **  in advance. 
  room: Butrovich 109

For the complete schedule of events, go to:

http://www.ndsu.nodak.edu/virtual-genomics/schedule_2003.htm

Parallel Programming with MPI

  Every Wednesday in October, 2003
  9am-1pm ADT
  room: Butrovich 109

For schedule of events, go to:

http://alliance.osc.edu/mpi/osc.html

--

Paul Mercer
voice: 474-6110
email: mercer@arsc.edu

Documentation on ARSC IBM Regatta

A 32 processor IBM p690 system, "iceflyer," plus a 2 processor frontend to the p690, have been available for at ARSC for almost a year. Last spring, the system was reconfigured using logical partitions (LPARs), and this has resulted in some name confusion.

If you're a new or prospective user, you should read this before logging on:

http://www.arsc.edu/support/howtos/usingibmp690.html

• iceflyer: a frontend intended for file transfers, compilations, and job submissions, has 2 RS64 processors and 2 GB of memory
• ferry: an interactive node intended for graphical and debugging runs, has 7 Power4 processors and 14 GB of memory
• umiak: a batch node intended for work scheduled through LoadLeveler, has 24 Power4 processors and 45 GB of memory

• iceflyer, aka f2n1
• ferry, aka f1n2
• umiak, aka f1n1

Cray assign To Make Files Memory Resident

Cray's assign command (man assign) is frequently used to transform data files from one binary type (like IEEE with f77 blocking) to another as the files are read or written at run-time. It has power beyond that, however.

The assign command can also allow a program to "store" a file in memory rather than on disk. When the program ends, the file can be copied to disk, as usual.

A profile of an application I've been porting to the X1 showed that about 50% of its time, in the serial version, was being spent reading/writing a handful of scratch files. In the MPI version, the number of scratch files grows with the number of processors, so this situation would get worse, with multiple tasks competing for the same set of I/O channels.

The files are small enough to simply use Cray's assign command to make them memory resident. From 50% in the profile, the read/writes dropped to essentially 0%.

Here's an example assign command to make the files memory resident:

assign -F mr.scr.ovfl.start_size=41368 p:WAVE%

Explanation:

  "mr" --   memory resident
  "scr" --  scratch (don't save to disk on program termination) 
  "ovfl" -- if file grows too large to keep in memory, let it overflow
	    to disk
  "start_size=41368" -- initial size of memory to block out for the file 
	    (in units of file blocks, 4096 bytes)
  "p:WAVE%" -- the file names match this pattern, where "%" matches
	    anything (like the regular expression "WAVE.*" ).

For details, see "man intro_ffio" and "man assign".

Don't forget to look at "intro_ffio" because a lot of details are there, not in "man assign."

--

Everyone loves an example, so I extracted the key elements from the application, and present them here. The program writes 3 arrays to the file to initialize it. It then reads the arrays, modifies them, and rewrites them, 20 times.

#QSUB -q batch
#QSUB -lM 2GB 
#QSUB -lT 30:00
#QSUB -eo

cd $QSUB_WORKDIR

export NCPUS=1


echo "=================================================================="
echo "Start test without memory resident assignment:"
  rm TEST_FILE
  assign -V
  date
  time ./test_mr
  date
echo "Done"
echo "=================================================================="
echo " "


echo "=================================================================="
echo "Start test without memory resident assignment:"
  rm TEST_FILE

  export FILENV=\$EVAR 
  eval $(assign -F mr.scr.ovfl.start_size=41368 u:11)
  assign -V

  date
  time ./test_mr
  date
echo "Done"
echo "=================================================================="
echo " "

      program test_mr
      implicit complex(c)
      PARAMETER(NGX=72,NGY=64,NGZ=64)
      PARAMETER(NRPLWV=60000,NSLICE=100)
      PARAMETER(NBNDT=174,NBSIZE=87)
      PARAMETER(NKPTS=1,KPTSET=4*1)

      integer, parameter :: niterations = 20
      integer(kind=4), parameter :: iounit = 11

!c     -----wavefunctions
      common /wfp/ cptwfp(nrplwv,nbndt,nkpts)
      common /nindpw/ nindpw(nrplwv,nkpts)
      common /nallkp/ nallkp(nkpts,kptset)
      common /datake/ datake(4,nrplwv,nkpts)

      print*,"Array sizes: (cptwfp,datake,nindpw)=",
     & (nrplwv*nbndt*nkpts),
     & (4*nrplwv*nkpts),
     & (nrplwv*nkpts)

      nbytes=
     & (nrplwv*nbndt*nkpts)*2*kind(cptwfp) +  ! complex 2x storage of type kind
     & (4*nrplwv*nkpts)*kind(datake) + 
     & (nrplwv*nkpts)*kind(nindpw)

      nblocks=(nbytes+4095)/4096

      print*,"Total file size required:", nbytes
      print*,"Total memory blocks (4096) required:", nblocks


      open(iounit,file="TEST_FILE",access='sequential',
     $        form='unformatted')


      write(iounit) cptwfp,datake,nindpw

      do n=1, niterations
        cptwfp= cmplx (0.0,0.0)
        datake=0.0
        nindpw=0

        rewind(iounit)
        read(iounit) cptwfp,datake,nindpw
 
        cptwfp = cptwfp + cmplx(1.0,1.0)
        datake = datake + 1.0
        nindpw = nindpw + 1

        rewind(iounit)
        write(iounit) cptwfp,datake,nindpw

        print*,"Iteration:",n," cptwfp:", cptwfp(1,1,1)," datake:",
     &         datake(1,1,1)," nindpw:",nindpw(1,1)
      enddo

      close(iounit)

      end

SV1:
        ftn -o test_mr test_mr.f

X1:
        ftn -sdefault64 -o test_mr test_mr.f

==================================================================
Start test without memory resident assignment:
# ASSIGN ENVIRONMENT FILE=/tmp/baring/.assign
Sun Sep 14 13:21:07 AKDT 2003
 Array sizes: (cptwfp,datake,nindpw)= 10440000,  240000,  60000
 Total file size required: 169440000
 Total memory blocks (4096) required: 41368
 Iteration: 1  cptwfp: (1.,1.)  datake: 1.  nindpw: 1
 Iteration: 2  cptwfp: (2.,2.)  datake: 2.  nindpw: 2
 Iteration: 3  cptwfp: (3.,3.)  datake: 3.  nindpw: 3
 Iteration: 4  cptwfp: (4.,4.)  datake: 4.  nindpw: 4
 Iteration: 5  cptwfp: (5.,5.)  datake: 5.  nindpw: 5
 Iteration: 6  cptwfp: (6.,6.)  datake: 6.  nindpw: 6
 Iteration: 7  cptwfp: (7.,7.)  datake: 7.  nindpw: 7
 Iteration: 8  cptwfp: (8.,8.)  datake: 8.  nindpw: 8
 Iteration: 9  cptwfp: (9.,9.)  datake: 9.  nindpw: 9
 Iteration: 10  cptwfp: (10.,10.)  datake: 10.  nindpw: 10
 Iteration: 11  cptwfp: (11.,11.)  datake: 11.  nindpw: 11
 Iteration: 12  cptwfp: (12.,12.)  datake: 12.  nindpw: 12
 Iteration: 13  cptwfp: (13.,13.)  datake: 13.  nindpw: 13
 Iteration: 14  cptwfp: (14.,14.)  datake: 14.  nindpw: 14
 Iteration: 15  cptwfp: (15.,15.)  datake: 15.  nindpw: 15
 Iteration: 16  cptwfp: (16.,16.)  datake: 16.  nindpw: 16
 Iteration: 17  cptwfp: (17.,17.)  datake: 17.  nindpw: 17
 Iteration: 18  cptwfp: (18.,18.)  datake: 18.  nindpw: 18
 Iteration: 19  cptwfp: (19.,19.)  datake: 19.  nindpw: 19
 Iteration: 20  cptwfp: (20.,20.)  datake: 20.  nindpw: 20

           seconds          clocks
elapsed  193.48645     19348644663
user      10.78702      1078701636
sys        5.69306       569305531
Sun Sep 14 13:24:20 AKDT 2003
Done
==================================================================
 
==================================================================
Start test without memory resident assignment:
# ASSIGN ENVIRONMENT VARIABLE=$EVAR
assign -F mr.scr.ovfl.start_size=41368 u:11
Sun Sep 14 13:24:20 AKDT 2003
 Array sizes: (cptwfp,datake,nindpw)= 10440000,  240000,  60000
 Total file size required: 169440000
 Total memory blocks (4096) required: 41368
 Iteration: 1  cptwfp: (1.,1.)  datake: 1.  nindpw: 1
 Iteration: 2  cptwfp: (2.,2.)  datake: 2.  nindpw: 2
 Iteration: 3  cptwfp: (3.,3.)  datake: 3.  nindpw: 3
 Iteration: 4  cptwfp: (4.,4.)  datake: 4.  nindpw: 4
 Iteration: 5  cptwfp: (5.,5.)  datake: 5.  nindpw: 5
 Iteration: 6  cptwfp: (6.,6.)  datake: 6.  nindpw: 6
 Iteration: 7  cptwfp: (7.,7.)  datake: 7.  nindpw: 7
 Iteration: 8  cptwfp: (8.,8.)  datake: 8.  nindpw: 8
 Iteration: 9  cptwfp: (9.,9.)  datake: 9.  nindpw: 9
 Iteration: 10  cptwfp: (10.,10.)  datake: 10.  nindpw: 10
 Iteration: 11  cptwfp: (11.,11.)  datake: 11.  nindpw: 11
 Iteration: 12  cptwfp: (12.,12.)  datake: 12.  nindpw: 12
 Iteration: 13  cptwfp: (13.,13.)  datake: 13.  nindpw: 13
 Iteration: 14  cptwfp: (14.,14.)  datake: 14.  nindpw: 14
 Iteration: 15  cptwfp: (15.,15.)  datake: 15.  nindpw: 15
 Iteration: 16  cptwfp: (16.,16.)  datake: 16.  nindpw: 16
 Iteration: 17  cptwfp: (17.,17.)  datake: 17.  nindpw: 17
 Iteration: 18  cptwfp: (18.,18.)  datake: 18.  nindpw: 18
 Iteration: 19  cptwfp: (19.,19.)  datake: 19.  nindpw: 19
 Iteration: 20  cptwfp: (20.,20.)  datake: 20.  nindpw: 20

           seconds          clocks
elapsed    9.11426       911426168
user       8.94074       894073879
sys        0.12400        12399749
Sun Sep 14 13:24:29 AKDT 2003
Done
==================================================================

Comments on the times:

The total runtime dropped on chilkoot from 193 seconds to 9. This was on a non-dedicatied system, so it had unknown competition for both memory and disk I/O bandwidth. The speedup on the X1 is more dramatic, but, since the X1 isn't open quite yet, I figured I'd better keep those numbers to myself for now.

A:[[ I like "mget" and "mput" in ftp, but I'm sick of answering "y", "y",
  [[ "y", "y", "y", "y", "y, "y", "y", "y", "y"... when I know I want ALL
  [[ the files!  You may have experienced it...  
  [[ 
  [[ ...
  [[ 
  [[ So I often log onto the remote system (when the files are in my own
  [[ account, of course), make a tar file, and just "get" the tar file.  Is
  [[ there another way?

# 
# In the order received, thanks to Kunho Kim, Greg Newby, Matt MacClean,
# Paul Mercer, Nic Brummell, Steve Munk, Jed Brown, Mark Dalton, Shawn
# Houston, and Kurt Carlson.
# 

The overwhelming favorite is: 

  Type "prompt".   It toggles the prompting off and on.
  
    ftp> prompt
    Interactive mode off.
    ftp> mget *
    
  will get you all the files with no typing.


Other suggestions from the respondants: 

  1) 'ftp -i'  
     From "man ftp":
       -i   Turns off interactive prompting during multiple file 
            transfers.
  
  2) ftp>  prompt off
     (Forget about toggling.  Just turn it off.)  
  
  3) Use your .netrc file to make "prompt"-off the default for all 
     sessions, or just to particular systems.  (For instructions on 
     using .netrc files, see: the QT answer in issue #273.)
  
  4) Instead of ftp, consider scp or sftp.  These are programs that layer
     over ssh, and therefore don't send your password or data
     unencrypted.  sftp has virtually the same syntax as ftp.  scp is
     like rcp (an old Berkeley "r" command), you can send wildcards.
     So, if you wanted all the *.f files in your "test" subdirectory
     from systemb.uaf.edu, try:

       scp username@systemb.uaf.edu:"test/*.f"   

     where "username" is your username.  This won't work, and isn't
     necessary, if you're using a Kerberized FTP which is likely to be
     the case with ARSC systems.  Kerberos will encrypt the session for
     you...on such a system, you can probably use a kerberized rcp
     instead, with similar syntax to the above.

  5) Guy Robinson actually prefers the tar file solution.  "It should 
     do a lot better in terms of bandwidth, and is a good way to take
     consistent groups of data and keep records, etc."


Q:#
  # Many thanks to the reader who submitted this question:
  # 

  I develop software in collaboration with other people. Each time they
  have a new version I have to carefully study it for changes. Most of
  them are small but important. I'm looking for ways to automatize the
  parts of the process that do not require much thinking.
  
  The best I found is to run the diff comand between files of the old and
  new version, study the differences and manually write the new code.  Is
  there a way to pipe the output of the diff command to a tool that would
  let me more or less automatically reject or accept a change?

[[ Answers, Questions, and Tips Graciously Accepted ]]

Thomas J. Baring	ARSC Web Specialist	ph: 907-450-8619
Donald Bahls	ARSC User Consultant	ph: 907-450-8674
Arctic Region Supercomputing Center
University of Alaska Fairbanks
PO Box 756020
Fairbanks AK 99775-6020

Send comments and questions to the current editors using this Contact Form.

To subscribe to the e-mail edition of the ARSC HPC Users' Newsletter, send this message to: "majordomo@arsc.edu":
    subscribe hpc_users
    end
For help with majordomo, send this:
    help
    end
Leave the "subject" line of your message blank.

The ASCII e-mail edition of the ARSC T3D/T3E/HPC Users' Newsletter is archived and available by anonymous FTP to: ftp://ftp.arsc.edu/pub/publications/newsletters/

Arctic Region Supercomputing Center
PO Box 756020, Fairbanks, AK 99775 | voice: 907-450-8600 | email:

home | search | about | support | news | science | resources