ARSC T3D Users' Newsletter 14, December 9, 1994

Upcoming T3D Software Upgrade

ARSC will be upgrading the T3D software to CrayLib_M 1.1.1.2, MAX 1.1.0.4 and SCC_M 4.0.2.11 on December 11th.

By now, users should have run their programs and saved the executables and results. These executables and results should be compared to the results after the new products are installed. If there are erroneous differences then Mike Ess should be contacted.

PE Limits

Next week we will be sending out notification by e-mail of PE limit changes. The changes will go into effect one week after the e-mail is sent. To have us hold up on this change or to increase the limits in the future, please contact Mike Ess by e-mail or phone.

At ARSC, most production runs can be handled by the 32 PE NQS queue. The use of large partitions of 64 or 128 PEs seems restricted to research problems of the "what if..." type. Basically, development is done on smaller partitions and then to complete a table of performance or problem sizes, the larger partitions are used.

We have been accommodating this situation by individually changing a user's interactive permissions with the understanding that users would share their results with us. We are in the process of notifying users of the change in this policy. In the future we will reset everyone back to the default settings and then make it easy for users to request changes that will be in effect for a short time period (perhaps a week per request).

Shmem Calls - Corrections and Improvements

Corrections

In the ARSC T3D 12th Newsletter I had some timings and experience with using a spin wait on a shared variable. Frank Chism of CRI modified my original code to use a barrier call instead of my spin waits and that gave me a chance to back over the original code. The original code had at least three mistakes:
  1. The arrays b and c were not initialized
  2. The order arrays in the shmem call should have been reversed
  3. The IF test for correctness was never executed
Here is the original code:

      parameter( NMAX = 100 000 )
      real a( NMAX ), b( NMAX ), c( NMAX )
      INTRINSIC MY_PE
      integer DONE
CDIR$ SHARED DONE
      DONE = 0
      do 10 i = 1, NMAX
        a( i ) = i
   10 continue
      me = MY_PE()
      n = 1 
      if( me .eq. 0 ) then
        do 100 i = 1, 6
           t1 = second( )
              call shmem_get( a, b, n, 0 )
           t2 = second( )
              call shmem_get( a, c, n, 1 )
           t3 = second( )
           rmb = 8*n/1000000.0
           write( 6, 600 ) i, n, t2-t1, t3-t2, rmb/(t2-t1), rmb/(t3-t2)
           n = n * 10
  100 continue
      DONE = 1
      else
    1 continue
      if( DONE .eq. 0 ) goto 1
      endif
  600 format( i4, i7, f10.6, f10.6, f10.2, f10.2 ) 
      do 200 i = 1, NMAX
        if( i .ne. b( i ) ) then
          write( 6, 601 ) i, a( i ), b( i )
          stop
        endif
        if( i .ne. c( i ) ) then
          write( 6, 602 ) i, a( i ), b( i )
          stop
        endif
  200 continue
  601 format( "b is bad ", i10, f10.1, f10.1 )
  602 format( "c is bad ", i10, f10.1, f10.1 )
      end
I could have found mistakes 1. and 2. myself, if I hadn't made the mistake 3. The problem is that the statement:

      if( i .ne. b( i ) ) then
is always false (as originally written) but the code hung before this. By the time I narrowed the problem down to the spin wait, I had removed the check for correct answers. It's a convoluted world.

Improvements

Frank Chism correctly points out that in coding the spin wait I was actually reinventing the wheel of parallel processing because that's what barriers are for. Using his ideas and wishing to time more than just communication on a single node (PE0 and PE1 are on the same node), I expanded the original program to:

      program sh
      parameter( NMAX = 100 000 )
      real a( NMAX ), b( NMAX ), c( NMAX ), d( NMAX ), e( NMAX) 
      real f( NMAX ), g( NMAX ), h( NMAX ), p( NMAX )
      INTRINSIC MY_PE
      do 10 i = 1, NMAX
           a( i ) = i 
           b( i ) = 0.0
           c( i ) = 0.0
           d( i ) = 0.0
           e( i ) = 0.0
           f( i ) = 0.0
           g( i ) = 0.0
           h( i ) = 0.0
           p( i ) = 0.0
   10 continue
      me = MY_PE()
      n = 1 
      if( me .eq. 0 ) then
        do 100 i = 1, 6
          t1 = second( )
          call shmem_get( b, a, n, 0 )
          t2 = second( )
          call shmem_get( c, a, n, 1 )
          t3 = second( )
          call shmem_get( d, a, n, 2 )
          t4 = second( )
          call shmem_get( e, a, n, 3 )
          t5 = second( )
          call shmem_get( f, a, n, 4 )
          t6 = second( )
          call shmem_get( g, a, n, 5 )
          t7 = second( )
          call shmem_get( h, a, n, 6 )
          t8 = second( )
          call shmem_get( p, a, n, 7 )
          t9 = second( )
          rmb = 8*n/1000000.0
          write(6,600)i,n,rmb/(t2-t1),rmb/(t3-t2),rmb/(t4-t3), 
     +          rmb/(t5-t4),rmb/(t6-t5),rmb/(t7-t6), 
     +          rmb/(t8-t7),rmb/(t9-t8)
          n = n * 10
  100 continue
      do 200 i = 1, NMAX
             if( a( i ) .ne. b( i ) ) then
               write( 6, 601 ) i, a( i ), b( i )
               stop
             endif
             if( a( i ) .ne. c( i ) ) then
              write( 6, 602 ) i, a( i ), c( i )
              stop
            endif
             if( a( i ) .ne. d( i ) ) then
              write( 6, 603 ) i, a( i ), d( i )
              stop
            endif
             if( a( i ) .ne. e( i ) ) then
              write( 6, 604 ) i, a( i ), e( i )
              stop
            endif
             if( a( i ) .ne. f( i ) ) then
              write( 6, 605 ) i, a( i ), f( i )
              stop
            endif
             if( a( i ) .ne. g( i ) ) then
              write( 6, 606 ) i, a( i ), g( i )
              stop
            endif
             if( a( i ) .ne. h( i ) ) then
              write( 6, 607 ) i, a( i ), h( i )
              stop
            endif
             if( a( i ) .ne. p( i ) ) then
              write( 6, 608 ) i, a( i ), p( i )
              stop
            endif
  200 continue
      endif
CDIR$ barrier
  600 format( i4, i7, 8f8.2 ) 
  601 format( "b is bad ", i10, f10.1, f10.1 )
  602 format( "c is bad ", i10, f10.1, f10.1 )
  603 format( "d is bad ", i10, f10.1, f10.1 )
  604 format( "e is bad ", i10, f10.1, f10.1 )
  605 format( "f is bad ", i10, f10.1, f10.1 )
  606 format( "g is bad ", i10, f10.1, f10.1 )
  607 format( "h is bad ", i10, f10.1, f10.1 )
  608 format( "p is bad ", i10, f10.1, f10.1 )
      end

      real function second( )
      second = float( irtc( ) ) / 150000000.0
      end
The program now times shmem_get calls on PE0 from PE0 to PE7, a barrier is used to synchronize PEs, and all the answers are checked for correctness.

The results are much better and more consistent that the times for the previous program with the spin wait. Of course the transfers on the same PE are fastest but there is a significant bump for PE1 to PE0 because they are on the same node. Why transfers from PE4 and PE5 are faster on all three runs below than transfers from PE2, PE3, PE6 and PE7 is a mystery to me. (Notice that optimization is on for this program.)


  /mpp/bin/cf77 -Wf"  " -c shmemb.f
  mppldr shmemb.o
  a.out

  Size of   
  transfer    Pe0   PE1   PE2   PE3   PE4   PE5   PE6   PE7

  1      1   0.43  1.24  1.25  1.21  1.23  1.28  1.27  1.25
  2     10   6.56  9.00  8.43  8.80  8.81  8.70  8.69  8.75
  3    100  40.97 29.78 27.62 27.40 28.69 28.97 27.77 27.66
  4   1000 105.95 38.35 34.89 34.86 36.31 36.31 34.88 34.86
  5  10000 114.05 38.85 33.75 35.05 37.01 37.03 35.07 33.97
  6 100000 116.46 38.67 34.83 34.94 36.85 36.89 34.94 34.88

  a.out 
  1      1   0.45  1.27  1.19  1.21  1.27  1.27  1.23  1.26
  2     10   6.50  9.06  8.58  8.72  8.91  9.02  8.63  8.55
  3    100  41.61 29.84 27.81 27.84 28.65 28.76 27.52 27.70
  4   1000 106.01 38.38 34.84 34.87 36.31 36.24 34.90 34.78
  5  10000 114.21 38.85 33.71 35.03 36.99 37.02 35.07 34.00
  6 100000 116.48 38.66 34.84 34.94 36.85 36.87 34.93 34.94

  a.out
  1      1   0.44  1.27  1.22  1.26  1.25  1.27  1.27  1.25
  2     10   6.39  8.94  8.49  8.80  8.75  8.84  8.67  8.75
  3    100  40.97 29.64 27.45 27.50 28.45 28.76 27.56 27.47
  4   1000 106.37 38.28 34.89 34.84 36.24 36.31 34.92 34.85
  5  10000 114.20 37.23 35.05 35.07 37.01 37.04 34.02 35.05
  6 100000 115.01 38.82 34.83 34.94 36.85 36.54 34.92 34.93

Reminders

List of Differences Between T3D and Y-MP

The current list of differences between the T3D and the Y-MP is:
  1. Data type sizes are not the same (Newsletter #5)
  2. Uninitialized variables are different (Newsletter #6)
  3. The effect of the -a static compiler switch (Newsletter #7)
  4. There is no GETENV on the T3D (Newsletter #8)
  5. Missing routine SMACH on T3D (Newsletter #9)
  6. Different Arithmetics (Newsletter #9)
  7. Different clock granularities for gettimeofday (Newsletter #11)
I encourage users to e-mail in differences that they have found, so we all can benefit from each other's experience.

Third Call for Test Programs

In putting together a regression test for upgrades of T3D software, I found two areas where I need more coverage with user supplied programs. I would like to ask again for users to submit test programs on:
  1. PVM called from Fortran
  2. Shmem routines called from either Fortran or C
Ideally each program would exercise many different routines, run for only minutes, produce a text file that provides timing information and could be diffed with previous results to show correct results. That is a lot to ask but that's what I need.

Future Newsletter

While at the past Supercomputing '94 conference I got a lot of information on the T3D. It will take me a while to outline it, but eventually you'll see it in the upcoming newsletters.
Current Editors:
Ed Kornkven ARSC HPC Specialist ph: 907-450-8669
Kate Hedstrom ARSC Oceanographic Specialist ph: 907-450-8678
Arctic Region Supercomputing Center
University of Alaska Fairbanks
PO Box 756020
Fairbanks AK 99775-6020
E-mail Subscriptions: Archives:
    Back issues of the ASCII e-mail edition of the ARSC T3D/T3E/HPC Users' Newsletter are available by request. Please contact the editors.
Back to Top