Most sources for CRAY T3E results give performance measurements for the CRAY T3E (300 MHz) or CRAY T3E-900 (450 MHz) models ([1],[2]). This paper presents results for the CRAY T3E-1200E (600 MHz), which differs from the earlier models in the following respects:

The clock speed of the microprocessor is 600 MHz.

The DRAM speed is 50 ns, vs. 60 ns in earlier models.

The support circuitry supports hardware read-ahead for stream operations (this is also true of the CRAY T3E-900).

1.0 Hardware overview

The CRAY T3E is a distributed memory multiprocessor with a globally addressable address space. The liquid-cooled model is scalable to eight cabinets, each housing one clock module and up to 34 processor modules. Each processor module contains eight nodes or processing elements (PEs), giving a maximum configuration of 272 PEs/cabinet. The PEs are site-configurable into three groups: application (APP) PEs, used to run parallel user jobs, command (CMD) PEs, used to run single-processor user jobs and to process interactive commands, and operating system (OS) PEs, used for system tasks. There must be at least two CMD and two OS PEs in a liquid-cooled system. For a fully-populated cabinet of 272 PEs, the typical configuration is 256 APP PEs and a total of 16 CMD and OS PEs. This ratio of application to support PEs scales to 2048 application processors.

Each node of a CRAY T3E system consists of an Alpha 21164 microprocessor, a system control chip, 64 MB - 2048 MB (*) of local memory, and a network router. The DEC Alpha 21164 is a superscalar microprocessor with 4-way instruction issue, an 8 KB primary data cache, an 8 KB instruction cache, and a 96 KB secondary cache for both data and instructions. The system logic operates at 75 MHz, and the microprocessor clock speed must be a multiple of this rate. Included on the control chip are a set of six hardware stream buffers, which provide prefetching and buffering for loads of consecutive cache lines from local memory, and an external register set (the "E-registers"), which are used for memory transfers. Although the main purpose of the E-registers is to handle inter-processor communication, they can also be used for non-cached access to local memory. A cache backmap maintains coherence of the local memory, caches, streams, and E-registers by flushing cached lines when a non-cached memory request is initiated.

(*) Most CRAY T3E systems have 128, 256, or 512 MB of memory per node.

There is one network router for each processor on a CRAY T3E system, in contrast to the CRAY T3D system architecture, in which two processors shared a network connection. As in the CRAY T3D, network routers are connected in a 3-D torus topology. Each network router has 16 uni-directional channels, two to connect to and from the support circuitry of its associated PE, two to connect to and from the I/O controller on the processor module, and 12 to connect to and from other nodes in the +/- x, +/- y, and +/- z directions of the 3-D torus. Torus links have a theoretical peak bandwidth of 650 MB/s in each direction.

The test platform for the benchmarks in this paper was the 72-processor CRAY T3E-1200E system at the National Environmental Supercomputing Center, operated by Lockheed Martin for the U. S. Environmental Protection Agency. This system was configured with 64 application PEs, 4 command PEs, and 4 OS PEs, each with 256 MB of memory. The operating system level was Unicos/mk 2.0.4.60 and the programming environment release level was 3.2.

2.0 Local memory bandwidth

The greatest measured bandwidths are obtained by assembly-coded kernels or library routines. Simple load/store tests are best written in assembly language because high-level languages can not easily express one-way data transfers except in the context of some other work. Table 1 shows the best measured bandwidth rates for cacheable loads and stores. The effect of the on-chip caches was minimized for these tests by loading or storing unit-stride vectors of millions of words until an asymptotic rate was reached. Cacheable store rates appear slow because the write-allocate secondary cache requires that stored data first be loaded into the cache, then updated and marked as dirty until flushed to memory at some later time, so the reported rate represents only half of the memory traffic. The load/store operation assumes different addresses for the load and store and is equivalent to a vector copy. Experiments with the relative memory addresses of the streams showed that separating the arrays by a multiple of 8192 words plus 64 or 128 words for avoiding cache conflicts gave the best performance.

Table 2 shows the best measured bandwidth rates for uncached loads and stores (typically called GETs and PUTs). These rates were measured using the one-sided E-register GET and PUT routines from benchlib ([2]) and the SHMEM library routines SHMEM_GET and SHMEM_PUT. In this context, "one-sided" means a transfer from local memory into the E-registers, or from the E-registers to local memory. A one-sided GET is a meaningless operation, but a one-sided PUT could have application in initializing a data area to a constant value. Because the data never makes it to the Alpha microprocessor during an E-register operation, the uncached load and store rates are useful only for modeling memory-to-memory transfers such as a vector copy or a matrix transpose.

A somewhat more standard measure of "sustainable" local memory bandwidth is the STREAM benchmark ([10]). The single-processor STREAM benchmark program evaluates the performance of the local memory system on the following unit-stride operations:

Name	Operation	Bytes transferred (at 8 bytes/word)
Copy	c(1:n) = a(1:n)	8*(16*n)
Scale	b(1:n) = s*c(1:n)	8*(16*n)
Add	c(1:n) = a(1:n) + b(1:n)	8*(24*n)
Triad	a(1:n) = b(1:n) + s*c(1:n)	8*(24*n)

The value of n is chosen large enough to exceed the size of any local caches. Optimization by use of assembly language kernels or library routines is not allowed. When STREAM benchmark results are given for parallel systems, they include the time to perform the local memory operations and the time to synchronize all the participating processors or processes.

Table 3 shows the best measured STREAM benchmark rates for each operation. The size n was 2000000 elements for all tests, but different values of the "offset" parameter, which helps to control the relative memory addresses, were optimal for the different cases.

3.0 Inter-processor bandwidth

Latency and bandwidth data extracted from the COMMS1 benchmark results are shown in Table 4. For the MPI tests, setting the environment variable MPI_BUFFER_MAX to 2048, thereby limiting buffering to messages less than 2048 bytes, improved the performance by 75% for the larger transfers. For the PVM tests, setting the environment variable PVM_DATA_MAX to 32000 (as suggested in [5]) improved the performance for messages up to that length, but did not affect the asymptotic rate.

The SYNCH1 benchmark measures the overhead for global synchronization by measuring the rate at which a barrier statement can be executed as a function of the number of processes (nodes) taking part in the global barrier synchronization. The SYNCH1 benchmark repeats a sequence of 10 BARRIER statements 1000 times.

4.0 Single processor performance

The libsci BLAS can attain near peak performance when operating on data in the cache. To illustrate, we performed some basic vector operations (1-norm, 2-norm, vector sum, and dot product) using 64-bit Level 1 BLAS with vectors sized to fit in the cache, and repeated the tests many times to get an in-cache performance rate. The results are shown in Table 6. The performance of SNRM2 is the same as SASUM despite doing twice as much work because a 2-pass algorithm is used in SNRM2 to avoid underflow or overflow during the sum of squares ([3]).

Compared to other distributed memory systems, the CRAY T3E has a relatively small cache on each node. To highlight the advantages of a larger cache, some vendors will only show the performance of library routines from the cache. However, when data is not already cached, the asymptotic performance rate is a better predictor of the performance that will actually be achieved. Table 7 shows asymptotic performance rates for representative libsci BLAS operations on 64-bit vectors and matrices.

Besides the Basic Linear Algebra Subprograms, the Cray Scientific Library also contains software from LAPACK (a Fortran library of linear algebra software), special solvers such as first- and second-order linear system solvers, and FFTs. A sample of the performance from some of the block algorithms in LAPACK, taken directly from the LAPACK timing program, is shown in Table 8. Although signficantly below the matrix multiply rate and probably optimizable, these results may be more indicative of the performance of Level 3 BLAS in the context of a user program. Table 9 shows the performance of the FFT library routine CCFFT, including both an in-cache performance rate and a rate with all the vectors starting from memory. The megaflop rate is based on an operation count of 5n log(n) for each complex-to-complex FFT. Parallel versions of some of this software are also available through the ScaLAPACK library and several well-optimized 2-D and 3-D FFTs.

A number of common memory-to-memory transfers can be performed more efficiently by using the E-registers to bypass the cache. Examples include block copies, matrix transposes, data initialization, and gather/scatter operations ([2]). Support for this type of data movement is provided through the SHMEM and benchlib libraries or use of the CACHE_BYPASS compiler directive. The next several tables summarize their various implementations and compare their performance to that of regular Fortran using the cache. Some of the operations do not have a straightforward implementation in every method.

      DO I = 1, N
         Y(I) = X(I)
      END DO

!DIR$ CACHE_BYPASS X, Y
      DO I = 1, N
         Y(I) = X(I)
      END DO

      DO I = 1, N, 480
         NI = MIN( 480, N-I+1 )
         CALL LGETV( X(I), 1, NI )
         CALL LPUTV( Y(I), 1, NI )
      END DO

      CALL SHMEM_GET( Y, X, N, MYPE )

      DO J = 1, N
         DO I = 1, N
            B(I,J) = A(J,I)
         END DO
      END DO

      DO J = 1, N
!DIR$ CACHE_BYPASS A, B
         DO I = 1, N
            B(I,J) = A(J,I)
         END DO
      END DO

      DO J = 1, N, 480
         NJ = MIN( N-J+1, 480 )
         DO I = 1, N
            CALL LGETV( A(I,J), LDA, NJ )
            CALL LPUTV( B(J,I), 1, NJ )
         END DO
      END DO
  123 IF( LPUTP().NE.0 ) GOTO 123

      DO J = 1, N
         CALL SHMEM_IGET( B(1,J), A(J,1),
     &      1, LDA, N, MYPE )
      END DO

      DO J = 1, N-1
         DO I = J+1, N
            TMP = A(I,J)
            A(I,J) = A(J,I)
            A(J,I) = TMP
         END DO
      END DO

      DO J = 1, N-1
!DIR$ CACHE_BYPASS A
         DO I = J+1, N
            TMP = A(I,J)
            A(I,J) = A(J,I)
            A(J,I) = TMP
         END DO
      END DO

      DO J = 1, N-1
         DO I = 1, N-J, 240
            NI = MIN( (N-J)-I+1, 240 )
            CALL LGETVO( A(J+I,J), 1, NI, 0 )
            CALL LGETVO( A(J,J+I), LDA, NI, 240 )
            CALL LPUTVO( A(J,J+I), LDA, NI, 0 )
            CALL LPUTVO( A(J+I,J), 1, NI, 240 )
         END DO
      END DO
  123 IF( LPUTP().NE.0 ) GOTO 123

      DO J = 1, N-1
         CALL SHMEM_IGET( B, A(J,J+1),
     &      1, LDA, N-J, MYPE )
         CALL SHMEM_IGET( A(J,J+1), A(J+1,J),
     &      LDA, 1, N-J, MYPE )
         CALL SHMEM_PUT( A(J+1,J), B, N-J, MYPE )
      END DO

      DO I = 1, N
         X(I) = SUM
      END DO

!DIR$ CACHE_BYPASS X
      DO I = 1, N
         X(I) = SUM
      END DO

      CALL LSETV( X, 1, N, SUM )
  123 IF( LPUTP().NE.0 ) GO TO 123

      DO I = 1, NGATH
         X(I) = A(INDEX(I))
      END DO

!DIR$ CACHE_BYPASS A
      DO I = 1, NGATH
         X(I) = A(INDEX(I))
      END DO

      DO I = 1, NGATH, 480
         NI = MIN( NGATH-I+1, 480 )
         CALL LGATH( A(0), INDEX(I), NI )
         CALL LPUTV( X(I), 1, NI )
      END DO
  123 IF( LPUTP().NE.0 ) GOTO 123

      CALL SHMEM_IXGET( X, A(0), INDEX,
     &   NGATH, MYPE )

5.0 Compiler optimizations

The LFK authors stipulate that statistics should be quoted from the summary table of 72 timings in the output file. The baseline performance of the LFK tests, using only the compiler options f90 -O2, was as follows:

         Maximum   Rate =    596.9183 Mega-Flops/Sec.
         Quartile  Q3   =    157.0089 Mega-Flops/Sec.
         Average   Rate =    143.0450 Mega-Flops/Sec.
         GEOMETRIC MEAN =    115.8271 Mega-Flops/Sec.
         Median    Q2   =    103.9888 Mega-Flops/Sec.
         Harmonic  Mean =     95.1324 Mega-Flops/Sec.
         Quartile  Q1   =     84.7023 Mega-Flops/Sec.
         Minimum   Rate =     19.5352 Mega-Flops/Sec.

We added compiler options one at a time in order to study the effects of each on performance. These experiments can be summarized by looking at the geometric means from each test:

Options	Rate [Mflop/s]
f90 -dp -O3	122
f90 -dp -O3,unroll2	147
f90 -dp -O3,unroll2,pipeline2	151
f90 -dp -O3,unroll2,pipeline2 -a pad	142
f90 -dp -O3,unroll2,pipeline2 -lmfastv	151
f90 -dp -O3,unroll2,pipeline2,split2 -lmfastv	147

Although there was no measurable difference from adding the -lmfastv option, it is effective in some situations ([2]), so we quote the full statistics for the case f90 -dp -O3,unroll2,pipeline2 -lmfastv:

         Maximum   Rate =    632.9465 Mega-Flops/Sec.
         Quartile  Q3   =    214.1753 Mega-Flops/Sec.
         Average   Rate =    189.2878 Mega-Flops/Sec.
         GEOMETRIC MEAN =    151.3626 Mega-Flops/Sec.
         Median    Q2   =    147.5358 Mega-Flops/Sec.
         Harmonic  Mean =    122.4048 Mega-Flops/Sec.
         Quartile  Q1   =     97.0975 Mega-Flops/Sec.
         Minimum   Rate =     31.2659 Mega-Flops/Sec.

Previous studies have indicated that loop splitting is best applied on a loop-by-loop basis, rather than globally through use of the compiler's -Osplit flag ([2]). Also, the compiler is typically unable to recognize when a scientific library substitution can be made. In the remainder of this section, specific hand optimizations to individual kernels are described that illustrate further opportunities for performance improvements.

Kernel 3: library substitution

Kernel 3 consists of the following three lines of code:

          Q= 0.000d0
          DO 3 k= 1,n
        3      Q= Q + Z(k) * X(k)

It can be replaced by a library call:

          Q = SDOT( N, Z, 1, X, 1 )

The library routine is faster than the compiler-generated code when the vectors X and Z are in the cache because the library routine is more effective at hiding the latency of an Scache load. In the LFK test suite, the vectors do become cache resident because each test is executed several times. The statistics for kernel 3 for each do span length are as follows:

 KERNEL FLOPS  MICROSEC  MFLOP/SEC SPAN WEIGHT  CHECK-SUMS            PRECIS
 ------ -----  --------  --------- ---- ------  ---------------------- -----
  3 1.598E+06 8.714E+03    183.439   27   1.00  1.0555606580531002E-01 16.90
  3 2.141E+06 8.251E+03    259.508  101   2.00  3.9489293708756362E-01 16.90
  3 1.802E+06 6.233E+03    289.073 1001   1.00  3.9140054768099826E+00 16.90

After optimization:

  3 1.598E+06 7.190E+03    222.323   27   1.00  1.0555606580531003E-01 16.90
  3 2.141E+06 4.547E+03    470.892  101   2.00  3.9489293708756368E-01 16.90
  3 1.802E+06 2.142E+03    841.000 1001   1.00  3.9140054768099817E+00 16.65

Similarly, Kernel 24 can be replaced by a call to ISMIN with a small increase in performance.

Kernel 13: cache bypass

Kernel 13, a 2-D particle in cell computation, benefitted from the common block padding option, -a pad, but the performance of the full LFK benchmark set degraded slightly with this option. The kernel is

                fw= 1.000d0
 1013 DO  13     k= 1,n
                i1= P(1,k)
                j1= P(2,k)
                i1=       1 + MOD2N(i1,64)
                j1=       1 + MOD2N(j1,64)
            P(3,k)= P(3,k)  + B(i1,j1)
            P(4,k)= P(4,k)  + C(i1,j1)
            P(1,k)= P(1,k)  + P(3,k)
            P(2,k)= P(2,k)  + P(4,k)
                i2= P(1,k)
                j2= P(2,k)
                i2=            MOD2N(i2,64)
                j2=            MOD2N(j2,64)
            P(1,k)= P(1,k)  + Y(i2+32)
            P(2,k)= P(2,k)  + Z(j2+32)
                i2= i2      + E(i2+32)
                j2= j2      + F(j2+32)
          H(i2,j2)= H(i2,j2) + fw
   13 CONTINUE

A contributing factor to the poor performance of this kernel is the non-unit stride pattern of access to the arrays B and C. The compiler always generates cacheable loads because it can not tell what is in cache and what is not, but we can request non-cached access by preceding the above DO loop with the directive

!dir$ cache_bypass b, c

The baseline performance of kernel 13 was

 KERNEL FLOPS  MICROSEC  MFLOP/SEC SPAN WEIGHT  CHECK-SUMS            PRECIS
 ------ -----  --------  --------- ---- ------  ---------------------- -----
 13 1.389E+06 3.815E+04     36.407    8   1.00  8.7535991110238361E+09 16.90
 13 1.837E+06 4.805E+04     38.230   32   2.00  3.0307123384693829E+10 16.90
 13 1.613E+06 4.347E+04     37.102   64   1.00  4.5614594609595291E+10 16.90

With cache_bypass:

 13 1.389E+06 4.971E+04     27.940    8   1.00  8.7535991110238361E+09 16.90
 13 1.837E+06 3.972E+04     46.247   32   2.00  3.0307123384693829E+10 16.90
 13 1.613E+06 3.392E+04     47.541   64   1.00  4.5614594609595291E+10 16.90

The degradation for the smallest DO span is expected because the non-cached access requested by the cache_bypass directive will force cache-resident elements of B and C to be loaded from memory instead.

Kernel 14: loop combining

Kernel 14 is a 1-D particle in cell code consisting of three loops from 1 to n. The first loop has 7 streams and accesses two additional arrays (EX and DEX) in a non-sequential order. The second loop also has 7 streams, while the third uses only two, with some non-unit stride stores to a third array. In general, one wants to limit the number of streams to 6 or fewer. In this case, however, there was a lot of reuse within a loop iteration, so stripmining the outer loops and combining the three loops into two was the most beneficial.

The original code looked like this:

 1014 DO   141  k= 1,n
            VX(k)= 0.0d0
            XX(k)= 0.0d0
            IX(k)= INT(  GRD(k))
            XI(k)= REAL( IX(k))
           EX1(k)= EX   ( IX(k))
          DEX1(k)= DEX  ( IX(k))
 141  CONTINUE
c
      DO   142  k= 1,n
            VX(k)= VX(k) + EX1(k) + (XX(k) - XI(k))*DEX1(k)
            XX(k)= XX(k) + VX(k)  + FLX
            IR(k)= XX(k)
            RX(k)= XX(k) - IR(k)
            IR(k)= MOD2N(  IR(k),2048) + 1
            XX(k)= RX(k) + IR(k)
 142  CONTINUE
c
      DO  14    k= 1,n
      RH(IR(k)  )= RH(IR(k)  ) + fw - RX(k)
      RH(IR(k)+1)= RH(IR(k)+1) + RX(k)
  14  CONTINUE

The optimized code is as follows:

      do kk = 1, n, 128
         kn = min(n-kk+1,128)
      DO   141  k = kk, kk+kn-1
            VX(k)= 0.0d0
            XX(k)= 0.0d0
            IX(k)= INT(  GRD(k))
            XI(k)= REAL( IX(k))
           EX1(k)= EX   ( IX(k))
          DEX1(k)= DEX  ( IX(k))
            VX(k)= VX(k) + EX1(k) + (XX(k) - XI(k))*DEX1(k)
            XX(k)= XX(k) + VX(k)  + FLX
 141  CONTINUE
c
      DO   142  k = kk, kk+kn-1
            IR(k)= XX(k)
            RX(k)= XX(k) - IR(k)
            IR(k)= MOD2N(  IR(k),2048) + 1
            XX(k)= RX(k) + IR(k)
            RH(IR(k)  )= RH(IR(k)  ) + fw - RX(k)
            RH(IR(k)+1)= RH(IR(k)+1) + RX(k)
 142  CONTINUE
c
      end do

The baseline performance was

 KERNEL FLOPS  MICROSEC  MFLOP/SEC SPAN WEIGHT  CHECK-SUMS            PRECIS
 ------ -----  --------  --------- ---- ------  ---------------------- -----
 14 1.901E+06 3.049E+04     62.339   27   1.00  1.9943880114661271E+06 16.90
 14 2.222E+06 3.297E+04     67.387  101   2.00  2.3107401197908435E+07 16.90
 14 2.202E+06 7.043E+04     31.266 1001   1.00  2.1783317062516003E+09 16.65

Improvements with the optimized code ranged from 20-70%:

 14 1.901E+06 1.813E+04    104.834   27   1.00  1.9943880114661271E+06 16.90
 14 2.222E+06 2.062E+04    107.755  101   2.00  2.3107401197908435E+07 16.90
 14 2.202E+06 5.610E+04     39.254 1001   1.00  2.1783317062516003E+09 16.65

Although dramatic in a few cases, the overall effect of the hand optimizations was only about a 5% improvement in the average or mean performance rates:

         Maximum   Rate =    831.2763 Mega-Flops/Sec.
         Quartile  Q3   =    215.1152 Mega-Flops/Sec.
         Average   Rate =    198.2022 Mega-Flops/Sec.
         GEOMETRIC MEAN =    157.3169 Mega-Flops/Sec.
         Median    Q2   =    152.3032 Mega-Flops/Sec.
         Harmonic  Mean =    128.0855 Mega-Flops/Sec.
         Quartile  Q1   =    100.2081 Mega-Flops/Sec.
         Minimum   Rate =     29.4546 Mega-Flops/Sec.

6.0 Multiple processor computational performance

(*) LINPACK benchmark results are from the LINPACK benchmark report, and were not run at NESC.

The NAS parallel benchmarks (NPB) are a collection of eight benchmarks developed as part of the Numerical Aerodynamic Simulation (NAS) program at the NASA Ames Research Center to measure and compare the performance of highly parallel computers. These benchmarks, which are derived from computational fluid dynamic codes, have become a standard measure of supercomputer performance. Five of the benchmarks (EP, CG, MG, FT, and IS) represent computational kernels, while the other three (LU, SP, and BT) represent simplified applications. Three problem classes of increasing size are specified for each benchmark: class A, class B, and class C. Although source code implementations of the benchmarks in MPI now exist, the original specification, now called NPB 1.0, was a "pencil and paper" benchmark, in which the actual implementation was left to the computer vendor.

NPB 1.0 results are the most optimized versions, and it is those results (courtesy of the Cray Research benchmarking group) that are presented in Table 16, Table 17, and Table 18. The benchmarks vary widely in their communication requirements and patterns. EP has almost no communication, MG uses ghost-cell update communication and FT and BT use data set transpose algorithms. The drop-off in FT performance occurs when we run out of "planes" to do in parallel and have to do domain decomposition within a plane, which adds another communication step.

7.0 I/O performance

Each Fortran OPEN statement issued by a processor has its own file descriptor and counts against the per job limit of open files (currently 256 at NESC), even if it is accessing the same file as another processor. It is easy to exceed this limit when using the Cray global I/O layer if every processor opens several files. That is why designating one processor as the I/O processor, and using the fast inter-processor connection network to move data as needed, is often seen as the best alternative.

In order to quantify the I/O performance of the CRAY T3E, we will look at two metrics: the time for a global file OPEN, as would be necessary when doing global I/O to a particular file, and the maximum read and write bandwidth to disk from a single processor. The latter of these is somewhat specific to the current NESC configuration, but it is not atypical for a T3E system.

The command "df -p" shows the partitions available on the currently mounted filesystems. A filesystem with more than one partition may benefit from user-level striping. On the NESC CRAY T3E system at the time of these tests, the following configuration was in place for the /tmp disk:

/tmp (/dev/dsk/tmp     ):   3923683 sectors          505891 I-nodes
                   total:   6250000 sectors          524288 I-nodes

Big file threshold:            32768 bytes
Big file allocation minimum:      24 blocks

Allocation Strategy:    round robin files
                        round robin all user data

Primary partitions allocation unit:         16K byte blocks

part   start     total         free (%)          frags (%)      device
----  --------  --------  -----------------  ----------------  --------
   0         0   6250000   4230096 ( 67.7%)     24 (  0.002%)  tmp1    
   1   6250000   6250000   3872580 ( 62.0%)     35 (  0.004%)  tmp3    
   2  12500000   6250000   4062884 ( 65.0%)     18 (  0.002%)  tmp2    
   3  18750000   6250000   3529172 ( 56.5%)     55 (  0.006%)  tmp4    

Our test programs are taken from the NERSC I/O guide ([9]), modified to write or read an unformatted 100 MB file. As documented in the I/O guide, we use the assign command with options to specify the size of the file in 4KB blocks (-n 25600), the chunksize, a blocking parameter (-q 128), and the global I/O layer (-F bufa). Results for different numbers of partitions (specified with the -p command) are shown in Table 19.

Table 20 shows the best observed file open times for the 100 MB file in the previous exercise assuming that every processor opens the file before the file is created. The time to execute the OPEN statement varied widely from run to run and in some cases was an order of magnitude larger than these times. These data support the NERSC I/O guide's statement that "it could take a significant amount of time to open a large number of large files" ([9]). Indeed, the assignment that optimizes the file transfer rate may cause the file open time to be larger than the time to read or write the file itself.

Acknowledgements

References

[1]

Ed Anderson, Jeff Brooks, Charles Grassl, and Steve Scott, Performance [Analysis] of the CRAY T3E Multiprocessor, Proceedings of SC97, http://www.supercomp.org/sc97/proceedings/TECH/ANDERSON/INDEX.HTM, November 1997.

[2]

Ed Anderson, Jeff Brooks, and Tom Hewitt, The Benchmarker's Guide to Single-processor Optimization for CRAY T3E Systems, http://www.sgi.com/t3e/images/benchmark.pdf, June 1997.

[3]

Edward Anderson and Mark Fahey, Performance Improvements to LAPACK for the Cray Scientific Library, LAPACK Working Note 126, http://www.netlib.org/lapack/lawns/lawn126.ps, April 1997.

[4]

Michael Berry and Roger Hockney, Public International Benchmarks for Parallel Computers, PARKBENCH Committee Report 1, http://www.netlib.org/parkbench/, February 1994.

[5]

Cray Research, CRAY T3E Fortran Optimization Guide, SG-2518 3.0, http://www.cray.com/products/software/publications/, 1997.

[6]

Cray Research, Application Programmer's I/O Guide, 007-3695-005, http://www.cray.com/products/software/publications/, 1997.

[7]

Jack J. Dongarra, Performance of Various Computers Using Standard Linear Equations Software, University of Tennessee, report number CS-89-85, http://www.netlib.org/benchmark/performance.ps, April 1999.

[8]

Jack J. Dongarra, Hans W. Meuer, and Erich Strohmaier, Top500 Supercomputer Sites, University of Tennessee, report number CS-99-425, http://www.netlib.org/benchmark/top500.html, June 1999.

[9]

Richard A. Gerber, I/O on the NERSC Cray T3E, National Energy Research Scientific Computing Center, http://hpcf.nersc.gov/training/tutorials/T3E/IO/, July 1998.

[10]

John D. McCalpin, Sustainable Memory Bandwidth in Current High Performance Computers, http://www.cs.virginia.edu/stream/, October 1995.