ARSC T3D Users' Newsletter 107, October 4, 1996

T3D Memory Layout

[ Thanks to Mark Dalton of CRI in Los Alamos for sending this in (I have made a few minor changes). The T3D Newsletter has touched on memory management in several previous issues (see the list at the end), but has not discussed the mppldr directives before. ]

The manpage on 'mppldr' describes four loader directives which can be used to control the T3D memory layout. The manpage description:

     defheap     Specifies the minimum heap size and heap increment value
                 for all programs.
     defsheap    Used in the default directives file to establish a minimum
                 heap size and heap increment value for all programs.
     defstack    Sets the default program stack size.
     defsstk     Used in the default directives file to establish a minimum
                 distributed stack size for all programs.

My description:

                    -i mppldr_opts
                       where the file "mppldr_opts" contains:
                       defheap=#           <-- Initial Private Heap
                       defsheap=#          <-- Initial Shared Heap
                       defstack=#          <-- Initial Private Stack
                       defsstck=#          <-- Initial Shared Stack

For Example:

cc -X <Num_PEs> -Wl&quot;-i mppldr_opts&quot; file_name.c

       mppsize ./a.out  <-- The executable must have a fixed number of PE's.

NOTE:  You must fix the Number of PE's or it will not give you useful
       information on the actual size of the executable.  You can do this 
       at compile/link time, or use mppfixpe.

The following shows sample output from mppsize (taken from the Cray Research Service Bulletin on mppsize):


Startup program is:     #!/mpp/bin/mppexec

Number of PEs:           1 (fixed)
OS partition required?  no
H/W partition required? no
               barriers: 1
                eurekas: 0
Shape hint (0,0,0)
Transfer address:       2000000000
Segment                   Size T S P G A L     D  Disk   Length   Offset Zeroed
2000000000          code B1DA0 T N 5 0 N 0     0   570    B1DA0        0      0
A000000000   shared data   C50 D Y 6 0 N 0  4C50 CE268        0        0    850
4000000000  private data DE280 D N 6 + N 0 E2280 B2310    1BF58        0   2328
6000000000 private stack C0000 S N 6 - Y 0 C4000     0        0        0      0
8200000000     registers     0 R N 6 0 N 0     0     0        0        0      0
        Symbols                                   CE268    23B58       0
        Dot o's                                       0        0       0
     Directives                                       0        0       0
   843000 (decimal) bytes will be initialized from disk.
   146264 (decimal) bytes are required for the symbol table.
     1392 (decimal) bytes are required for the header.
   990656 (decimal) bytes total


The CRSB on mppsize appears in Newsletter 27 , and describes how to interpret this information. Basically, add up the size of: code, shared data, private data, and private stack. These are the default size for the data and stacks and they can allocate more, depending on the code. However some codes will need more of a particular area.

For more information:

  • Use 'docview' or 'cdoc' to view the mppldr section of the segldr manual.
  • man mppldr, mppsize, mppfixpe, IHPSTAT
  • For mppsize: Newsletters 23 , 27
  • For IHPSTAT: Newsletter 97
  • For mppfixpe: Newsletter 101
  • For mppldr directives: Newsletter 103

Barriers AND Eurekas Revisited

Two points:

  1. Barriers on the T3E:

    Frank Chism of CRI tells me that he has run some of his barrier testing codes on the T3E, and they perform as expected. That is, as they perform as on the T3D.

  2. A barrier call on a PE invalidates a eureka event set previously on that PE.

    The following one-sentence paragraph is from docview, mpp.fortran.62, and appeared (with context) in Newsletter 104 :

    "Events in eureka mode cannot be posted across barriers."

The manual didn't explain what would happen if one violated this policy, but, as you can imagine, it happened. A reader sent in a test which used barriers and eurekas and seemed to behave inconsistently. It turns out that its barrier call was cancelling an event.

The following program attempts to show this situation. Since it is testing the barrier network, it must avoid using it. To synchronize the PEs, it, therefore, uses the sleep command: there may be a better approach, but this seems to work.

Here's what it does:

All PEs clear the eureka event (clear_event() is always required). A barrier call is included in the clear_event() call, so we know we're synched at that point. Then,

  1. PEs 2 and 3 test for a eureka event
  2. PE0 sets a eureka event
  3. PEs 2 and 3 test
  4. PE1 sets a barrier
  5. PEs 2 and 3 test
  6. PE0 sets a barrier
  7. PEs 2 and 3 test

The output:

  === PE0: before PE0 does set_event
  PE2: test_event returns FALSE
  PE3: test_event returns FALSE
  === PE0: after PE0 does set_event / before PE1 does set_barrier
  PE2: test_event returns TRUE
  PE3: test_event returns TRUE
  === PE1: after PE1 does set_barrier
  PE3: test_event returns TRUE
  PE2: test_event returns TRUE
  === PE0: after PE0 does set_barrier
  PE2: test_event returns FALSE
  PE3: test_event returns FALSE

The program:

#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>
void print_test_results (int self);

int main( int argc, char** argv ) {
        int npes,self;
        npes = _num_pes();
        self = _my_pe();
        clear_event();  /* contains barrier */

        if( self == 0 ) {
                printf("=== PE%i: before PE0 does set_event\n",self);
                sleep (2);

                printf("=== PE%i: after PE0 does set_event / before PE1 does set_barrier\n",self);
                sleep (2);

                /* PE1 sets its barrier here. */
                sleep (2);

                printf("=== PE%i: after PE0 does set_barrier\n",self);
        else if( self == 1 ) {
                sleep (2);

                /*  PE0 does set_event() here  */
                sleep (2);

                printf("=== PE%i: after PE1 does set_barrier\n",self);

                sleep (2);
                /* PE0 sets its barrier here */
        else {
                sleep (1);
                print_test_results (self);
                sleep (2);
                print_test_results (self);
                sleep (2);
                print_test_results (self);
                sleep (2);
                print_test_results (self);
void print_test_results (int self) {
/*        if( test_barrier() ) 
                printf("PE%i: test_barrier returns TRUE\n",self);
                printf("PE%i: test_barrier returns FALSE\n",self);
        if( test_event() ) 
                printf("PE%i: test_event returns TRUE\n",self);
                printf("PE%i: test_event returns FALSE\n",self);

Quick-Tip Q & A

A: {{ What's an easy way to remove all the 'core' and 'mppcore' files in
      any of your directories (but have 'rm' ask before removing)? }}

    # This 'find' command starts its search from "." -- the current
    # directory.  It descends recursively into all subdirectories,
    # searching for files named core or mppcore (the "-o" specifies "or",
    # and the escaped parentheses are required). When it finds one, it
    # executes the command "rm -i {}", substituting the complete path of
    # the file for the token, "{}".

    find . \( -name core -o -name mppcore \) -exec rm -i {} \;

    # A reader sent in this:

    I use a ksh function in my .profile file:

      menage () { find . \( -name core -o -name '*.o' -o -name mppcore -o 
         -name '*.l' -o -name '*.T' \) -atime +7 -exec $* {} \; ; }

    "-fstype nfs" is able to exclude NFS files but does not work on
    CRAY.  It's a shame, because this function can be dangerous if you
    access NFS files you did not want to remove after having forgotten

        menage ls -l
        menage rm -i

Q: How do you figure out what version of a library your code has been
   loaded with, or what version of a library you are using?

[ Answers, questions, and tips graciously accepted. ]

Current Editors:
Ed Kornkven ARSC HPC Specialist ph: 907-450-8669
Kate Hedstrom ARSC Oceanographic Specialist ph: 907-450-8678
Arctic Region Supercomputing Center
University of Alaska Fairbanks
PO Box 756020
Fairbanks AK 99775-6020
E-mail Subscriptions: Archives:
    Back issues of the ASCII e-mail edition of the ARSC T3D/T3E/HPC Users' Newsletter are available by request. Please contact the editors.
Back to Top