ARSC T3D Users' Newsletter 32, April 21, 1995

The Switch to the 1.2 Programming Environment

We are now running the 1.2 Programming Environment. If you have any problems that you think are related to this upgrade, please contact Mike Ess.

PVM on the 1.2 Programming Environment

In the past few newsletters I've been going over the changes with the ARSC T3D due to the switch to the 1.2 PE. In this newsletter, I concentrate on changes with PVM. There are 2 PVM products from CRI of interest:
  1. Network PVM that runs on the Y-MP, and allows the Y-MP to communicate with PVM to the T3D and other PVM platforms, like workstations.
  2. The PVM in /mpp/lib/libpvm3.a that runs on the T3D alone and allows processing elements to communicate with each other with PVM subroutines or functions.
CRI products follow the functionality of the publicly available PVM distributed by Oak Ridge National Labs. From the Programming Environment 1.2 and Compiler Products Releases letter we have the following differences for Network PVM:

  > 1.2.3 Network PVM-3 2.1 release dependency
  > Sites supporting Network PVM-3 and any of the following Cray MPP systems
  > programming environments: CF77 6.2, CF90 1.2, or Cray Standard 1.2, must
  > upgrade to the Cray Network PVM-3 2.1 release, or later. The reason for this
  > upgrade is that changes in libpvm3 1.2 for Cray MPP systems are incompatible
  > with older versions of Cray Network PVM-3. These changes affect users with
  > distributed applications that run partly on the Cray T3D.
  > Two C structures defined and used by libpvm3 have been changed (hostinfo
  > became pvmhostinfo and taskinfo became pvm taskinfo). This is an incompatible
  > change for those few codes that use these structures requiring a change and
  > recompilation. These changes were made in the ORNL version of PVM 3.3.

  > 2.4.1 PVM structure name changes
  > Description:
  > Two C structures defined and used by LIBPVM will change in this release. This
  > is an incompatible change for those few codes that use these structures.
  > OLD NAME NEW NAME USED BY hostinfo pvmhostinfo pvm_config() taskinfo pvmtaskinfopvm_tasks()
  > These changes were made in the ORNL version of PVM 3.3. The reason for the change
  > was a name conflict between PVM and a UNICOS socket-related include file.
  > In addition to the changes described above, the protocols used in building
  > PVM will also change. This change will affect users who run distributed applications
  > partly on Cray T3D systems. Because of these changes, sites supporting the
  > network version of PVM must upgrade to the Cray Network PVM-3
  > 2.1 release (or later) to run distributed applications.
  > Platform:
  > Cray MPP systems
  > Compatibilities and Differences:
  > This is an incompatible change. Products using these C structure will have
  > to change their names.
  > These structures are defined in the pvm3.h include file. Only the structure
  > names have changed. No other changes have been made to these structures.
  > External impact:
  > Users that use the hostinfo or taskinfo C structures will have to change their
  > code to call the new structure names. See "Description" for details.
  > Performance impact:
  > None
  > Documentation affected:
  > PVM and HeNCE Programmer's Manual, publication SR-2501
Users who use these structs hostinfo and taskinfo on their Y-MP PVM codes will have to make these because as part of upgrading to the 1.2 PE we upgraded to Cray Network PVM-3 2.1. Neither of these changes effect the T3D PVM.

My testing of PVM for this release showed up a few differences and additions from the 1.1 PE:

  1. When using network PVM to communicate between the Y-MP and T3D using the option:
      pvm_setopt( PvmRoute, PvmRouteDirect );
    in the Y-MP program caused a T3D program to crash and therefore the Y-MP program hangs waiting for a message from the crashed T3D program. This option works OK on T3D to T3D communication. And the default setting:
      pvm_setopt( PvmRoute, PvmAllowDirect );
    works correctly on Y-MP to T3D communication. In the 1.1 PE, the use of PvmRouteDirect did not crash the T3D program. This problem has been reported to CRI.
  2. In the 1.1 PE, I could spawn two T3D programs from the PVM console running on the Y-MP, the performance was comparable to the Y-MP to T3D speeds of less than 2 Mbytes/second, so it was mostly just a curiosity. With the 1.2 PE, I can now initiate 2 different T3D jobs in two different partitions and they can communicate using PVM. Done this way, the performance is closer to the T3D to T3D speeds in a single partition of 100 Mbytes/second. In my case, this allows the master and slave programs to be separate executables and use all of the 8MW of memory for their own purposes.
  3. At the Denver CUG, I heard about two new PVM routines in the 1.2 PE, that combine the the pack and send and the receive and unpack into one unified operation. In C, the functions are:
      pvm_psend() and pvm_precv()
    and the Fortran equivalents are:
    There are comprehensive manpages for these routines on Denali. Having fewer calls performing the same operation boosts message passing speeds. On T3D to T3D sends I have seen a 10% to 40% improvement for round trip communication between two PEs on short messages. The send routines should also be faster because they are "asynchronous" routines that return control to the sending processor before the receiving processor has processed the message. These routines are also part of the 3.3 version of PVM distributed by Oak Ridge National Labs.

Current Documentation for PVM on the T3D:

PVM and HeNCE Programmer's Manual, publication SR-2501 5.0 Parallel Virtual Machine (PVM) Reference Card, publication SQ-2512 5.0
pvm.hence.50 PVM and HeNCE Manual (SR-2501 5.0)
There is a very comprehensive set of manpages on Denali for PVM calls and functions.

Makefiles That Make Both Y-MP and T3D Executables

I have timing programs for several different PVM machines:
  1. T3D to T3D in the same partition
  2. T3D to T3D in different partitions
  3. T3D to Y-MP
  4. Y-MP to Y-MP (with PvmDataRaw and PvmDataDefault options)
I've always had trouble with the makefile for the third case because the executable was linked with either segldr or mppldr depending on the environmental variable TARGET. Previously, I would make the Y-MP executable with:

   setenv TARGET cray-ymp
and then change the TARGET variable with:

   setenv TARGET cray-t3d
and then make the T3D executable. The switch of the TARGET value is necessary because both segldr and mppldr use the value of the TARGET to locate their default libraries. But now, Mike Dority, an ARSC CRI site analyst, has shown me how to make both executables in the same makefile. Here is a sample makefile:

  # Start of makefile making both Y-MP and T3D executables
  SHELL = /bin/sh
  CCY-MP = cc -Tcray-ymp
  LDY-MP = segldr

  CCT3D = cc -Tcray-t3d -X $(NPES)
  LDT3D = /mpp/bin/mppldr

  CFLAGS = -O -c


  PVMDIR = /u1/uaf/ess/pvm3

  run:    timing timing_slave
          -rm /tmp/pvmd.736 /tmp/pvmd.736
          pvmd3 &
          sleep 1
          -(export MPP_NPES=$(NPES); timing )
          echo halt > halt
          -cat halt 

  timing: timing.c
          $(CCY-MP) $(CFLAGS) -I/usr/include/pvm3 timing.c
          (export $(TY-MP); $(LDY-MP) -o timing timing.o -lpvm3 )

  timing_slave:   timing_slave.c
          $(CCT3D) $(CFLAGS) -I/usr/include/mpp timing_slave.c
          (export $(TT3D); $(LDT3D) -o timing_slave timing_slave.o -lpvm3 )
          cp timing_slave $(PVMDIR)/bin/$(ARCH)

          -rm -f *.o timing timing_slave halt
  # end of makefile
This makefile has several useful features:
  1. The SHELL variable is the Bourne shell where environmental variables can be changed with the "export" command. (I can't get setenv to work within a makefile with the C shell and I'd like to see a working example with the C shell.) The parentheses are necessary in the steps using mppldr and segldr to apply the export to the command.
  2. The rules for the Y-MP and T3D executables change the TARGET variable for the appropriate target machine.
  3. The execution of this master(Y-MP)/slave(T3D) program goes in the following steps:
    1. PVM daemon started in the background (wait for it to start)
    2. Y-MP program (timing) started
      1. spawns T3D program (timing_slave)
      2. communicates with T3D program
      3. signals T3D program to die
    3. PVM daemon is killed by sending a halt command to the PVM console
This makefile is much more convenient than the almost by-hand method of changing environmental variables and invoking one rule at a time.

Correction - "It's pref not perf "

In newsletter #30 I described the utility "pref" that is in benchlib. Ted Clee of CRI points out that in that article, I used "perf" several times when I should have used "pref". It should be "pref" as is "prefetch".

Additional NQS Queues on the T3D

There are now 5 queues on Denali for accessing the T3D. The two newest queues facilitate using large PE configurations on the T3D. These two new queues will allow one 64 PE job and one 128 PE job to run for 5 minutes and 10 minutes respectively. The T3D queues on denali are now:

  ----------   --- --- --- --- --- --- --- --- --- ---
  m_8pe_24hr    4   0  yes  on  0   0   0   0   0   0 
  m_16pe_24hr   2   0  yes  on  0   0   0   0   0   0 
  m_32pe_24hr   1   0  yes  on  0   0   0   0   0   0 
  m_64pe        1   0  yes  on  0   0   0   0   0   0 
  m_128pe       1   0  yes  on  0   0   0   0   0   0 
If your batch PE limit is too small to access these new NQS queues, please contact Mike Ess to have the PE batch limits increased.

List of Differences Between T3D and Y-MP

The current list of differences between the T3D and the Y-MP is:
  1. Data type sizes are not the same (Newsletter #5)
  2. Uninitialized variables are different (Newsletter #6)
  3. The effect of the -a static compiler switch (Newsletter #7)
  4. There is no GETENV on the T3D (Newsletter #8)
  5. Missing routine SMACH on T3D (Newsletter #9)
  6. Different Arithmetics (Newsletter #9)
  7. Different clock granularities for gettimeofday (Newsletter #11)
  8. Restrictions on record length for direct I/O files (Newsletter #19)
  9. Implied DO loop is not "vectorized" on the T3D (Newsletter #20)
  10. Missing Linpack and Eispack routines in libsci (Newsletter #25)
  11. F90 manual for Y-MP, no manual for T3D (Newsletter #31)
I encourage users to e-mail in differences that they have found, so we all can benefit from each other's experience.
Current Editors:
Ed Kornkven ARSC HPC Specialist ph: 907-450-8669
Kate Hedstrom ARSC Oceanographic Specialist ph: 907-450-8678
Arctic Region Supercomputing Center
University of Alaska Fairbanks
PO Box 756020
Fairbanks AK 99775-6020
E-mail Subscriptions: Archives:
    Back issues of the ASCII e-mail edition of the ARSC T3D/T3E/HPC Users' Newsletter are available by request. Please contact the editors.
Back to Top