ARSC T3D Users' Newsletter 90, June 7, 1996
Porting Heterogeneous Distributed Codes to the T3D
[Don Morton, visiting ARSC from Cameron University, contributes this article. See his contact information, provided below.]
Many of us come to the T3D with previously-written heterogeneous codes - a classic example is the master/slave paradigm in which a single, master process "controls" one or more slave processes. Since the T3D requires that we place the same executable on each Processing Element (PE) in a partition, our codes may require modification in order to execute on this platform.
A first approach is to make use of the Y-MP/T3D computing environment, running the "master" on the Y-MP and spawning slaves on the T3D, all easily done by making use of PVM. This is not necessarily the best way to do things, but it serves as a useful starting point. The programs master.f and slave.f, below, illustrate how this is done.
master.f
Since the master process is executed on the Y-MP, we need to insure that it is compiled with "Network PVM," much like the standard PVM many of us have become used to. We spawn the slave processes on the T3D, and note that only the first entry of the array "list" contains a valid TID. By default, the Y-MP will communicate only with PE0 on the T3D, and only PEs that can communicate with the Y-MP have valid TID's listed; the other entries of "list" contain a 1. There are ways to establish connections to other T3D PEs, but these methods rely on additional socket connections between the two machines. CRI recommends that programmers pass Y-MP/T3D data through PE0.Thus, the remainder of the master program involves initializing some data, sending it to PE0 on the T3D, then waiting for an acknowledgment, which will be sent by PE0 after the other slaves have received the message.
slave.f
In this example, slave.f is intended to run on the T3D in SPMD mode, and therefore makes use of the Cray-specific pvmfgetpe() function. For illustrative purposes only, all PEs call pvmfparent(), but because the default behavior has only PE0 communicating with the Y-MP, only PE0 will have a parent. PE0 receives the data from the Y-MP, then sends it on to the other slave processes. The slaves simply receive the data, then all T3D processes participate in the pvmfbarrier() call. On the T3D, the use of the group 'PVMALL' implies that all PEs in the partition participate - therefore, a "count" argument is not necessary. Finally, PE0 sends an acknowledgment to the Y-MP process and all slaves terminate.Closing Note
The Y-MP/T3D distributed programming model seems to be shunned by most, although in some cases it is an effective way of partitioning various workloads. For example, with careful programming, vector intensive computations can be performed on the Y-MP while MPP computations are performed on the T3D. However, communication between Y-MP and T3D processes is quite slow for many reasons. Additionally, those who are thinking of migrating to the T3E should consider the fact that there will no longer be a Y-MP front-end.Related discussion of Y-MP/T3D cooperation via PVM can be found in Newsletters #15 and #16.
program master:
include 'fpvm3.h'
parameter (NUMPROCS=5, ISIZE=1000)
dimension vec(ISIZE)
dimension list(0:NUMPROCS-1) ! For storing TID's
ccc Enroll in PVM
call pvmfmytid(mytid)
print *, 'Hello World, I am task ', mytid
ccc Spawn slaves
call pvmfspawn("slave", 0, '*', NUMPROCS-1, list, info)
print *, 'Task ', mytid, ' spawned ', info, ' slaves...'
print *, 'TID List of Slaves:'
do i=0,info-1
print *, list(i)
enddo
ccc Initialize a vector for transmission to slaves
do i=1,ISIZE
vec(ISIZE) = real(i)
enddo
ccc Send it to slave process on PE0
msgtag = 1
call pvmfinitsend(PVMDATADEFAULT, info)
call pvmfpack(REAL8, vec, ISIZE, 1, info)
call pvmfsend(list(0), msgtag, info)
print *, 'Task ', mytid, ' sent message...'
ccc Wait for ACK from PE0
msgtag = 2
call pvmfrecv(list(0), msgtag, info)
print *, 'Task ', mytid, ' received an ACK ...'
ccc exit PVM
call pvmfexit(info)
print *, 'Task ', mytid, ' exiting...'
end
program slave:
include 'fpvm3.h'
parameter (ISIZE=10)
dimension vec(ISIZE)
ccc Enroll in PVM
call pvmfmytid(mytid)
call pvmfgetpe(mytid, mype)
call pvmfparent(myparent)
print *, 'Hello World, I am task ', mytid
ccc By default, only PE0 is guaranteed to have the Y-MP process as
ccc a PVM parent. Therefore, all data transfers between the Y-MP
ccc and T3D will go through PE0. This is recommended in CRI's PVM
ccc Manual.
print *, 'Task ' , mytid, ' Parent ', myparent
msgtag = 1
if (mype .eq. 0) then
ccc Receive data from master process
call pvmfrecv(myparent, msgtag, ibuf)
call pvmfunpack(REAL8, vec, ISIZE, 1, info)
print *, 'Task ', mytid, ' received data ...'
ccc Now, relay this data to the other T3D slave processes
call pvmfinitsend(PVMDATADEFAULT, info)
call pvmfpack(REAL8, vec, ISIZE, 1, info)
call pvmfbcast(PVMALL, msgtag, info)
else
ccc Receive data from PE0
call pvmfrecv(0, msgtag, ibuf)
call pvmfunpack(REAL8, vec, ISIZE, 1, info)
print *, 'Task ', mytid, ' received data ...'
endif
ccc Wait for all slaves to reach this point
call pvmfbarrier(PVMALL, -1, info)
ccc PE0 sends ACK to Y-MP process
if (mype .eq. 0) then
msgtag = 2
call pvmfinitsend(PVMDATADEFAULT, info)
call pvmfpack(INTEGER8, 1, 1, 1, info)
call pvmfsend(myparent, msgtag, info)
endif
ccc Everybody exits PVM
call pvmfexit(info)
print *, 'Task ', mytid, ' exiting...'
end
Makefile:
all: master slave
slave: slave.f
(export TARGET=cray-t3d; \
/mpp/bin/cf77 -I/usr/include/mpp -o slave slave.f)
cp slave ${HOME}/pvm3/bin/CRAY/
master: master.f
(export TARGET=target; \
cf77 -I/usr/include/pvm3 -o master master.f -lpvm3)
From:
=======================================================================
Don Morton Email: morton@arsc.edu
Visiting Scientist (summer) Voice: (907) 474-5507
Arctic Region Supercomputing Center Fax : (907) 450-8601
University of Alaska
Fairbanks, AK 99775
http://grizzly.cameron.edu/morton.html
=======================================================================
The "TARGET" Environment Variable
If you do development work on both the Y-MP and the T3D, and switch back and forth between the two, you might have some handy aliases defined. For instance, users of the korn shell might have the following in their .kshrc file:alias t3d1='TARGET=cray-t3d;MPP_NPES=1;PVM_PE_LIST=0' alias t3d2='TARGET=cray-t3d;MPP_NPES=2;PVM_PE_LIST=0' alias t3d4='TARGET=cray-t3d;MPP_NPES=4;PVM_PE_LIST=0' alias t3d8='TARGET=cray-t3d;MPP_NPES=8;PVM_PE_LIST=0'The following alias for switching back to the Y-MP environment, however, would be a mistake (were it not commented out):
#alias ymp='TARGET=cray-ymp'The correct Y-MP alias is:
alias ymp='TARGET=target'The setting, "target," is the configured default, which should be set correctly for each site. The setting, "cray-ymp," sounds good, but according to the man page for the UNICOS command, "target," is for the "CRAY Y-MP model 832."
I discovered this when cf77 suddenly balked on a test program, and gave the following message:
ldr-159 cf77: CAUTION The program in file 'a.out' requires 40150829 words of memory to begin execution. The target computer system contains only 33554432 words of memory.The linker refused to produce an executable because it thought that denali had only 33 MWs of memory -- less than required for the given program. As we know, though, denali has, not 33 MWs, but 1000 MWs of memory.
The problem was that I had set TARGET to "cray-ymp" in order to switch it back from "cray-t3d." I should have set it to "target."
The following (abridged) listing shows the "cray-ymp", "target", and "cray-t3d" configurations, as known on denali.
denali$
denali$ TARGET=cray-ymp
denali$ target
Primary machine type is: CRAY-Y-MP
banks = 256
numcpus = 8
ibufsize = 32
memsize = 33554432
memspeed = 17
clocktim = 6000
numclstr = 9
bankbusy = 5
denali$ TARGET=target
denali$ target
Primary machine type is: CRAY-Y-MP
banks = 256
numcpus = 8
ibufsize = 32
memsize = 1073741824
memspeed = 27
clocktim = 6000
numclstr = 9
bankbusy = 20
denali$ TARGET=cray-t3d
denali$ target
Primary machine type is: CRAY-T3D
memsize = 8388608
denali$
Setting Environment Variables for a Single Sub-Process
While we're discussing environment variables... Here's something for everyone who's made the switch to the Korn shell.The Korn shell lets you change the setting of environment variables for the scope of single sub-processes, without altering their global values. You simply preface any command with one or more variable definitions. The environment for that command will be set accordingly.
Examples:
denali$ TARGET=cray-t3d target denali$ TARGET=cray-ymp target denali$ TARGET=target cf77 -I/usr/include/pvm3 -o master master.f -lpvm3 denali$ TARGET=cray-t3d /mpp/bin/cf77 -I/usr/include/mpp -o slave slave.f denali$ TMPDIR=/tmp/baring/960607 MPP_NPES=4 a.out denali$ TMPDIR=/tmp/baring/960607 MPP_NPES=8 a.outUsing this approach, we have an alternative structure for makefiles with multiple targets. The following example is based on Don Morton's makefile:
# Makefile
all: master slave
slave: slave.f
TARGET=cray-t3d /mpp/bin/cf77 -I/usr/include/mpp -o slave slave.f
cp slave ${HOME}/pvm3/bin/CRAY/
master: master.f
TARGET=cray-ymp cf77 -I/usr/include/pvm3 -o master master.f -lpvm3
European CRAY MPP Workshop
This announcement came through CUG's MPP Mailing List:> > ********************************************* > * PROGRAMME AND CALL FOR PARTICIPATION * > * FOR THE 2ND EUROPEAN CRAY MPP WORKSHOP * > * * > * http://www.epcc.ed.ac.uk/t3d/workshop * > ********************************************* > > > The programme for The 2nd European Cray MPP Workshop has now been > finalised. Most of the major Cray MPP sites in Europe are represented > with speakers from Finland, France, Germany, Italy, Spain, Switzerland, > and the UK. There will also be talks from Cray US and Pittsburgh > Supercomputer Center about early experiences with the Cray T3E. The > programme is available at > > http://www.epcc.ed.ac.uk/t3d/workshop/programme.html > > We currently have over 100 registrations, so as space is limited, please > register as soon as possible if you are interested in attending; the > deadline is June 21. A registration form is available at our WWW site. > > Although we can no longer accept offers of talks, there is still some > space left for the poster session. We are interested in posters > describing any aspects of Applications in Science and Engineering, > Systems Management, Programming Models or Performance Optimisation on > Cray MPP platforms. See the WWW pages for more details. > > > Please distribute this information to anyone who might be interested. > We look forward to meeting you in Edinburgh in July, > > Dr David Henty > T3D In-Depth Support Team > Edinburgh Parallel Computing Centre > workshop@epcc.ed.ac.uk >
A Call for Material
If you have discovered a good technique or information concerning the T3D and you think it might benefit others, please send it to the e-mail address above and it will be passed on through this newsletter.Current Editors:
E-mail Subscriptions:
Ed Kornkven ARSC HPC Specialist ph: 907-450-8669 Kate Hedstrom ARSC Oceanographic Specialist ph: 907-450-8678 Arctic Region Supercomputing Center University of Alaska Fairbanks PO Box 756020 Fairbanks AK 99775-6020
-
Subscribe to (or unsubscribe from) the e-mail edition of the
ARSC HPC Users' Newsletter.
-
Back issues of the ASCII e-mail edition of the ARSC T3D/T3E/HPC Users' Newsletter are available by request. Please contact the editors.