ARSC HPC Users' Newsletter 328, November 04, 2005
ARSC at SC05
ARSC will be at SC05 in Seattle from November 12-18. We will be in booth 112, so stop by to see what we are doing. If you have story ideas or Quick-Tip questions for the HPC Users' Newsletter, we would enjoy hearing from you.
Introduction to PMPI Part I
The MPI Standard includes provisions to allow developers to create profiled libraries on top of the system implementation of MPI. This is done by providing a name shifted entry point for each MPI functions. All functions beginning with the prefix "MPI_" are required by the MPI standard to be accessible through an alternate name beginning with the prefix "PMPI_".
E.g. MPI_SEND is also available as PMPI_SEND.
This allows developers to create a profiled implementation of an MPI subroutine which uses the underlying implementation of MPI without having access to the source code for the MPI library.
Profiled libraries could be useful for a number of reasons including:
- performance timing and statistics.
- to provide additional debugging information without changing the source code for your model.
- to trap invalid conditions that are not caught by the default implementation of MPI.
Here's a simple example which counts the number of MPI_SEND and MPI_RECV calls and displays the totals when MPI_FINALIZE is called.
klondike 1% cat prof_mpi.f90
module perf_data
! module to store performance data on a task by task basis
!
IMPLICIT NONE
INTEGER :: i_my_rank
INTEGER :: i_recv_calls
INTEGER :: i_send_calls
end module
subroutine MPI_INIT(ierror)
! alternate implementation of MPI_INIT which initializes data
! in the perf_data module.
!
use perf_data
IMPLICIT NONE
INTEGER, INTENT(OUT) :: ierror
INTEGER err
i_recv_calls=0
i_send_calls=0
i_my_rank=-1
CALL PMPI_INIT(ierror)
CALL SET_RANK
PRINT *,"Starting task", i_my_rank
end subroutine
subroutine SET_RANK
! sets i_my_rank in the perf_data module
!
use perf_data
use mpi
IMPLICIT NONE
INTEGER err
CALL PMPI_COMM_RANK(MPI_COMM_WORLD, i_my_rank, err)
end subroutine
subroutine MPI_SEND(buf, count, typ, dest, tag, comm, err)
! alternate implementation of MPI_SEND which tracks the number
! of MPI_SEND calls.
!
use perf_data
IMPLICIT NONE
INTEGER, INTENT(IN) :: buf, count, typ, dest, tag, comm
INTEGER, INTENT(OUT) :: err
INTEGER :: total_tasks, ierr
i_send_calls=i_send_calls+1
CALL PMPI_SEND(buf, count, typ, dest, tag, comm, err)
end subroutine
subroutine MPI_RECV(buf, count, typ, source, tag, comm, status, err)
use perf_data
IMPLICIT NONE
INTEGER, INTENT(OUT) :: buf, count, typ, source, tag, comm
INTEGER, INTENT(OUT) :: status(*), err
INTEGER :: total_tasks
INTEGER :: ierr
i_recv_calls=i_recv_calls+1
CALL PMPI_RECV(buf, count, typ, source, tag, comm, status, err)
end subroutine
subroutine MPI_FINALIZE(ierror)
! alternate implementation of MPI_FINALIZE which displays the
! number of MPI_SEND calls that were made by each task
!
use perf_data
IMPLICIT NONE
INTEGER, INTENT(OUT) :: ierror
PRINT *, "Task:", i_my_rank, " MPI_SEND Calls: ", i_send_calls
PRINT *, "Task:", i_my_rank, " MPI_RECV Calls: ", i_recv_calls
call PMPI_FINALIZE(ierror)
end subroutine
Notice that each subroutine calls the underlying system subroutine somewhere in the body of the routine. This ensures that the output of the profiled routines are the same as the default implementations.
The profiled subroutines can be compiled with an MPI compiler:
E.g.
Cray X1 (with the mpt module loaded):
klondike 2% ftn prof_mpi.f90 -c
IBM AIX
iceberg2 1% mpxlf90_r -qsuffix=f=f90 prof_mpi.f90 -c
This results in an object file which has profiled implementations of MPI_INIT, MPI_SEND, MPI_RECV and MPI_FINALIZE. At this point we could build a profiled MPI library by replacing the MPI routines in the default MPI library. This task can be done trivially on the Cray X1 using the 'ar' command, but requires a number of steps on AIX due to the "unusual" design of shared libraries on AIX. See reference B below for details on how to do this on AIX. For this example we will simply add prof_mpi.o to the compile statements and use the default MPI library to resolve the other symbols.
Here's a simple code which sends a message from even numbered tasks to odd numbered tasks.
iceberg2 3% cat mpi-send-recv.f90
PROGRAM MPI_SEND_RECV
use mpi
INTEGER ierr
INTEGER task,total_tasks
INTEGER stat(MPI_STATUS_SIZE)
REAL IN(10)
REAL BUF(10)
INTEGER II
CALL MPI_INIT(ierr)
CALL MPI_COMM_RANK(MPI_COMM_WORLD,task,ierr)
CALL MPI_COMM_SIZE(MPI_COMM_WORLD,total_tasks,ierr)
DO II=1, 10
IN(II)=task
END DO
CALL MPI_BARRIER(MPI_COMM_WORLD)
IF ( mod(task,2) == 0 ) then
CALL MPI_SEND(IN, 10, MPI_REAL, task+1, 99, MPI_COMM_WORLD, ierr)
ELSE
CALL MPI_RECV(BUF, 10, MPI_REAL, task-1, 99, MPI_COMM_WORLD, stat, ierr)
END IF
CALL MPI_FINALIZE(ierr)
END PROGRAM
This code does not produce any output by default. If we link against the profiled subroutines in prof_mpi.o, the executable will display a message from each task as it starts and the total number of MPI_SEND and MPI_RECV calls when MPI_FINALIZE is called.
Here are example compile statements for klondike and iceberg.
Cray X1:
klondike 3% ftn mpi-send-recv.f90 prof_mpi.o
IBM AIX:
iceberg2 4% mpxlf90_r -qsuffix=f=f90 mpi-send-recv.f90 prof_mpi.o
Running the resultant executable reports the numbers of sends and receives from each task.
iceberg2 72% poe ./a.out -hostfile hosts -procs 4 Starting task 0 Starting task 1 Task: 1 MPI_SEND Calls: 0 Task: 1 MPI_RECV Calls: 1 Starting task 2 Task: 2 MPI_SEND Calls: 1 Task: 2 MPI_RECV Calls: 0 Starting task 3 Task: 3 MPI_SEND Calls: 0 Task: 3 MPI_RECV Calls: 1 Task: 0 MPI_SEND Calls: 1 Task: 0 MPI_RECV Calls: 0
In the next part of this series will take a look how PMPI can be used to trap invalid conditions in MPI calls.
References:
- MPI 1.1 Standard: http://www.mpi-forum.org/docs/mpi-11-html/mpi-report.html
- IBM Parallel Environment for AIX 5L: MPI Programming Guide: Version 4 Release 2; pg 11-13; SA22-7945-02
Utilization Information Available
Utilization information is now available on both iceberg and klondike. The 'show_usage' command will display allocation, foreground utilization, background utilization and remaining allocation by project.
iceberg2 1% show_usage
Allocation for 'projecta' as of Nov 4 2005 at 04:00:01
Allocation: 50000.00
Usage: 704.69
Remaining Allocation: 49295.31
------------------
Background Usage: 0.00
Allocation for 'projectb' as of Nov 4 2005 at 04:00:01
Allocation: 10000.00
Usage: 2890.01
Remaining Allocation: 7109.99
------------------
Background Usage: 0.00
For a complete breakdown by project, please contact the
ARSC help desk:
email: consult@arsc.edu
phone: 907-450-8602
Quick-Tip Q & A
A:[[ I love vim's syntax highlighting, but, to save my eyes, I always work
[[ with light text on a dark background. Unfortunately, vim uses dark
[[ blue for C comments and they practically disappear. (So, of course, I
[[ end up burning my eyes out squinting and getting too close!)
[[
[[ Is there some way to change vim's colors for syntax highlighting?
#
# Thanks to Jed Brown:
#
The Short answer--
:h hi
Longer--
The "highlight" command is what you are looking for. For this problem,
I have
:highlight Comment ctermfg=cyan cterm=NONE
in my vimrc which makes comments appear as nonbold cyan when running
vim in the console. (The GUI has a special blue for comments which I
find very readable.)
#
# From Bob Clark
#
In vim, ':help highlight' will get you all the highlighting
help you ever wanted.
The ':hi' command will display your active settings.
I use white-background xterms and have the following
highlight settings in my .vimrc file:
" XTerm colors are:
" 0:black 1:red 2:green 3:yellow 4:blue 5:magenta 6:cyan 7:white
" (yellow and green do not work well on light backgrounds)
highlight MoreMsg ctermfg=5
highlight LineNr ctermfg=5
highlight Question ctermfg=5
highlight Statement ctermfg=5
highlight Type ctermfg=5
highlight Identifier ctermfg=4
highlight DiffAdd ctermfg=7
(ctermfg stands for color terminal foreground)
So, assuming you are using an xterm-like terminal, you
could turn your comments green with ':hi Comment ctermfg=2'
If you want more color options, you can try gvim.
#
# For advanced users, here's one from Jesse Niles
#
To use the extended color scheme set in vim:
1) Create the directory ~/.vim if it doesn't exist.
2) Download the color sampler pack located at:
http://www.vim.org/scripts/download_script.php?src_id=4010
and unpack
it in that directory. It will create the two necessary subdirectories
plugin and colors, if they don't already exist.
3) Open vim in graphical mode ('gvim' or 'vim -g'), if available, and
browse the color schemes.
4) When you find the one you like, add the line:
:colorscheme newcolors
to your ~/.vimrc file ("newcolors" is the name of the color scheme you
like). Create one if it doesn't already exist.
The next time you fire up vim or gvim, you'll get the new color
scheme. Note that the colors of the same scheme may differ between vim
and gvim, so you'll want to check both. If you just want the comments
to be a different color in the same scheme:
1) Copy the color scheme file from the system vim colors file to your
~/.vim/colors directory and open it in a text editor (you might as well
use vim because it has syntax highlighting for vim files).
2) Find the line that looks like:
highlight Comment ctermfg=8 guifg=#ff00ff
This line specifies the color that vim will use first, followed by what
gvim will use. If you want to change the gvim color to white for
example, just change the hex value. If you are using terminal-style
vim, you can just set the ctermfg value equal to the name of one of the
common colors available on terminals (e.g. white, black, blue, etc.).
To make it White, you can simply change the line to:
highlight Comment ctermfg=white guifg=#ffffff
--
As a bonus, if you'd like certain symbols to stand out, e.g. OpenGL
functions or MPI functions, you can add the functions or keywords to
the language syntax file. I use OpenGL a lot, and I sometimes make
errors in the capitalization of the symbol names. In vim you can
create a specialized syntax file that can color arbitrary symbols.
Thus, if you make a typo you won't have to wait until the compiler
whines about it. If you are as nerdy as I am, you can:
1) Copy the cpp.vim (or whatever language syntax file you are using)
from the system $VIMRUNTIME/syntax directory to your ~/.vim/syntax
directory and open it in a text editor.
2) Add the following lines to the file where the similar declarations
are, adding the functions/types/constants you want to add, separated by
spaces:
syn keyword specialFunctions glBegin glAlphaFunc glAreTexturesResident ...
syn keyword specialTypes GLbitfield GLboolean GLbyte ...
syn keyword specialConsts GL_2_BYTES GL_2D GL_2_BYTES ...
3) Add the following lines to the section with all of the "HiLink"
statements:
HiLink specialFunctions Special
HiLink specialTypes Type
HiLink specialConsts Constant
It may then look something like:
HiLink cppAccess cppStatement
HiLink cppCast cppStatement
HiLink cppExceptions Exception
HiLink cppOperator Operator
HiLink cppStatement Statement
HiLink cppType Type
HiLink cppStorageClass StorageClass
HiLink cppStructure Structure
HiLink cppNumber Number
HiLink cppBoolean Boolean
HiLink specialFunctions Special
HiLink specialTypes Type
HiLink specialConsts Constant
The above changes will get vim to highlight GLbitfield the same way it
would int, float or char, and will highlight GL_2_BYTES as it would
INT_MAX. Functions in the list above should show up in an entirely
different color so you can easily see errors. There are much more
advanced and generic ways to color text in vim, but this is as fancy as
I ever needed to get. Lastly, if you forget what the parameters in one
of the functions is, you can put the cursor over the function and hit
'K' (shift-k) to view the man page.
#
# Editor's solution:
#
Taken from vim's internal help ("*mysyntaxfile-add*"),
create this directory tree and file: ~/.vim/after/syntax/c.vim .
To retain all default C syntax colors, but turn comments
green, the file would contain this line of text:
highlight cComment ctermfg=Green guifg=Green
Q: [[Thanks to Ed Kornkven of ARSC for this week's Quick Tip question]]
I use a calculator for evaluating arithmetic expressions. It accepts
an arithmetic expression as stdin and outputs the answer on stdout.
E.g.,
cl 1024*1024*1024*15
= 16106127360
[Actually, all "cl" does is 'echo "print $*"
python' -- but I digress].
My only complaint with this "calculator" is that I have a hard time
reading long-digit numbers.
What I would like is a filter that accepts such a number on stdin, and
writes to stdout the number formatted with commas separating the
thousands places. E.g.,
cl 1024*1024*1024*15
comma_adder
= 16,106,127,360
[[ Answers, Questions, and Tips Graciously Accepted ]]
Current Editors:
E-mail Subscriptions:
Ed Kornkven ARSC HPC Specialist ph: 907-450-8669 Kate Hedstrom ARSC Oceanographic Specialist ph: 907-450-8678 Arctic Region Supercomputing Center University of Alaska Fairbanks PO Box 756020 Fairbanks AK 99775-6020
-
Subscribe to (or unsubscribe from) the e-mail edition of the
ARSC HPC Users' Newsletter.
-
Back issues of the ASCII e-mail edition of the ARSC T3D/T3E/HPC Users' Newsletter are available by request. Please contact the editors.