ARSC HPC Users' Newsletter 328, November 04, 2005
ARSC at SC05
ARSC will be at SC05 in Seattle from November 12-18. We will be in booth 112, so stop by to see what we are doing. If you have story ideas or Quick-Tip questions for the HPC Users' Newsletter, we would enjoy hearing from you.
Introduction to PMPI Part I
The MPI Standard includes provisions to allow developers to create profiled libraries on top of the system implementation of MPI. This is done by providing a name shifted entry point for each MPI functions. All functions beginning with the prefix "MPI_" are required by the MPI standard to be accessible through an alternate name beginning with the prefix "PMPI_".
E.g. MPI_SEND is also available as PMPI_SEND.
This allows developers to create a profiled implementation of an MPI subroutine which uses the underlying implementation of MPI without having access to the source code for the MPI library.
Profiled libraries could be useful for a number of reasons including:
- performance timing and statistics.
- to provide additional debugging information without changing the source code for your model.
- to trap invalid conditions that are not caught by the default implementation of MPI.
Here's a simple example which counts the number of MPI_SEND and MPI_RECV calls and displays the totals when MPI_FINALIZE is called.
klondike 1% cat prof_mpi.f90 module perf_data ! module to store performance data on a task by task basis ! IMPLICIT NONE INTEGER :: i_my_rank INTEGER :: i_recv_calls INTEGER :: i_send_calls end module subroutine MPI_INIT(ierror) ! alternate implementation of MPI_INIT which initializes data ! in the perf_data module. ! use perf_data IMPLICIT NONE INTEGER, INTENT(OUT) :: ierror INTEGER err i_recv_calls=0 i_send_calls=0 i_my_rank=-1 CALL PMPI_INIT(ierror) CALL SET_RANK PRINT *,"Starting task", i_my_rank end subroutine subroutine SET_RANK ! sets i_my_rank in the perf_data module ! use perf_data use mpi IMPLICIT NONE INTEGER err CALL PMPI_COMM_RANK(MPI_COMM_WORLD, i_my_rank, err) end subroutine subroutine MPI_SEND(buf, count, typ, dest, tag, comm, err) ! alternate implementation of MPI_SEND which tracks the number ! of MPI_SEND calls. ! use perf_data IMPLICIT NONE INTEGER, INTENT(IN) :: buf, count, typ, dest, tag, comm INTEGER, INTENT(OUT) :: err INTEGER :: total_tasks, ierr i_send_calls=i_send_calls+1 CALL PMPI_SEND(buf, count, typ, dest, tag, comm, err) end subroutine subroutine MPI_RECV(buf, count, typ, source, tag, comm, status, err) use perf_data IMPLICIT NONE INTEGER, INTENT(OUT) :: buf, count, typ, source, tag, comm INTEGER, INTENT(OUT) :: status(*), err INTEGER :: total_tasks INTEGER :: ierr i_recv_calls=i_recv_calls+1 CALL PMPI_RECV(buf, count, typ, source, tag, comm, status, err) end subroutine subroutine MPI_FINALIZE(ierror) ! alternate implementation of MPI_FINALIZE which displays the ! number of MPI_SEND calls that were made by each task ! use perf_data IMPLICIT NONE INTEGER, INTENT(OUT) :: ierror PRINT *, "Task:", i_my_rank, " MPI_SEND Calls: ", i_send_calls PRINT *, "Task:", i_my_rank, " MPI_RECV Calls: ", i_recv_calls call PMPI_FINALIZE(ierror) end subroutine
Notice that each subroutine calls the underlying system subroutine somewhere in the body of the routine. This ensures that the output of the profiled routines are the same as the default implementations.
The profiled subroutines can be compiled with an MPI compiler:
Cray X1 (with the mpt module loaded):
klondike 2% ftn prof_mpi.f90 -c
iceberg2 1% mpxlf90_r -qsuffix=f=f90 prof_mpi.f90 -c
This results in an object file which has profiled implementations of MPI_INIT, MPI_SEND, MPI_RECV and MPI_FINALIZE. At this point we could build a profiled MPI library by replacing the MPI routines in the default MPI library. This task can be done trivially on the Cray X1 using the 'ar' command, but requires a number of steps on AIX due to the "unusual" design of shared libraries on AIX. See reference B below for details on how to do this on AIX. For this example we will simply add prof_mpi.o to the compile statements and use the default MPI library to resolve the other symbols.
Here's a simple code which sends a message from even numbered tasks to odd numbered tasks.
iceberg2 3% cat mpi-send-recv.f90 PROGRAM MPI_SEND_RECV use mpi INTEGER ierr INTEGER task,total_tasks INTEGER stat(MPI_STATUS_SIZE) REAL IN(10) REAL BUF(10) INTEGER II CALL MPI_INIT(ierr) CALL MPI_COMM_RANK(MPI_COMM_WORLD,task,ierr) CALL MPI_COMM_SIZE(MPI_COMM_WORLD,total_tasks,ierr) DO II=1, 10 IN(II)=task END DO CALL MPI_BARRIER(MPI_COMM_WORLD) IF ( mod(task,2) == 0 ) then CALL MPI_SEND(IN, 10, MPI_REAL, task+1, 99, MPI_COMM_WORLD, ierr) ELSE CALL MPI_RECV(BUF, 10, MPI_REAL, task-1, 99, MPI_COMM_WORLD, stat, ierr) END IF CALL MPI_FINALIZE(ierr) END PROGRAM
This code does not produce any output by default. If we link against the profiled subroutines in prof_mpi.o, the executable will display a message from each task as it starts and the total number of MPI_SEND and MPI_RECV calls when MPI_FINALIZE is called.
Here are example compile statements for klondike and iceberg.
klondike 3% ftn mpi-send-recv.f90 prof_mpi.o
iceberg2 4% mpxlf90_r -qsuffix=f=f90 mpi-send-recv.f90 prof_mpi.o
Running the resultant executable reports the numbers of sends and receives from each task.
iceberg2 72% poe ./a.out -hostfile hosts -procs 4 Starting task 0 Starting task 1 Task: 1 MPI_SEND Calls: 0 Task: 1 MPI_RECV Calls: 1 Starting task 2 Task: 2 MPI_SEND Calls: 1 Task: 2 MPI_RECV Calls: 0 Starting task 3 Task: 3 MPI_SEND Calls: 0 Task: 3 MPI_RECV Calls: 1 Task: 0 MPI_SEND Calls: 1 Task: 0 MPI_RECV Calls: 0
In the next part of this series will take a look how PMPI can be used to trap invalid conditions in MPI calls.
- MPI 1.1 Standard: http://www.mpi-forum.org/docs/mpi-11-html/mpi-report.html
- IBM Parallel Environment for AIX 5L: MPI Programming Guide: Version 4 Release 2; pg 11-13; SA22-7945-02
Utilization Information Available
Utilization information is now available on both iceberg and klondike. The 'show_usage' command will display allocation, foreground utilization, background utilization and remaining allocation by project.
iceberg2 1% show_usage Allocation for 'projecta' as of Nov 4 2005 at 04:00:01 Allocation: 50000.00 Usage: 704.69 Remaining Allocation: 49295.31 ------------------ Background Usage: 0.00 Allocation for 'projectb' as of Nov 4 2005 at 04:00:01 Allocation: 10000.00 Usage: 2890.01 Remaining Allocation: 7109.99 ------------------ Background Usage: 0.00 For a complete breakdown by project, please contact the ARSC help desk: email: email@example.com phone: 907-450-8602
Quick-Tip Q & A
A:[[ I love vim's syntax highlighting, but, to save my eyes, I always work [[ with light text on a dark background. Unfortunately, vim uses dark [[ blue for C comments and they practically disappear. (So, of course, I [[ end up burning my eyes out squinting and getting too close!) [[ [[ Is there some way to change vim's colors for syntax highlighting? # # Thanks to Jed Brown: # The Short answer-- :h hi Longer-- The "highlight" command is what you are looking for. For this problem, I have :highlight Comment ctermfg=cyan cterm=NONE in my vimrc which makes comments appear as nonbold cyan when running vim in the console. (The GUI has a special blue for comments which I find very readable.) # # From Bob Clark # In vim, ':help highlight' will get you all the highlighting help you ever wanted. The ':hi' command will display your active settings. I use white-background xterms and have the following highlight settings in my .vimrc file: " XTerm colors are: " 0:black 1:red 2:green 3:yellow 4:blue 5:magenta 6:cyan 7:white " (yellow and green do not work well on light backgrounds) highlight MoreMsg ctermfg=5 highlight LineNr ctermfg=5 highlight Question ctermfg=5 highlight Statement ctermfg=5 highlight Type ctermfg=5 highlight Identifier ctermfg=4 highlight DiffAdd ctermfg=7 (ctermfg stands for color terminal foreground) So, assuming you are using an xterm-like terminal, you could turn your comments green with ':hi Comment ctermfg=2' If you want more color options, you can try gvim. # # For advanced users, here's one from Jesse Niles # To use the extended color scheme set in vim: 1) Create the directory ~/.vim if it doesn't exist. 2) Download the color sampler pack located at: http://www.vim.org/scripts/download_script.php?src_id=4010 and unpack it in that directory. It will create the two necessary subdirectories plugin and colors, if they don't already exist. 3) Open vim in graphical mode ('gvim' or 'vim -g'), if available, and browse the color schemes. 4) When you find the one you like, add the line: :colorscheme newcolors to your ~/.vimrc file ("newcolors" is the name of the color scheme you like). Create one if it doesn't already exist. The next time you fire up vim or gvim, you'll get the new color scheme. Note that the colors of the same scheme may differ between vim and gvim, so you'll want to check both. If you just want the comments to be a different color in the same scheme: 1) Copy the color scheme file from the system vim colors file to your ~/.vim/colors directory and open it in a text editor (you might as well use vim because it has syntax highlighting for vim files). 2) Find the line that looks like: highlight Comment ctermfg=8 guifg=#ff00ff This line specifies the color that vim will use first, followed by what gvim will use. If you want to change the gvim color to white for example, just change the hex value. If you are using terminal-style vim, you can just set the ctermfg value equal to the name of one of the common colors available on terminals (e.g. white, black, blue, etc.). To make it White, you can simply change the line to: highlight Comment ctermfg=white guifg=#ffffff -- As a bonus, if you'd like certain symbols to stand out, e.g. OpenGL functions or MPI functions, you can add the functions or keywords to the language syntax file. I use OpenGL a lot, and I sometimes make errors in the capitalization of the symbol names. In vim you can create a specialized syntax file that can color arbitrary symbols. Thus, if you make a typo you won't have to wait until the compiler whines about it. If you are as nerdy as I am, you can: 1) Copy the cpp.vim (or whatever language syntax file you are using) from the system $VIMRUNTIME/syntax directory to your ~/.vim/syntax directory and open it in a text editor. 2) Add the following lines to the file where the similar declarations are, adding the functions/types/constants you want to add, separated by spaces: syn keyword specialFunctions glBegin glAlphaFunc glAreTexturesResident ... syn keyword specialTypes GLbitfield GLboolean GLbyte ... syn keyword specialConsts GL_2_BYTES GL_2D GL_2_BYTES ... 3) Add the following lines to the section with all of the "HiLink" statements: HiLink specialFunctions Special HiLink specialTypes Type HiLink specialConsts Constant It may then look something like: HiLink cppAccess cppStatement HiLink cppCast cppStatement HiLink cppExceptions Exception HiLink cppOperator Operator HiLink cppStatement Statement HiLink cppType Type HiLink cppStorageClass StorageClass HiLink cppStructure Structure HiLink cppNumber Number HiLink cppBoolean Boolean HiLink specialFunctions Special HiLink specialTypes Type HiLink specialConsts Constant The above changes will get vim to highlight GLbitfield the same way it would int, float or char, and will highlight GL_2_BYTES as it would INT_MAX. Functions in the list above should show up in an entirely different color so you can easily see errors. There are much more advanced and generic ways to color text in vim, but this is as fancy as I ever needed to get. Lastly, if you forget what the parameters in one of the functions is, you can put the cursor over the function and hit 'K' (shift-k) to view the man page. # # Editor's solution: # Taken from vim's internal help ("*mysyntaxfile-add*"), create this directory tree and file: ~/.vim/after/syntax/c.vim . To retain all default C syntax colors, but turn comments green, the file would contain this line of text: highlight cComment ctermfg=Green guifg=Green Q: [[Thanks to Ed Kornkven of ARSC for this week's Quick Tip question]] I use a calculator for evaluating arithmetic expressions. It accepts an arithmetic expression as stdin and outputs the answer on stdout. E.g., cl 1024*1024*1024*15 = 16106127360 [Actually, all "cl" does is 'echo "print $*" python' -- but I digress]. My only complaint with this "calculator" is that I have a hard time reading long-digit numbers. What I would like is a filter that accepts such a number on stdin, and writes to stdout the number formatted with commas separating the thousands places. E.g., cl 1024*1024*1024*15 comma_adder = 16,106,127,360
[[ Answers, Questions, and Tips Graciously Accepted ]]
Ed Kornkven ARSC HPC Specialist ph: 907-450-8669 Kate Hedstrom ARSC Oceanographic Specialist ph: 907-450-8678 Arctic Region Supercomputing Center University of Alaska Fairbanks PO Box 756020 Fairbanks AK 99775-6020
Subscribe to (or unsubscribe from) the e-mail edition of the
ARSC HPC Users' Newsletter.
Back issues of the ASCII e-mail edition of the ARSC T3D/T3E/HPC Users' Newsletter are available by request. Please contact the editors.