ARSC HPC Users' Newsletter 417, December 10, 2010

North Pole Letters

We have recovered from a freezing rain event which left a sheet of ice on all the roads. Mail service is back to normal, just in time for our HPC Newsletter holiday tradition. If you know someone who would be thrilled by a letter postmarked "North Pole", your editors can help make that happen.

Put that stamped, addressed letter in another envelope and mail it to us. We will mail the enclosed letter from North Pole, Alaska, which uses a special North Pole Christmas postmark during the Christmas season. We plan to send these letters around December 17, so mail them to us by then so they have plenty of time to get here and then over the next sleigh out of town.

Send to:


    Ed Kornkven and Kate Hedstrom
    Arctic Region Supercomputing Center
    University of Alaska Fairbanks
    P.O. Box 756020
    Fairbanks, AK 99775-6020 

Simple Representation of Profiling Information with gprof2dot and Graphviz

[ By Anton Kulchitsky ]

Have you ever tried to figure out the details of where your program spends most of its time using only text output like gprof/pgprof or valgrind/callgrind? If the program is complicated enough that some functions are called from different places, it can be challenging to see the whole picture. From one side, there are some useful visualization tools and GUIs available. Some editors can highlight the information and make it more fancy. However, there is a very fast way to visualize your profiling information without installing a new GUI.

In this small introductory letter, a not-so-little Python script is described. Gprof2Dot is a script that converts text output from your profiler to a call graph description. The script is free and it is hosted on Google Code (see links below). A call graph description is given in the dot language that can be processed by the dot utility of the Graphviz package to produce a nice image of your call graph. Thank UNIX, all these procedures can be written as a short one-liner in your favorite terminal. Before we talk about easy and quick installation of these tools, let us look at a few examples.

All examples below were performed on a Linux workstation at ARSC. These results depend on the version of gprof installed and will not necessarily be the same on your machine. The GCC compiler is required to compile the examples.

Let us consider a very simple example to illustrate the point. For this we need to have a program to play with. Let us consider an integration of f(x)*f(y)*f(z) in a [0,1]x[0,1]x[0,1] cube. To make things more interesting, some complication is added to this trivial problem:


/* file: dot-example1.c
         integral of f(x)*f(y)*f(z) from 0 to 1 */
#include <stdio.h>
#include <stdlib.h>
#include <math.h>

double f( double x ) { return sin(x) + cos(x); } double fx( double x ) { return f(x); } double fy( double y ) { return f(y); } double fz( double z ) { return sin(z) + cos(z); }
int main() { const size_t N = 300; /* number of segments */ const double dx = 1.0 / N; /* space step */ double e = sin(1.0) - cos(1.0) + 1.0; /* exact value */ e = e*e*e; /* triple integral */
double S = 0; /* integral */ for ( size_t i = 0; i < N; ++i ) { double x = 0.5*dx + i * dx; for ( size_t j = 0; j < N; ++j ) { double y = 0.5*dx + j * dx; for ( size_t k = 0; k < N; ++k ) { double z = 0.5*dx + k * dx; S += dx*dx*dx * fx(x) * fy(y) * fz(z); } } }
printf( "Exact value: %lf\n", e ); printf( "Calculated value: %lf\n", S ); printf( "Difference: %lf\n", S-e );
return 0; }

Let us save this into a file named dot-example1.c . To compile this program we use the following command:


    gcc -std=gnu99 -g -pg -lm dot-example1.c

The executable file will be a.out . Option -g will include some debug information that will be helpful for the profiler. Option -pg will insert profiling information. Because we use variable declarations within loop constructs, we need to ask for the C99 standard version with the option -std=gnu99. The option -lm will link against the math library.


    Run ./a.out

Now we have a gmon.out file. Let us run the following command in the directory where we ran ./a.out :


    gprof ./a.out 
 gprof2dot.py 
 dot -Tpng -o example1.png

That is it. We should have example1.png in our directory. Take a look at this file. Does it seem slightly clearer than the text output produced by gprof?

You can play with any other applications. Please take a look at a more complicated picture that I had profiling one of my own projects:

This little article provides just an introduction. The reader will be able to find much more information. Good starts are at the following links:

Trying out Github

[ By Kate Hedstrom ]

As you probably know by now, I’ve been playing with git for a while. I was sharing files with some external colleagues through a machine at Rutgers, but the sys admin on that computer somehow messed up git and has been travelling so much that he’ll never get it fixed. Yesterday it became clear that we were going to have to find another way to share files.

There is a company at github.com which provides git hosting services. If your projects are open source, it is completely free. If your projects are proprietary, there is a fee structure, depending on how many projects, how many colleagues, and how many gigabytes of storage you need. Because we’re developing things privately, I went for a "small" account. I’m sure I’ll run out of colleagues before I hit the other limits.

At github, they have plenty of help files for setting up your account, including the ssh handshaking with RSA keys (new to me). It all worked fine on midnight and pacman, but it failed on some of the other ARSC systems. They suggest trying "ssh -v git@github.com" to see all the debug messages. Everything seemed to be going perfectly until it failed with:

debug1: Authentications that can continue: publickey debug1: No more authentication methods to try. Permission denied (publickey).

This didn’t seem very helpful - why wasn’t it continuing with the publickey? I got the answer from one of the system people here, who told me that there’s a system-level config file for ssh. On the DoD systems at ARSC, they are set to have these defaults:

PubkeyAuthentication no RSAAuthentication no

Changing these from "no" to "yes" in my $HOME/.ssh/config file allowed me to continue. I now have a few projects over there and a pair of collaborators. Github seems like a very well put together site and they are even going to offer an all-day online Git course.

Dynamic Shared Objects on the Cray XE6

[ By Ed Kornkven ]

Users of Pingo have become accustomed to the absence of shared libraries on the Cray XT. In order to provide a lean and low-jitter operating system for the XT5s, Cray elected not to provide dynamic objects on the XT5 compute nodes (though the XT5 does have a more fully-featured Linux than was present on the earlier XT machines). It is possible to work around that limitation by copying one’s own shared objects to the compute nodes before job launch, but that is obviously a cumbersome approach.

On Chugach, ARSC’s new Cray XE6, users will once again be able to link and load dynamic shared objects. Cray has provided a file system service they call Data Virtualization Services (DVS). DVS will project a file system mounted on the service nodes to the compute nodes. Special-purpose DVS nodes will handle that traffic and the number of DVS nodes is scaled to the number of compute nodes.

DVS enables the XE6 compute nodes to access a wide possibility of file systems. It is also the mechanism that Cray is using to enable compute nodes to "see" dynamic objects by providing an alternate root file system on the compute node. The process of using dynamic objects is now straightforward. Simply link with the -dynamic flag or alternatively, set the XTPE_LINK_TYPE environment variable to dynamic before linking.

Quick-Tip Q & A


A:[[ On Solaris there’s a cool command called "pargs" that can list the
  [[ environment variables set for a running process.
  [[ 
  [[ E.g.
  [[ 
  [[     host 13% pargs -e 28119
  [[     28119:  -bash
  [[     envp[0]: USER=myusername
  [[     ...
  [[     envp[12]: DISPLAY=localhost:15.0
  [[ 
  [[ There’s no pargs command on Linux.   Is there a way to get the environment
  [[ variables for a process I own on a Linux system?
  [[


Kudos to readers Samy Al Bahra, Dan Stahlke and Richard Griswold for their tips.
# # From Samy Al Bahra: #
The /proc/<PID>/environ file (in the example given, "cat /proc/28119/environ" would suffice).
# # Dan Stahlke was on the same page: #
In response to this newsletter’s question, here is how to show the environment variables in Linux:
cat /proc/4573/environ tr ’\0’ ’\n’
Of course, substitute your process ID for the number. Linux also has a nice command ’pgrep’ to get the process ID, so you can just do this:
cat /proc/$(pgrep -n firefox)/environ tr ’\0’ ’\n’
By the way, there are lots of other goodies in /proc.
# # From Richard Griswold: #
The ps command has an option (e) that will show the environment for a process. For example:
> ps e PID TTY STAT TIME COMMAND 3594 pts/1 SNs 0:00 zsh term=linux SHELL=/bin/zsh TERM=xterm GTK_MODULES= 8571 pts/1 RN+ 0:00 ps e term=xterm SHELL=/bin/zsh TERM=xterm GTK_MODULES
You can mix in other options, such as "w" for wide output, "ww" for full output, or "-e" for all users (you’ll need to run as root to see the environment for other users). Check the ps man page for more info.

Q: Building Makefiles always seems like a work of art. Half the time I see a Makefile define everything inside of it for the various includes and libraries that an application needs.
For example:
INCLUDE = -I. -I/usr/include/ -I/usr/include/X11/ -I/usr/local/include/GL INCH5 = -I/import/home/u1/uaf/webb/hdf5/include INCOSG = -I/usr/local/pkg/osg/osg-2.8.0/include
LDFLAGS = -L. -L/usr/lib64 -L/usr/X11R6/lib64 -L/usr/local/lib64 LDH5 = -L/import/home/u1/uaf/webb/hdf5/lib LDOSG = -L/usr/local/pkg/osg/osg-2.8.0/lib64 -losg -losgViewer -losgSim
Other times, a Makefile will rely on outside environment variables to provide paths to these resources. When building your Makefiles, which style do you use and when? Is there a powerful compelling reason to use one over the other?


[[ Answers, Questions, and Tips Graciously Accepted ]]


Current Editors:
Ed Kornkven ARSC HPC Specialist ph: 907-450-8669
Kate Hedstrom ARSC Oceanographic Specialist ph: 907-450-8678
Arctic Region Supercomputing Center
University of Alaska Fairbanks
PO Box 756020
Fairbanks AK 99775-6020
E-mail Subscriptions: Archives:
    Back issues of the ASCII e-mail edition of the ARSC T3D/T3E/HPC Users' Newsletter are available by request. Please contact the editors.
Back to Top