ARSC HPC Users' Newsletter 377, January 11, 2008



Midnight: New Programming Environment: Portland Group

[ By: Alec Bennett ]

The Portland Group compiler suite version 7.0.2 is now available on Midnight. While Pathscale is still the default compiler, if you are having difficulties compiling certain applications, you may find the PGI compilers to be a nice alternative.

In order to use the PGI compilers on Midnight, you will need to load the PGI Programming Environment. Assuming you have the default module loaded, you can switch modules by doing the following:

        module switch PrgEnv PrgEnv.pgi

This will make the PGI compilers and debuggers available within your environment.

If you plan to use PGI as your default compiler suite, please make sure to load to the PrgEnv.pgi module in your ~/.login or ~/.profile files.

Additionally, to complement the new compiler, we have installed a number of libraries for use with the PGI compiler suite. Current versions of these libraries are available in /usr/local/pgi as well as being included in the default compilation paths, as follows:

        acml-3.6.0, blacs-1.1.3, fftw-2.1.5, fftw-3.1.2, hdf-4.2r1,
        ncarg-4.4.1, netcdf-3.6.1, scalapack-1.8.0

This information is also available on the ARSC website on the Midnight how-to page at:


Please feel free to try out the new compiler environment, and let the ARSC Help Desk ( know of your results or if you have any trouble!


Fortran Code Timers

[ By: Lee Higbie ]

One of the common tasks for Fortran developers is code timing, frequently in preparation for code optimization. Well designed applications often include timing functions so they can collect information about the program's execution during normal runs. Here are some of the available Fortran functions for such timings.

There are quite a few generally-available timing functions. Fortran 90 specifies two clock subroutines:

   date_and_time(date, time, zone, values)
   system_clock(count, count_rate, maximum_count)
Fortran 95 specifies an additional subroutine:

The MPI library supplies the function:

and the PAPI library includes several functions:

   papif_flops(real_time, proc_time, flpops, mflops, check)

Most, if not all, of these functions should be available on HPC machines. I tested these routines to get a rough idea of their resolution. This first round of tests was done on Midnight with the PathScale and PGI compilers. I hope to report further results soon.


The mpi_wtime routine had the highest resolution of those that were easy to use and well documented. mpi_wtime() has an associated function mpi_wtick() that is supposed to report the resolution. With Midnight's MPI stack, it reported 10 microsec, but the actual resolution I saw was closer to 1 microsec.


The subroutine system_clock reported a rate of 1 MHz and delivered that resolution on PathScale. It uses integer counts, so extracting time is slightly more cumbersome than with a floating point clock such as mpi_wtime().


The subroutine cpu_time had a 1 millisecond resolution when called from PathScale and 1 microsecond from PGI.

The fact that system_clock and cpu_time are subroutines instead of functions makes them less convenient to use.


The PAPI subroutine papif_flops can return both the computational speed and the time. It appeared to deliver microsec resolution, but with an overhead of several microseconds so the times were consistently high.

The standard Fortran subroutines system_clock and date_and_time both delivered resolutions of a millisec, so they are suitable only for timing large code blocks.

papif_get_real_cyc & papif_get_real_usec

The PAPI subroutines papif_get_real_cyc(time) and papif_get_real_usec(time) were more difficult to use. Because PAPI uses Fortran 77 and the Fortran API for PAPI is not as well documented, I don't recommend them. However, if resolution greater than 1 microsecond is needed, papif_get_real_cyc(cycleCount) does return the actual number of clock cycles. Thus, on Midnight it returns times of about a half nanosecond.

In the future I hope to further test papif_get_real_cyc for small code segments. The initial tests showed that it discriminates down to 100 nanosecond times.


PGI Floating Point Traps

[ By: Don Bahls ] 

The last issue showed a generalized way to trap floating point exceptions on Linux systems. As it turns out, the newly installed PGI compilers have the -Ktrap option which also allows you to selectively enable floating point exceptions. All of the standard IEEE exceptions are supported:

inv Invalid Operation
denorm Denormalized Value
divz Division By Zero
ovf Overflow
unf Underflow
inexact Inexact value

There is also a special option "fp" which enables "inv,divz,ovf". If you happen to be using the PGI compiler suite, this is an easy way to enable floating point traps.

Here's an example compile statement:

   mg56 % pgf90 -Ktrap=fp exceptions2.f90 -o exceptions2
This is equivalent to:

   mg56 % pgf90 -Ktrap=inv,divz,ovf exceptions2.f90 -o exceptions2    

The -Ktrap option only applies to the main function or program, so you can't selectively turn floating point exceptions on and off for selected code segments. If that's something you want to do, the solution from last issue will work, but you will have to implement fedisableexcept. If you are trying to track down a bug quickly... this is a good solution.


Quick-Tip Q & A

A:[[ I downloaded and untarred a mondo source code, spanning many
  [[ sub-directories.  I've made changes here and there, to about
  [[ 25 files.  Now I need to figure out which files I changed, copy
  [[ them out, and back them up.
  [[ My problem is that at least 10 of the updated files have the
  [[ same name ("makefile"). Thus, the complete PATH to each file
  [[ is critical information.
  [[ How can I identify the changed files, extract them, and associate
  [[ them with their PATHs so I'll know later where they came from?

# Thanks to Martin Luthi:

Something like this will work, assuming you have GNU "cp":

  mkdir ../backup; find . -mtime 3 -exec cp -p --parents {} ../backup  \;

Some notes:
 * make a backup directory outside the directory tree (e.g. ../backup),
   or it will be backed up as well

 * change the mtime argument to whatever suits you
   -mtime 3 means: changed in the last 3*24 hours. There is also an

 * -p preserves modification times

 * --parents preserves the path under ./

# From Kurt Carlson

I use uals (available on ARSC systems). For any file newer than 60 days:
  /usr/local/adm/bin/uals -R --newer 60d --fields f

For any file newer than file 'ChangeLog':
  /usr/local/adm/bin/uals -RzZA --newer ChangeLog

In both cases you get a list of changed files with relative path.  The
first example gets just the filenames, the second is full uals output
since --fields was not specified.  Add '--type f' to get only files and
not directories or links.  You can also get a similar list with 'find .
-mtime' or 'find . -newer'.  How you back them up depends on your
maintenance philosophy: subversion, tarballs, master copy, ...

# From an editor:

This uses tar to append files one-by-one, to a tar file, as "find"
selects them.  Tar automatically includes the entire path of each

  find build_dir -type f -mtime -90 -exec tar uf source_updates.tar {} \;

Q: I just moved my code to midnight and ran my first job.  Now
   I've got a netcdf output file (  Is there an easy way
   to compare my new output file with the results I got from iceberg
   for the same input.  I want to make sure the optimizations I'm
   using haven't given me incorrect results.  Say I want to verify
   the differences between corresponding values in the two files
   isn't larger than 0.001, is there a straightforward way to do that?

   I did an ncdump of each file and started comparing them manually,
   but it's taking way too long!

[[ Answers, Questions, and Tips Graciously Accepted ]]

Current Editors:
Ed Kornkven ARSC HPC Specialist ph: 907-450-8669
Kate Hedstrom ARSC Oceanographic Specialist ph: 907-450-8678
Arctic Region Supercomputing Center
University of Alaska Fairbanks
PO Box 756020
Fairbanks AK 99775-6020
E-mail Subscriptions: Archives:
    Back issues of the ASCII e-mail edition of the ARSC T3D/T3E/HPC Users' Newsletter are available by request. Please contact the editors.
Back to Top