ARSC HPC Users' Newsletter 379, February 8, 2008

Call for Abstracts for the "Little Alaska Weather Symposium"

Abstracts for the Little Alaska Weather Symposium (LAWS) are due March 4. The May 12-13 LAWS at UAF provides a forum to exchange Alaska-specific operational forecasting and weather research data. For additional information:

http://weather.arsc.edu/Events/LAWS08/

CGI Library Available, C/C++

A library of CGI functionality is available for download. This library includes C and C++ functions, test programs and utilities for the common gateway interface (CGI) capabilities of Web servers such as Apache's httpd.

Development began in 1993, and the CGI library remains in use. With emphasis on security and stability, the library has been extensively tested in production environments. It is released under the BSD license.

Download location:

http://mirrors.arsc.edu/arsc/CGIlibs.tgz

or

ftp://mirrors.arsc.edu/mirrors/arsc/CGIlibs.tgz

For more information, contact:

Shawn Houston, houston@alaska.edu

NetCDF Conversion Example

[ By: Lee Higbie ]

I recently returned to a program I'd worked on several months ago. It uses data from other sources and seemed to be having data access problems. I decided to convert all those binary data files into a NetCDF file because:

  1. ncview would quickly show gross errors
  2. It would decouple my program from other programs whose output it uses
  3. The next time I wondered about the data, ncdump -h would describe the data

Here are all the Fortran 90 statements required for the conversion program. I've stripped out much of the program but left what you'll need for creating a large NetCDF dataset. (BTW: UCAR, which developed NetCDF spells the name both with and without a capital N.)


  use netcdf        ! use modules where you can
  implicit none     ! use it or die
....

   ! These data were defined on a flat, cartesian coordinate grid.
   ! I created two arrays defining the grid: lons(:, :), lats(:, :)
   ! The variable names ending in "ID" or "IDs" are defined
   ! by NetCDF.  They define the (NetCDF dataset) object.

  nf90_create  (ncFilename, nf90_clobber, ncID)       ! open file

   ! next define grid longitude dimension, then same for lats
! lonDim = number of longitudes,
! dimIDs is an integer array for netCDF's identifiers
  nf90_def_dim (ncID, 'longitude', lonDim, dimIDs(1))
  nf90_def_dim (ncID, 'latitude', dimIDs(2))

   ! next define the grid-defining variables
! I defined dimIDs with dimensions(3), here I only want 2 values
! lonvarID and latvarID are NetCDF's way of describing the vars
  nf90_def_var (ncID, 'longitudes', nf90_real, dimIDs(1: 2),         &
                          lonvarID)

   ! next 3 statements define metadata for grid longitudes
  nf90_put_att (ncID, lonvarID, 'units', 'index')
  nf90_put_att (ncID, lonvarID, 'stagger', 'none')
  nf90_put_att (ncID, lonvarID, 'description', 'computed')

   ! next 3 statements define metadata for grid latitudes
  nf90_put_att (ncID, latvarID, 'units', 'index')
  nf90_put_att (ncID, latvarID, 'stagger', 'none')
  nf90_put_att (ncID, latvarID, 'description', 'computed')
  nf90_enddef (ncID)            ! ends header (meta data defn)

   ! store the longitudes and latitudes of each grid cell
  nf90_put_var (ncID, lonvarID, lons)
  nf90_put_var (ncID, latvarID, lats)

This code shows all the statements for writing the grid-defining variables to a NetCDF dataset. The dataset is still open. An important note, each of the function calls above was actually the argument to a subroutine that printed diagnostic information if an error was detected. The next steps were in a loop over all the binary files to add to this dataset. (do; open; read ....)


....
    nf90_redef (ncID)     ! re-open the metadata defining segment
    nf90_def_dim (ncID, dimNames(i), dims(i), dimIDs(i))
    nf90_def_var (ncID, dataName, dataIndex, dimIDs(1: rank),     &
                                varID)
    nf90_put_att (ncID, varID, 'units', units)
    nf90_put_att (ncID, varID, 'stagger', stagger)
.....            ! put in additional descriptive info you need
    nf90_enddef (ncID)        ! close meta-defining for this file
! for real data:
    nf90_put_var (ncID, varID, rData)
! or, for integer data:
    nf90_put_var (ncID, varID, iData)
...       
! then, outside the loop, when all the data are written:
  nf90_close (ncID)

These are all the NetCDF calls necessary to create the file. That program uses a series of namelists (samples there) to define the input data. If you want to use it as written, you will also need DebugClass.F95 (available there).

Checking the data (only a small part of the header is shown here):


% ncdump -h  FuelMoisture.nc

netcdf FuelMoisture {
dimensions:
       longitude = 99 ;
       latitude = 99 ;
       hours = 24 ;
variables:
       float longitudes(latitude, longitude) ;
               longitudes:units = "index" ;
               longitudes:stagger = "none" ;
               longitudes:description = "computed" ;
....
       float MC10(hours, latitude, longitude) ;
               MC10:units = "percent" ;
               MC10:stagger = "none" ;
               MC10:description = "ten day percent moisture content" ;
               MC10:memoryOrder = "XYZ" ;
       float BNDRY1000(days, latitude, longitude) ;
               BNDRY1000:units = "forgotten" ;
               BNDRY1000:stagger = "none" ;
               BNDRY1000:description = "100 day % moisture cntt" ;
               BNDRY1000:memoryOrder = "XYZ" ;
       float MC1000(days, latitude, longitude) ;
....

Compiler Stories

[ By: Kate Hedstrom ]

Those of us who develop software know how hard it is and know how many bugs are out there running around. What you may not realize is that the occasional bug isn't even yours, but that of the compiler vendor.

  1. The optimizer is a good candidate for many troubles. Try compiling without optimization (-O0) to see if you get the same answer. Your code will run like a snail, but you need to find out if it's right before performance tuning. When I started using the ocean model I use, they liked to keep it all in one file, but had it in three for the Masscomp computer. Through much agony and effort, my colleagues had determined that they could aggressively optimize the number crunching routines, but not the I/O routines.

  2. Automatic parallelizers can also do automatic bug insertion. I once spent a week tracking down what looked like an off-by-one indexing error in my plots to the auto-parallel compiler on the Alliant. I don't know why it took me so long to go back to serial mode for comparison.

  3. Adding a print statement might change how the compiler views your code. I was having some odd behavior on the Connection Machine and I added a print statement to see what the compiler thought was the value of something. It then printed the value I expected - and the code ran perfectly!

  4. Hidden corners of the language are less well tested than the commonly used parts. I was an alpha tester for g77 and had the most fun with a program using complex arithmetic. Most people use reals and integers so the complex data type had more lingering bugs.

  5. Try other compilers if you possibly can. If I'm having problems, I try different optimizations, but I also love to try other compilers, if only for getting alternate warning messages. The NEC compiler was incredibly picky and verbose, which is great when you're trying to track something down. Also, many compilers have some sort of bounds-checking capability which can be worth trying.

  6. If think you smell a compiler bug, what next? You can find a way to work around it and get on with life, like my friends did on the Masscomp. If it is on one of the ARSC systems, you can report it to consult, who will get it reported to the compiler vendor. The vendors will be more or less responsive, depending, but most of my reported IBM bugs have been fixed. Finding a way to get your work done in the mean time is usually a good idea, though.

  7. If you report it, the shorter the code the better. No one wants to wade through 200 source files in 15 directories, using umpteen external libraries. Get an example code demonstrating the problem into one short file if possible. I'm not saying this is easy, but it will be easier for you who know the code than for the compiler people. As you shorten the code, keep a prior version demonstrating the trouble because at some point in your whacking the trouble will likely disappear. Go back to the previous version and whack in a different area. Once you get it "short enough", write up the expected results, the actual results, and how to compile and run your code snippet.

Quick-Tip Q & A



A:[[ Do vi or VIM offer any way to repeat commands issued at the colon
  [[ prompt (i.e., "ex" commands)?  For instance, I might want to execute
  [[ the following search and replace on one line and then hop to another
  [[ line and do it again:
  [[
  [[     :s/-/=/g
  [[



  #
  # Thanks to Ryan Czerwiec
  #

  I'm not aware of such a command, and the vi/vim help doesn't list one.
  If you happen to want to change every instance in the file, you can hit
  Ctrl-g to see how many lines there are, then preface your search command
  with the range of line numbers desired:

    :1,100 s/-/=/g

  for instance.  I believe there's a quicker shorthand than looking up the
  last line number if you do want the entire file.  If you want to check
  each instance rather than do a blanket search-and-replace, use gc
  instead of just g as the option, and it will ask you for confirmation at
  each instance of the search string.  If there are too many instances for
  this to be practical, I usually just highlight the command with my mouse
  and use the cut-and-paste capabilities of the mouse buttons to
  re-execute with a click.  As a bit of a shorthand, you can eliminate the
  search strings, and it will assume they are the same if you repeat your
  search:
  :s g
  :101,200 s g 


  #
  # Thanks to Lee Higbie
  #

  One easy way to repeat vi or vim commands:

  Once you type your command, ":s/-/=/g," for example, copy it to your
  terminal's buffer, then type the Enter key.  Move to the new location,
  paste from the terminal and type the Enter key and so on.

  Using a local machine or its terminal program to an HPC machine running
  the editor:

     Type the command (don't type the Enter key yet) Use the mouse to
     highlight the command and copy it (Ctrl+c, Cmd+c, or mouse click
     depending on your setup) Type the Enter key move the cursor to the
     new line you want to edit Paste (Ctrl+v, Cmd+v, or mouse click)
     Type the Enter key

  This works because the first character in the copy buffer is the first
  character of the command, the colon.  When you paste from the buffer,
  vi or vim interpret it just as though you had typed the command.


  #
  # Thanks to Kate Hedstrom
  #

  Yes, you can. ":%" maps to all lines. So you can do:

    :%s/^C/!/

  to convert old Fortran comments to new ones. In fact, I've got a .exrc
  file with "map g :%" in it (from an ancient Sun starting Unix guide)
  so that what I type is "gs/^C/!/" for global substitution.


  #
  # Thanks to Oralee Nudson
  #

  Type an "&" while in escape mode to repeat the last "s" command.


  #
  # Editor's solution
  #

  In VIM, the register ":" contains the most recent ex command.  The
  command "@:" executes the register ":".  So, to answer the question,
  simply hop to the next line and type, "@:".

  As another example, an easy way to replace the first 16 "-'s" with
  "='s" on the same line would be this sequence:

    :s/-/=/
    15@:





Q: I am currently debugging my code by submitting jobs to the "debug"
   queue on midnight.  To properly test my PBS scripts, I need to submit
   a handful of jobs at a time to see if each job will pick up where the
   previous job left off.  I often find a bug early in the job sequence,
   resulting in a handful of test jobs still queued without any real
   purpose.  Rather than waiting for each of these jobs to finish,
   I have been using the "qdel" command to cancel them.  For example:

     % qdel 123456 123459 123479 123480

   Depending on the number of debug jobs still queued, it can become
   tedious to type or copy and paste each of these job IDs for every
   iteration of my debugging efforts.  Is there an easier way to delete
   all of my jobs in the debug queue?

[[ Answers, Questions, and Tips Graciously Accepted ]]


Current Editors:
Ed Kornkven ARSC HPC Specialist ph: 907-450-8669
Kate Hedstrom ARSC Oceanographic Specialist ph: 907-450-8678
Arctic Region Supercomputing Center
University of Alaska Fairbanks
PO Box 756020
Fairbanks AK 99775-6020
E-mail Subscriptions: Archives:
    Back issues of the ASCII e-mail edition of the ARSC T3D/T3E/HPC Users' Newsletter are available by request. Please contact the editors.
Back to Top