ARSC HPC Users' Newsletter 333, January 27, 2006

IBM: Using Linker Address Maps with Multiple Libraries

Last week's unforgettable article on LAPACK/ESSL and ScaLAPACK/PESSL described how LAPACK on iceberg is installed with the Call Conversion Interface (CCI) which substitutes ESSL for LAPACK routines when possible.

For various reasons, you might what to know which library a given LAPACK routine is coming from (e.g. if you're getting unexplained numeric or performance differences).

Turns out it's pretty easy...

Simply pass this option to the linker (where you replace "<FILE NAME>" with a name for an actual output file):

  "-bmap:<FILE NAME>"  

If you're linking with xlf90, use this xlf90 syntax to pass an option through to the linker:

  "-Wl,-bmap:<FILE NAME>"

From "man ld," the "-bmap" option "writes an address map of the output object file." Part of the address map is the source file of each routine, so the next step is to simply grep "<FILE NAME>" for your subroutine of interest.

Here's an example, using the "testeig" program from this tutorial at ZIB/HLRN:

The "testeig" program calls the eigenvalues/eigenvectors routine DSPEV. DSPEV is one of several routines which has the same name in both LAPACK and ESSL, but has different call argument sequences. CCI knows about the name conflict and can't substitute the ESSL version for the LAPACK version. ("Testeig" uses its own pre-processor macros, USE_LAPACK and USE_ESSL, to select between the LAPACK and ESSL argument sequences, respectively.)

First, we compile to use the ESSL argument sequence, and link, giving the linker the "-bmap" option and specifying "-lessl" before "-llapack":

  xlf90 -qsuffix=cpp=F90 -WF,-DUSE_ESSL -q64 -I/u2/wes/PET_HOME/MATH/IBM/include -c testeig.F90
  xlf90 -q64 -L/u2/wes/PET_HOME/MATH/IBM/lib64 -Wl,-bmap:map.out -o testeig testeig.o dlatms.o dlatm1.o dlarnd.o dlarot.o dlagge.o dlagsy.o dlaran.o -lessl -llapack 

A grep of the address map file shows us that DSPEV was taken from libessl.a:

  $ egrep -i dspev map.out   
   I                               DS ER S22   dspev      {/usr/lib/libessl.a[essl_64.o]}
      0000000100016068 00000028  2 GL SD S272  <.dspev>   glink64.s(/usr/lib/glink64.o)
      0000000100016068             GL LD S273  .dspev
      00000001100045A8 00000008  3 TC SD S423  <dspev>

Now, compile to use the LAPACK argument sequence, and link, giving the linker the "-bmap" option and this time specifying "-llapack" before "-lessl":

  xlf90 -qsuffix=cpp=F90 -WF,-DUSE_LAPACK -q64 -I/u2/wes/PET_HOME/MATH/IBM/include -c testeig.F90
  xlf90 -q64 -L/u2/wes/PET_HOME/MATH/IBM/lib64 -Wl,-bmap:map.out -o testeig testeig.o dlatms.o dlatm1.o dlarnd.o dlarot.o dlagge.o dlagsy.o dlaran.o -llapack -lessl 

A grep of the new address map file shows us that, this time, DSPEV was taken from liblapack.a:

  $ egrep -i dspev map.out                                               
      000000010001D590 00000548  3 PR SD S271  <>             dspev.f(/u2/wes/PET_HOME/MATH/IBM/lib64/liblapack.a[dspev.o])
      000000010001D590             PR LD S272  .dspev
      0000000100029530 00000058  3 RO SD S354  <>             dspev.f(/u2/wes/PET_HOME/MATH/IBM/lib64/liblapack.a[dspev.o])
      0000000110008E88 00000078  3 RW CM S561  <_$STATIC_BSS> dspev.f(/u2/wes/PET_HOME/MATH/IBM/lib64/liblapack.a[dspev.o])

Numerical Instability

[ Thanks to Lee Higbie of ARSC ]

ARSC Consult recently received a request to help locate the source of a floating point overflow in a program. This interrupt occurred after the program had been running for a long time, so it was expensive to rerun to the fault point repeatedly, trying to isolate the error.

In a case like this, one error source comes to mind that should be included in your "bag of errors." Classically, numeric instability arises in a numerical partial differential equation solver when the time step is too large or the integration scheme needs to be modified for the specific domain. This may occur after a program has been running for a long time and some interesting results are starting to appear.

Typical Symptoms:
  1. A working differential-equation-solving program blows up with a floating point error after running its integration for many time steps.
  2. The solution to a program starts having jaggies, i.e., the solution curve starts developing a jagged appearance. Because output graphs are normally done only at long intervals (many time steps) jaggies may never show in the output. Adjusting the output so you can see the data just before the crash should show the jaggies in at least one of the variables.
Numerical Execution:

Normally, numerical instability causes the solution to oscillate around the desired curve with solution errors alternating in sign and doubling in magnitude with each integration step. In typical double precision computation, a one-bit error in a field that is near 1.0 has an error of approximately order 1.0e-15. In 50 steps, with 50 doublings of the error, the error is about the size of the data. In another 1000 steps, the doubling in error magnitudes causes an overflow (2**1000 is the approximate max double precision number).

Solution Techniques:
  1. Decrease the time step in the integration.
  2. Use a solver that is more stable. For example, implicit solvers tend to be much more stable but much slower than explicit ones.
  3. Add some viscosity to the physics. When adding viscosity it is important to be sure that momentum, energy, and so on are still conserved.
  4. Rethink the entire solution approach and change the physics. For example, instead of trying to predict weather far into the future using short-term models, use climactic-variable physics.

Often a combination of these approaches is used. For example,

  1. An implicit integration step may be used occasionally to smooth the data fields in an integration that nearly always uses a faster explicit integrator.
  2. The time step may be decreased in zones where the fluid is behaving in interesting ways. Areas of steep gradients are where instability is more likely to begin.
  3. An artificial viscosity may be used that only affects grid cells that cover or are adjacent to shock waves. Because shock waves represent such extreme gradients, they often require special attention.

IA Training Reminder

Reminder: the deadline to complete Information Assurance Awareness Training is Jan 31st. ARSC users that have not completed IA Training by that date will have their accounts disabled until the training is completed.

Links to IA Training documents are available here:

Winter Entertainment in Fairbanks

The cold winter air has settled into Fairbanks once again. This morning temperatures were reported as -51 F (-46 C) at the Fairbanks International Airport. At this temperature a cup of hot water thrown into the air will freeze before it hits the ground. If you don't believe it, check out these videos:

Note: These videos were recorded when the temperature was a balmy -42 F (-41 C).

Quick-Tip Q & A

A:[[ I tried "nm" on an IBM ".a" file, trying to figure out what routines
  [[ it contains, but it returns nada:
  [[  $  file liblapack.a
  [[  liblapack.a: archive (big format)
  [[  $  nm liblapack.a
  [[  $
  [[ Is the archive really empty?  What's wrong?

  # The answer, courtesy of John Skinner: 

  Use -X flag to list any 32-bit and 64-bit object files in an archive.
  Some IBM libraries contain both 32-bit (the default object mode under
  AIX) and 64-bit object files in the same library, while other
  libraries may contain only 32-bit or 64-bit objects.

  However, nm's default output is to only list 32-bit objects.

  From the nm man page:
    -X mode
      Specifies the type of object file nm should examine. The mode must be
      one of the following:

      Processes only 32-bit object files
      Processes only 64-bit object files
      Processes both 32-bit and 64-bit object files

  The default is to process 32-bit object files (ignore 64-bit
  objects).  The mode can also be set with the OBJECT_MODE environment
  variable. For example, OBJECT_MODE=64 causes nm to process any 64-bit
  objects and ignore 32-bit objects. The -X flag overrides the
  OBJECT_MODE variable.


  # Nothing is found because only 32-bit objects inside archive

    % setenv OBJECT_MODE 64
    % nm /usr/lib/libpessl.a

  # Now try "nm -X32_64 ... ":

    % nm -X32_64 -A /usr/lib/libpessl.a
    /usr/lib/libpessl.a[pessl.o]:                      f           -
    /usr/lib/libpessl.a[pessl.o]: .$PTRGL              T      468560
    /usr/lib/libpessl.a[pessl.o]: .$RESTF14            T      100644
    /usr/lib/libpessl.a[pessl.o]: .$RESTF15            T      100648

Q: I've noticed that a lot of people, maybe 2-5%, put blankets on the
   hoods of their cars when it's really cold.  Does this actually do
   any good?  Why or why not?

[[ Answers, Questions, and Tips Graciously Accepted ]]

Current Editors:
Ed Kornkven ARSC HPC Specialist ph: 907-450-8669
Kate Hedstrom ARSC Oceanographic Specialist ph: 907-450-8678
Arctic Region Supercomputing Center
University of Alaska Fairbanks
PO Box 756020
Fairbanks AK 99775-6020
E-mail Subscriptions: Archives:
    Back issues of the ASCII e-mail edition of the ARSC T3D/T3E/HPC Users' Newsletter are available by request. Please contact the editors.
Back to Top