ARSC HPC Users' Newsletter 355, January 26, 2007

Volcano Research and ARSC Post-Doc

Here's an interesting feature on one of the ARSC Post-Doctoral Fellows and his work:

Passing Environment Variables to Loadleveler Jobs

One of the things that surprised me when I started using LoadLeveler was that you could not pass command line arguments to a script when it was submitted. If you try, you will get an error like this:


   iceberg2 1% llsubmit myjob.ll 100 200
     llsubmit: 2512-034 File 200 not found.
     llsubmit: 2512-055 Unable to process the job command file "200" for
               input, the error is: A file or directory in the path name
               does not exist..
     llsubmit: 2512-051 This job has not been submitted to LoadLeveler.

I've used a couple of methods to solve this:

  1. hard code the information into the job script, or
  2. read the information from a text file.

Neither of the options is really elegant. With option (a), you need a new script for each set of command line arguments you use. With option (b), you need a new input file for each additional job you run.

However there's a slightly more graceful way to pass information to Loadleveler scripts, using environment variables.

If the LoadLeveler keyword "environment" is set to "COPY_ALL", all environment variables from your login environment will be copied to your LoadLeveler script when you submit it. With this feature you can set an arbitrary environment variable to be used by your job script:


   iceberg2 2% export NUM_ITERATIONS=19
   iceberg2 3% llsubmit myjob.ll

Where the job script looks like this:

   iceberg2 4% cat myjob.ll
   # @ error   = $(executable).$(jobid).eo
   # @ output  = $(executable).$(jobid).eo
   # @ notification  = never
   # @ job_type = parallel
   # @ environment = COPY_ALL
   # @ node = 1  
   # @ tasks_per_node = 8
   # @ network.MPI = sn_all,shared,us
   # @ node_usage = not_shared
   # @ class = standard
   # @ wall_clock_limit=3600
   # @ queue

   # Check to see if $NUM_ITERATIONS is set:
   if [ ! -z "$NUM_ITERATIONS" ]; then
       echo Job input=$NUM_ITERATIONS
       # Run a job using $NUM_ITERATIONS as a command line argument
       poe ./a.out $NUM_ITERATIONS 

 exit 1   
       echo "Error \$NUM_ITERATIONS not set" 1>&2
       echo "exiting..." 1>&2
       exit 1

There's one minor annoyance with this design. If you forget to set the environment variables (e.g., $NUM_ITERATIONS) before you submit the job, it will exit immediately. One improvement would be to write a simple wrapper script to ensure that all of the necessary variables are set, e.g.,:

   iceberg2 5% run_job.bash
   if [ $# -eq 1 ]; then
       export NUM_ITERATIONS=$1
       llsubmit myjob.ll
       BASE=$(basename $0)
       echo "Usage: $BASE iterations" 2>&1
       echo "" 2>&1                      
       exit 1

Using this wrapper script, if you happen to forget to set the command-line options, an error message will be displayed:

   iceberg2 6% ./run_job.bash
   Usage: run_job.bash iterations

Environment variables can also be used in chained jobs to pass information between successive runs. Just be sure to include the "environment=COPY_ALL" keyword and option combination in all of the Loadleveler scripts and update the environment variables you are using before submitting the chained script.

Iceberg, Multiple Compiler Versions Available

Occasionally you may want to reproduce an earlier compilation of your code, using an old compiler version (for comparison with a new compiler, for instance). More often, you may want to try the very latest compiler, which has not yet been made the general default.

"Modules," which Cray has used for almost a decade to manage multiple versions of programming environments and packages on its systems, are gaining popularity elsewhere. E.g., they're now included in the SUSE Linux distribution.

ARSC has installed modules on both iceflyer and iceberg as a local modification.

To give you the flavor of modules on iceberg, here's a sample session in which the user does the following:

  1. Checks the current mpxlf90 version (mpxlf90 -qversion)
  2. Learns about the modules command (news modules)
  3. Initializes modules for ksh user (. /usr/local/pkg/modules/init/ksh)
  4. Determines which modules are available (module avail)
  5. Determines which modules are already loaded (module list)
  6. Loads the latest xlf module (module load xlf.
  7. Checks which modules are already loaded, again (module list)
  8. Checks the current mpxlf90 version, again (mpxlf90 -qversion)

b1n1 % mpxlf90 -qversion
IBM XL Fortran Enterprise Edition V10.1 for AIX   
Version 10.01.0000.0001  
b1n1 % news modules 
[... cut ...]
b1n1 % . /usr/local/pkg/modules/init/ksh      
b1n1 % module avail

--------------- /usr/local/pkg/modules/modules-3.2.2/Modules/versions ----------------

----------- /usr/local/pkg/modules/modules-3.2.2/Modules/3.2.2/modulefiles -----------
dot         module-cvs  module-info modules     null        use.own

---------------------------- /usr/local/pkg/modulefiles/ -----------------------------
SciPy             idl               ncview            tau               vac.
ferret            idl-6.2           ncview-1.92e      tau_64            xlf.
ferret-5.41b      ncl               ncview-1.93b      vac.       xlf.
gnuplot           ncl-4.2.0.a030_64 numpy-0.9.8       vac.      xlf.
gnuplot-4.0.0     ncl-4.2.0.a033_64 python-2.4.3      vac.      xlf.
b1n1 % module list
No Modulefiles Currently Loaded.
b1n1 % module load xlf.
b1n1 % module list             
Currently Loaded Modulefiles
  1) /xlf.
b1n1 % mpxlf90   -qversion                    
IBM XL Fortran Enterprise Edition V10.1 for AIX   
Version 10.01.0000.0003  

Stay tuned to "news compilers" for announcements of updates to the default compilers. We recommend you avail yourself of the "modules" command to test new compilers, prior to upgrades. If you use "modules" frequently, you might add the appropriate initialization step to your shell startup file.

Quick-Tip Q & A

A:[[ Too many times, I've wanted to clean out a few files with a command
  [[ like this:
  [[     % rm 200612*.dat old *.txt  junk*
  [[ but was typing too fast and entered something like this,
  [[     % rm 200612*.dat old * txt  junk*
  [[ thus deleting everything.
  [[ I absolutely do not have the patience to make a habit of "rm -i". I'd
  [[ just like to disable "rm *".  Is there a way to do this?  Any other
  [[ great ideas to keep me from deleting everything with the flub of
  [[ a finger?

  # Thanks to Liam Forbes:

  Write your own rm command to first echo the globbed command line and
  then prompt only once whether to continue or not.  A simple script
  placed in one's own bin directory that does this can then be aliased to


  echo /bin/rm $*
  echo "Are you sure [y/N]"
  read answer

  if [ ":y:" == ":$answer:" ]; then
    /bin/rm $*
    echo "okay then"

  Here's an example; I want to delete all the numbered ones but keep the original, unnumbered copy.

  nelchina$ alias rm="~/bin/rm"
  nelchina$ ls -l
  total 8
  drwx------   2 lforbes staff 4096 2007-01-12 18:11 ./
  drwxr-xr-x  15 lforbes staff 4096 2007-01-12 18:10 ../
  -rw-------   1 lforbes staff    0 2007-01-12 18:10 test
  -rw-------   1 lforbes staff    0 2007-01-12 18:11 test.1
  -rw-------   1 lforbes staff    0 2007-01-12 18:11 test.2
  -rw-------   1 lforbes staff    0 2007-01-12 18:11 test.3
  -rw-------   1 lforbes staff    0 2007-01-12 18:11 test.4
  -rw-------   1 lforbes staff    0 2007-01-12 18:11 test.5
  nelchina$ rm * .*
  /bin/rm test test.0 test.1 test.2 test.3 test.4 test.5 . ..
  Are you sure [y/N]
  okay then

  The nice thing is that any command line options you would use with rm can be used this way because they are passed to the command by $*.   
  So if you wanted to use "-i" with rm, you would be prompted one extra time.

  nelchina$ rm -i *.*
  /bin/rm -i test.0 test.1 test.2 test.3 test.4 test.5
  Are you sure [y/N]
  /bin/rm: remove regular empty file `test.0'? y
  /bin/rm: remove regular empty file `test.1'? y
  /bin/rm: remove regular empty file `test.2'? y
  /bin/rm: remove regular empty file `test.3'? y
  /bin/rm: remove regular empty file `test.4'? y
  /bin/rm: remove regular empty file `test.5'? y

  # Rich Griswold gives us the ZSH answer:

  ZSH handles rm as a special case:

  % rm 200612*.dat old * txt  junk*
  zsh: sure you want to delete all the files in /home/richard/data [yn]? n

  # And Don Bahls stumbled on this tcsh answer:

  If you are using tcsh, you can set rmstar=1 to have tcsh prompt you 
  when "rm *" is used:

    nelchina% set rmstar=1
    nelchina% touch 1 2 3 4 5
    nelchina% rm *
      Do you really want to delete all files? [n/y] y

  If you don't want tcsh to prompt, simply do this:

    nelchina% unset rmstar

Q: I'd like to pipe "ls -l" into "cut" to extract certain fields,
  like group, size, and name, and I'd like to use spaces as delimiters,
  because it's easer to count the fields.  So with this "ls -l" output:

    -rw-------  1 robert jstme  3183 2007-01-23 14:57 file1
    -rw-------  1 robert  them   973 2006-09-15 08:19 file3
    -rw-------  5   bert justm 15096 2006-11-16 13:04 file2

  I tried:  

    ls -l 
 cut -f4,5,8 -d' '

  but it gave me this garbage:

    robert jstme 2007-01-23

  Am I doing something wrong?

[[ Answers, Questions, and Tips Graciously Accepted ]]

Current Editors:
Ed Kornkven ARSC HPC Specialist ph: 907-450-8669
Kate Hedstrom ARSC Oceanographic Specialist ph: 907-450-8678
Arctic Region Supercomputing Center
University of Alaska Fairbanks
PO Box 756020
Fairbanks AK 99775-6020
E-mail Subscriptions: Archives:
    Back issues of the ASCII e-mail edition of the ARSC T3D/T3E/HPC Users' Newsletter are available by request. Please contact the editors.
Back to Top