ARSC HPC Users' Newsletter 353, December 01, 2006

Storage Upgrades

On December 15th and 16th, ARSC will be performing several significant long-term storage infrastructure upgrades. During this upgrade, the StorageTek Powderhorn tape silos which have been in operation since the mid-1990's will be retired and replaced with a StorageTek SL8500 tape silo. The SL8500 silo will have a number of improvements over the older Powderhorn silos including:

  • multiple robotic arms for tape retrieval providing improved performance.
  • redundant robotics for improved reliability.
  • increased storage capacity through the use of T10000 (500GB) and T9940 (200GB) tapes.
  • more cost effective expansion options for the tape library.

While the silo transition is occuring, both seawolf and nanook will be undergoing IO boat upgrades. The new IO boats include PCI-X bus slots which will allow both systems to use higher bandwidth fibre channel adapters to improve data transfer rates.

These upgrades are the first of several upgrades to long term storage infrastructure planned over the next few months. Watch for additional information in the HPC Users Newsletter as those upgrades get nearer.

Environment Modules Available on iceberg and iceflyer

The module command is now available on iceberg and iceflyer. As we've mentioned in the past (see > issue 342 ) the module command is a great way to handle multiple versions of software. It's also a convenient way to document environment variables that are required for a package.

The use of modules on iceberg and iceflyer is completely optional, however if you would like to use new packages before they are installed for general use, you may want to try out modules. Also if you use a package like ncl or ferret that requires that a number of environment variables be set, you may want to use the module so you don't have to figure out how to set up the package.

To use modules you first source the appropriate init file for your shell.

csh and tcsh users:


  source /usr/local/pkg/modules/init/
<shell>

e.g. for tcsh:


  source /usr/local/pkg/modules/init/tcsh  

sh, ksh and bash users:


   . /usr/local/pkg/modules/init/
<shell>

e.g. for ksh:


   . /usr/local/pkg/modules/init/ksh   

If you intend to use modules each time you login, you can also add the init command above to your .login (for csh and tcsh) or your .profile (for sh, ksh and bash).

Once the init file has been sourced you will be able to use modules.


e.g.
iceberg2 1% module avail
...
...
------------------------ /usr/local/pkg/modulefiles/ ---------------
SciPy             gnuplot-4.0.0     ncl-4.2.0.a030_64 numpy-0.9.8
ferret            idl               ncl-4.2.0.a033_64 python-2.4.3
ferret-5.41b      idl-6.2           ncview-1.92e      tau
gnuplot           ncl               ncview-1.93b      tau_64

Assuming you have sourced the init file and loaded the module(s) you need, the environment variables set by the module(s) can be copied into a batch job, by including the "environment = COPY_ALL" option within your loadleveler script.

e.g.


# @ environment = COPY_ALL

Modules can also set aliases. If you are using such a module, you will need to source the init file and load the module from within the loadleveler script itself.


#!/bin/bash
# @ output = $(executable).$(jobid).eo 
# @ error = $(executable).$(jobid).eo 
# ...
# ...
# @ queue

# If a module uses aliases you will need to source the init file
# from within the loadleveler script.

# Be sure to use the init file for the shell used in the script.
# Here the script has /bin/bash specified on the first line.

. /usr/local/pkg/modules/init/bash
module load module-using-aliases 

If you have questions about using modules on iceberg or iceflyer, contact the ARSC Help Desk (consult@arsc.edu).

Deleting Files from Fortran

There are a few ways to delete files from within a Fortran program.

  1. Open the file with " status=scratch ":

    While you can't assign it a specific name, if a file is opened with " status=scratch " it will be automatically deleted when closed.

    
        open (29, form='unformatted', status='scratch')
    
        ...
    
        close (29)
    
  2. Use " system ":

    It always strikes me as bad form to do something that can be done within the constructs of a language, but you can issue shell commands, including " rm ", using the intrinsic function, " system ":

    
          character(len=64) :: cmd, filename
          logical :: filefound
    
          ...
    
          inquire (file=filename, exist=filefound)
          if (filefound) then
            write (cmd, '("/bin/rm ", A)' ) trim (filename)
            call system (cmd)
          endif
    
  3. Close the file with " status=delete ":
  4. When a file is closed with " status=delete ", the file will be removed as soon as the close operation is performed.

    
          character(len=64) :: cmd, filename
    
          ...
    
          open (29, file=filename, form='unformatted')
    
          ... 
    
          close (29, status='delete')
    

Quick-Tip Q & A


A:[[ I really like command completion in bash and tcsh.  Is there a 
  [[ way to get the shell to do command completion for hostnames? It
  [[ would be nice not to have to type the whole hostname when I ssh
  [[ to another machine.  E.g., I'd like:
  [[
  [[   ssh iceb[TAB]
  [[
  [[ To be expanded to: 
  [[
  [[   ssh iceberg.arsc.edu
  [[
  [[ Yes, I am really this lazy.
  [[

#
# Thanks to Martin Luthi for this response
#

Even easier to use are aliases for every machine you log in
frequently. You can give all arguments (different usernames or
additional flags), and get tab completion like for any other command.

[in .bashrc if you use bash]
  alias iceberg='ssh -X -l luthi iceberg.arsc.edu'

[in .cshrc if you use csh or tcsh]
  alias iceberg "ssh -X -l luthi iceberg.arsc.edu"

I usually have a file named .alias that gets imported from .bashrc
which also contains aliases for the directories I frequently change
to, like

  alias cdpj='cd ~/projects/jako'

and also for documentation I frequently need to access

  alias doc_numpy='zxpdf /home/tinu/doc/numpybook.pdf.gz &'

Quite convenient if you do everything from the command line..


#
# And thanks to Greg Newby for another alias-based solution.
#

If you're as lazy as I am, why not just create an alias for your
common hosts.  Syntax is different for bash vs. tcsh.

For csh/tcsh, put this in your .cshrc or similar:
  alias iceberg 'ssh iceberg.arsc.edu'

For bash & variants, the syntax requires an equals sign:
  alias iceberg='ssh iceberg.arsc.edu'

You could add arguments too.  Of course, if you're as lazy as me,
you'll make much shorter aliases (just watch out for aliases that
might mask legitimate system commands):

  alias i 'ssh iceberg.arsc.edu'
  alias ix 'ssh -X iceberg.arsc.edu'


#
# Lorin Hochstein shared this solution using ssh config file
# options.  
#

It doesn't use tab-completion, but you can get the same effect
by creating a file called "config" in your ~/.ssh directory. This
allows you to define ssh aliases. Just create the file and add the
following lines:

  Host iceb
  HostName iceberg.arsc.edu


At the shell, you can now just type:

  ssh iceb

and it will have the desired effect. You can also use this if your
username is different on the remote machine, as follows:

  Host iceb
  HostName iceberg.arsc.edu
  User mylogin

Now "ssh iceb" is an alias to  "ssh mylogin@iceberg.arsc.edu.

Doing "man ssh_config" will tell you about all of the other options
available.


#
# Rich Griswold shared this solution for the zsh shell.
# 

In ZSH, you can use the compctl command with a hosts list:

  hosts=(google.com iceberg.arsc.edu foo.bar.net localhost.localdomain)
  compctl -k hosts ssh telnet rlogin finger rsh ftp ncftp ping traceroute

#
# Editors Note: for more information on the compctl command in ZSH see
# "man zshcompctl"
#


#
# Thanks to Jeremiah Dabney for this solution using the complete 
# command in tcsh.
#

The ssh command saves all hosts that have been accessed in 
your ~/.ssh/known_hosts file.  In tcsh this can be parsed and
saved into the hosts variable like this.

  set hosts=`cat .ssh/known_hosts 
 sed -e 's/,/ /' 
 awk '{print $1}'`
        
Then setting up tcsh command completion to use that list when using
ssh, slogin, sftp, and kftp, you would using the following completion
command.

  complete {ssh,slogin,sftp,kftp} 'p/1/$hosts/'

Other applications can be added that also need hostname completion by
added it along with a comma before the ssh. So adding krlogin would
look like     this.
                        
  complete {krlogin,ssh,slogin,sftp,kftp} 'p/1/$hosts/'

                
#
# Here's a solution from the editors using the 'complete' command which  
# is built into bash. 
#

Bash has a built-in command called "complete" which lets you specify
what options to use with a command for command completion.  The "-W"
flag for complete lets you specify a valid "word list" for a command,
in the case of ssh the word list is hostnames.

Here's a simple example which will expand the hostnames iceberg, iceflyer 
or seawolf.

% complete -W "iceflyer.arsc.edu iceberg.arsc.edu seawolf.arsc.edu" ssh

In the case of ssh, you can use the ~/.ssh/known_hosts files to get the 
names of the systems that you regularly access to build the word list.

Here's one way to do that:

e.g.
% complete -W "$(cat ~/.ssh/known_hosts 
 tr , ' ' 
 cut -d ' '  -f 1  
 tr '\n' ' ')" ssh



Q: When writing "for" loops in ksh, I really hate typing a sequence
   of numbers in.

   e.g.
   for num in 1 2 3 4 5 6 7 8 9 10 11 12; do
       # do something with $num   
   done

   Is there a way to specify a range of values to iterate through
   in ksh?  
   
   In C it's so easy.
   
   e.g.
   for(ii=1;ii<=12;++ii) 
   {
   /* do something */
   }

   There's got to be a better way to do this!
      

[[ Answers, Questions, and Tips Graciously Accepted ]]


Current Editors:
Ed Kornkven ARSC HPC Specialist ph: 907-450-8669
Kate Hedstrom ARSC Oceanographic Specialist ph: 907-450-8678
Arctic Region Supercomputing Center
University of Alaska Fairbanks
PO Box 756020
Fairbanks AK 99775-6020
E-mail Subscriptions: Archives:
    Back issues of the ASCII e-mail edition of the ARSC T3D/T3E/HPC Users' Newsletter are available by request. Please contact the editors.
Back to Top