ARSC HPC Users' Newsletter 338, April 14, 2006

Contents

• Interactive Shell Nuances
• Long Term Storage Commands Available
• Faculty Camp 2006, July 31 - August 18
• Quick Tip

Interactive Shell Nuances

Over the past few years, I've heard a lot of questions about shells. Many of these questions have to do with the interactive use of the shell (i.e. at the command line). Here are some of the more common things that I've been asked and questions that people would ask if they knew to ask.

Flow-Control Characters:

shells: tcsh, bash

description:

Not so long ago I found myself typing away when my shell mysteriously "locked up". When I typed in the terminal nothing appeared on the screen. My other connections to the system were all responding so I knew something was up.

solution:

The culprit in this case was flow-control characters. You see, I had mistakenly typed Ctrl+S, which stops the flow of output to the terminal and makes it look like your shell has hung. You can undo the operation by typing Ctrl+Q.

Or better yet, if you decide that Ctrl+S and Ctrl+Q are an annoyance rather than a useful feature, you can disable both using the stty command.

E.g.

    stty -ixon
    

If you really want to turn the flow-control characters back on for some reason, issue:

    stty ixon
    

See "man stty" for other options.

time command

shells: tcsh, csh

description:

When running time on a command from a csh or tcsh shell, the output looks different than the output I get with bash or ksh.

solution:

In csh and tcsh time is a built-in shell command.

    > which time
    time: shell built-in command.
    

As with aliases, built-in commands are preferred over executables in the PATH.

To use the time executable rather than the shell built-in command run:

    #uses time from PATH
    iceberg1 % \time ./executable
    

or

    #specify the full path to the time command
    iceberg1 % /usr/bin/time ./executable
    

instead of

    #uses built-in time command
    iceberg1 % time ./executable
    

Here's another example showing the output of which with and without the time command escaped.

    iceberg1 % echo $0
    tcsh
    iceberg1 % which \time
    /usr/bin/time
    iceberg1 % which time
    time: shell built-in command.
    

Ignoring an alias

shells: bash, ksh, csh, tcsh

description:

Aliases can be quite convenient, but occasionally I find that they get in the way. Is there a way to ignore an alias without unaliasing it?

solution:

Assume the following ls alias is set:

    #ksh/bash syntax
    alias ls="ls --color"

    #csh/tcsh syntax
    alias ls "ls --color"
    

As with the shell built-in commands, you can override aliases by escaping the command.

    #uses the alias
    nelchina % ls 
    
    #ignores the alias    
    nelchina % \ls
    

logout on EOF

shells:bash, ksh, csh, tcsh

description:

Okay, I may not be the most gracefully typist in the world, but do I really deserve to get logged off a system for a simple typing mistake? I pressed Ctrl+D and it logged me out!

solution:

In bash and ksh you can tell the shell to ignore EOF (i.e. Ctrl+D) for logout by issuing the following:

    set -o ignoreeof
    

In csh and tcsh the variable is named the same, but is set in a slightly different way:

    set ignoreeof
    

Command History

shells: ksh
description:

Does korn shell have command history, I'm trying the arrow keys and they don't do anything?

solution:

If you are using the vi bindings, you can access command history by pressing ESC then 'j' (down) or 'k' (up)

Setting Vi Bindings

shells: bash, tcsh
description:

If you like to use vi bindings for your shell, you're in luck, both tcsh and bash support the vi and emacs binds.

solution:

In bash, you can switch to vi bindings by issuing:

    set -o vi
    

Or switch to emacs binding using:

    set -o emacs
    

In tcsh, the bindkey shell built-in command allows one to switch between binding types

To use vi bindings:

    bindkey -v 
    

To use emacs bindings:

    bindkey -e 
	

Long Term Storage Commands Available

ARSC has two SunFire 6800 servers (seawolf and nanook) which house data stored on the long term storage filesystem ($ARCHIVE). Files on the $ARCHIVE filesystem can be either online or offline. When a file is online it is stored on disk, while offline files are stored only on tape. If you attempt to access a file which is offline, the storage server will automatically bring the file back online from tape. While this operation is done automatically, it can be less than optimal when attempting to retrieve large numbers of files from tape.

In the past, the best way to improve retrieval of offline files was to log directly on to one of the storage servers and issue a "stage" command on the offline files. To improve access to offline files, a select set of long term storage commands are now available on klondike, iceberg, and iceflyer. These commands do not require a kerberos ticket, so they are suitable for use in pre/post processing scripts.

The commands are:

sls

SAM ls, which can show whether or not a file in $ARCHIVE is offline.

    iceberg1 1% sls -2 /archive/u1/uaf/username/myfile.tar.gz 
    -rwx------   1 username mygroup  773486674 Jun 15  2005 /archive/u1/uaf/username/myfile.tar.gz
    O--------  guv-- -- --  sg sf       
    
    iceberg1 2% sls -D /archive/u1/uaf/username/myfile.tar.gz  
    /archive/u1/uaf/username/myfile.tar.gz:
      mode: -rwx------  links:   1  owner: username  group: mygroup
      length: 773486674  admin id:      0  inode:   5070982.1
      offline;  archdone;
      copy 1: ---- Jan 15 06:52     1bc44.3615d sg 501092
      copy 2: ---- Jun 15  2005     66596.1    sf 450190
      access:      Oct 20 11:23  modification: Jun 15  2005
      changed:     Jun 15  2005  attributes:   Jun 15  2005
      creation:    Jun 15  2005  residence:    Feb 15 11:38
    
      checksum: gen  use  val  algo: 1
      

stage

Requests that an offline file be brought online (i.e. on to disk).

    #bring an offline file back online.  The "-w" flag makes stage wait
    #until the file is back online before returning.
  
    iceberg1 3% stage -w  $ARCHIVE/myfile
    

release

Removes an online file from disk leaving only tape copies.

    #
    iceberg1 4% release $ARCHIVE/myfile
    

batch_stage

Brings a set of files online. Files are staged in the order they are written on tape. This minimizes tape seeks and changes and typically reduces the amount of time it takes to bring a file back online.

    #stage all netcdf files in a directory tree
    iceberg1 5% batch_stage `find $ARCHIVE/somedirectory/ -name \*.nc` 
  
    #alternate version reading file names from stdin.
    iceberg1 6% find $ARCHIVE/somedirectory/ -name \*.nc | batch_stage -i
    

Additional documentation for sls, stage and release is available by adding the "--help" flag.

NOTE: these commands are valid for files in the $ARCHIVE filesystem only.

For more information on SAM commands see:

Faculty Camp 2006, July 31 - August 18

Need your programs to run faster?

Overwhelmed by the quantity of your data and frustrated searching through it?

Tired of using flat graphs and contour plots to explain your discoveries?

Want to see your data visualized in 3D?

Learn how to tackle these issues and more at ARSC Faculty Camp University of Alaska Fairbanks

The three-week Faculty Camp, beginning Monday, July 31st, encompasses the necessary survival skills for High Performance Computing (HPC). Topics include, but are not limited to, Unix basics, batch processing, scripting, parallel programming concepts and techniques, and basic familiarization with various visualization packages. More advanced topics will be covered as attendees request them.

A series of seminars and hands-on experiences, presented by ARSC staff, UAF/ARSC Joint Faculty and current users, provide participants with assistance and expertise while they focus on independent work and self-guided study for individual projects.

This year's faculty camp will again include sessions from Angelo Rossi, a former employee of the Advanced Computing Technology Center at IBM who now teaches computational chemistry at York College of the City University of New York. He will be providing IBM hardware, software, and tools lectures as well as his expertise in computational chemistry.

Those interested are invited to submit a short, 250-word description of the skills you want to develop and the project you intend to pursue. This will assist ARSC in organizing events and speakers to address the specific needs of the attendees. Abstracts should be submitted to Tom Logan (logan (at) arsc.edu).

Please submit your text in ASCII or pdf format no later than May 17.

Successful applicants will be notified by May 26. Those accepted are expected to participate full-time during the three weeks of Faculty Camp. UA researchers will be compensated at their regular salary.

A:[[ I want to "diff" two files.  The catch: one of them is on a remote 
  [[ system. Is there an alternative to "scp'ing" or "ftp'ing" the 
  [[ remote file to my local system and doing the diff there?

   #
   # Thanks to Derek Bastille:
   #
   In ksh, a simple option is to use ssh.  Here is an example with 
   test1 on my local system:

   $ cat test1
   This is line 1
   This is not line 1

   and test1 on iceberg:

   $ cat test1
   This is line 1
   This is line 2

   Comparing:

   $ (ssh iceberg /bin/cat /u1/uaf/bastille/test1 2> /dev/null) | diff - ~/test1
   2,3c2
   < This is line 2
   <
   ---
   This is not line 1


   Note: The full path is required since my local $HOME differs from
   the remote $HOME.  The "-" (i.e., stdin) can be either the first or
   second file passed to diff.


   #
   # From the _other_ editor (Don't drive like HE does!):
   #

   Since diff expects a file, we can simply cat the file we want to
   look at on the remote machine then pipe the output to diff,
   substituting "/dev/stdin" for one of the file names expected by
   diff.

   This doesn't work on AIX, IRIX, or UNICOS/mp (since they don't have
   /dev/stdin), but on a Solaris, BSD, OSX, or Linux system you're
   set...  and you'll probably find other ways to use /dev/stdin:

   $ ssh machine cat ~/test.it2 | diff /dev/stdin test.it
   1c1
   < Who
   ---



Q: I'm new to C programming  and just discovered that you can't nest
   comments.  So the following isn't legal:

   /* comment A /* comment B */  */
   
   Is there a way to comment out part of my code without getting rid of
   the existing comments or having to add an absurd number of C style
   comments?

   Say I want to comment out the second two printf statements in the
   code below without losing the existing comments.  

   #include <stdio.h>
   #include <stdlib.h>
    
   int main()
   {
       int a;
       a=100;
    
       printf("%-8s = %d\n","a",a);          /* print out a */
       printf("%-8s = %d\n","a*a",a*a);      /* print out a*a */
       printf("%-8s = %d\n","a*a*a",a*a*a);  /* print out a*a*a */
   }
    

[[ Answers, Questions, and Tips Graciously Accepted ]]

Thomas J. Baring	ARSC Web Specialist	ph: 907-450-8619
Donald Bahls	ARSC User Consultant	ph: 907-450-8674
Arctic Region Supercomputing Center
University of Alaska Fairbanks
PO Box 756020
Fairbanks AK 99775-6020

Send comments and questions to the current editors using this Contact Form.

To subscribe to the e-mail edition of the ARSC HPC Users' Newsletter, send this message to: "majordomo@arsc.edu":
    subscribe hpc_users
    end
For help with majordomo, send this:
    help
    end
Leave the "subject" line of your message blank.

The ASCII e-mail edition of the ARSC T3D/T3E/HPC Users' Newsletter is archived and available by anonymous FTP to: ftp://ftp.arsc.edu/pub/publications/newsletters/

Arctic Region Supercomputing Center
PO Box 756020, Fairbanks, AK 99775 | voice: 907-474-6935 | email:

home | search | about | support | news | science | resources