ARSC HPC Users' Newsletter 370, September 21, 2007

ParaView and ezVIZ Workshops at ARSC, Oct 11-12

ARSC is offering users and prospective users training on the following two visualization packages :

ParaView (Parallel Visualization Application) Oct 11: 10 am - 5 pm

ParaView is an open-source, multi-platform application designed to visualize data sets of size varying from small to very large. See: http://www.paraview.org/

ezVIZ Oct 12: 9 am - 12 pm

ezVIZ is an offline visualization tool meant for post-processing or covisualization of simulation outputs. See: http://daac.hpc.mil/ezviz

The instructor for both workshops is Randall Hand, a Visualization Scientist with the Unclassified Data Analysis and Assessment Center for the DoD High Performance Computing Modernization Program (HPCMP).

There is no charge for these workshops, but pre-registration is required.

Model Coupling Options

[By: Kate Hedstrom]

As someone who does ocean and sea ice modeling, I know that one often wants to couple the two models together. Like many things, there's more than one way to do it.

The code I'm most familiar with has a very tight coupling between the ocean and ice models. In fact, you could say the ice model is now part of the ocean model, since they use the same grid and time step, and the ice is simply a collection of subroutines within the ocean model.

This choice works very well - until someone goes and finds a better ice model. It also works in this case because using the same grid and time step isn't a terrible idea, but there can be situations where the time and space scales in your models don't naturally match, like say a glacier and the atmosphere. Maybe a glacier can have a time step of once per day or even longer while the natural timescales in the atmosphere are much shorter.

Dan Schaffer at NOAA once sent me a talk he'd given on model coupling choices. He points out that many choices are orthogonal and can be made independently. One of the choices is synchronous versus sequential: does the ocean run on one pool of processors at the same them the ice runs on another pool? Or do both use all the processors and take turns?

The National Center for Atmospheric Research (NCAR) has a community climate model (CCSM). Its structure is that of a coupler communicating with all the other components - land, ice, air, and ocean. Each component can be computed or read from disk for testing the individual components separately. In this case, the current version run synchronously, with the coupler knowing at startup time how many processors to give to each component. There is an effort to create a sequential version for use on IBM's Blue Gene system, in which each process has to be doing the same thing as all the rest.

There are a number of tools out there to help when coupling various geophysical models. I can't show you code, but I can give the URL's. CCSM now uses the Model Coupling Toolkit (MCT) from Argonne Labs, as does the Regional Ocean Modeling System (ROMS) when used with a wave model (SWAN). ROMS uses a variant written to look like WRF I/O calls, where WRF is the Weather & Research Forecasting Model.

http://www-unix.mcs.anl.gov/mct/

NASA has a multimillion dollar effort underway to develop a newer (better?) coupler called the Earth System Modeling Framework (ESMF).

http://www.esmf.ucar.edu/

And finally, you might take a look at the European PRogram for Integrated Earth System Modeling (PRISM).

http://www.prism.enes.org/

Advanced VIM

[By: Tom Baring]

Once you master vim, you can edit without touching a "mouse" and there's not a motion wasted.

This is not a complete list of advanced vim techniques, but it's a good start. If you don't already know vi or vim, this list, while not exhaustive, will definitely be exhaustING, and you should probably just skip the entire article.

VIM TIP #1: "Cursor movement perfection"

Challenge yourself to drop the cursor exactly where you want it with the fewest keystrokes.
  • precede any cursor movement command with a number, and it will repeat that many times (E.g.: 8W hops ahead 8 Words.)

  • f and F (find) and / and ? (regular expression search) are cursor movement commands. (E.g.: 8/vim<RET> hops ahead 8 occurrences of "vim".)

  • mA and 'A set a mark and jump back to it, where "A" is any letter. (E.g., mm sets the mark "m". To come back later, hit 'm . Syntax note: the ' in 'm indicates a literal single-quote.)

  • show line numbers ( :se nu ) and use them to move around. (Do you want to be on line 876? Just type 876G .)

VIM TIP #2: "Combine cursor movement with change/delete/pipe operations"

Once you master cursor movement, combine it with certain operations to increase your editing efficiency.
  • d followed by any cursor movement command deletes the block of text defined by the current cursor position to where ever it would go--even if the movement would be backward. (E.g., d'm deletes everything to the mark, "m".)

  • c (change) lets you change everything to where ever the cursor would go. It effectively deletes the block of text and puts you in insert-mode at the same time. (E.g., c2f>. changes everything to the second > on the current line.)

  • ! followed by a cursor movement command, and a Unix filter, pipes the block into the filter as STDIN, and replaces the block with STDOUT from the filter. (E.g.: !876Gsort<RET> sorts all lines from the current line to line 876.)

VIM TIP #3: "Use regular expressions"

Learn and use regular expression search and search/replace.

Some of the perl, awk, python, or other scripts you might write to perform batch edits can probably be eliminated. Perform them right in the editor, in ex-mode ( : ).

Vim regex's can do non-greedy matching, alternation, grouping, and replacement strings can include backrefernces. Unfortunately, most of the operators have to be escaped with a backslash.

For example, here's a command to swap the first two words on every line between 1000 and 2000:

:1000,2000s/\([^ ]*\) \([^ ]*\)\(.*\)/\2 \1\3/

Since complicated expressions are nearly impossible to type correctly without editing, don't even try to type them directly on the ex ( : ) prompt. Instead, edit them in the document, yank into a register, and execute as described below, under TIP #4. (Don't forget to remove them from the document afterwards!)

Here's a nice tutorial on vim regular expressions:

http://www.geocities.com/volontir/

VIM TIP #4: "Let VIM handle repetitive tasks"

Master the MANY ways to repeat what you've done before. VIM understands that editing tasks are often repetitive. It captures what you do, so you can do it again with minimal thinking, typing, or "mousing" around:

  • . repeats the last edit-mode command (insertion, change, deletion) (E.g., dd...j... deletes this line and the next three lines, skips a line and deletes three more. cfgANG<ESC>/ing.n.n.n.n. changes everthing up to the next "g" to "ANG", and then searches for the next five occurrences of "ing", and changes them all to "ANG".)

  • ; and , repeat the last f (forward or backward)

  • n and N repeat the last regular expression search (forward or backward)

  • @ treats the text stored in a register as a command, and executes it

  • @ and q qA (where 'A' is any alphanum) starts recording whatever you do into register A. Hitting q again stops recording. To repeat the entire sequence, execute the register with @A . (See example, below.)

  • @ and yy "Ayy yanks the current line into register A where you can execute it with @A . So, you can type a complicated search/replace command in the text of your file, yank it, execute it, undo the result if it wasn't correct, and keep editing it until it works perfectly. (Syntax note: the " in "Ayy indicates a literal double-quote.)

  • @@ executes the last executed register, again.

  • precede any of the above with a number, to perform the repeat that many times. (E.g., 5; jumps to the fifth occurance of whatever you most recently found with f . 2007@@ executes the most recently executed register again, 2007 times.)

EXAMPLE: VIM REGISTER EXECUTION

vim maps, search/replace, register execution, and custom perl scripts are all alternatives for repetitive operations.

There are good times to prefer executed recordings. For instance:

  • when you need to make a context-based decision about each possible execution
  • when you can create the sequence of steps while mucking around in the file, but just can't visualize the steps well enough to create the map or script.

For instance, say you're editing HTML, and decide to replace most (but not all) of the h2 headers with h4. You want to:

  • search for the next opening h2 tag
  • look at it, and decide whether or not to replace it
  • if you decide to replace it, replace both the opening and closing tags

To do this, start a recording (to register "a" in this example), and then perform the steps, right in the file, and stop the recording:


qa/<h2
f2r4/<\/h2>
f2r4/<h2
q

Oops! A mistake. We don't need to search twice. Let's skip the first search, but keep the last. Rerecord register "a" with this better idea:


qaf2r4/<\/h2
f2r4/<h2
q

Now we can scan the rest of the file quickly:

  • find the first opening h2 header with, /h2<RET>
  • if it needs to be replaced, execute register "a" with @a
  • this takes us to the next h2 header. Look at it, and:
    • if it needs to be replaced: execute register "a" again with @@
    • if it doesn't need to be replaced: hit n to hop to the next h2 header

Scan and process the rest of the file with a combination of @@ and n .

--

VIM or VI? With the exception of recordings, all of the above vim techniques work in standard vi, but there's no reason to use vi, as vim is available on all ARSC systems.

Quick-Tip Q & A


A:[[ i have a whole bunch of files with uppercase letters in the names.
  [[ unfortunately, i do not care for uppercase letters.  could you please
  [[ tell me how to rename the files in an automated fashion such that
  [[ there are no uppercase letters in my filenames?



  ## 
  ## Thanks to all eight (!) respondents... proving again that Great 
  ## Minds don't always think alike.
  ## 
  
  # 
  # Thanks to Ashwin Ramasubramaniam 
  # 
  ls -l * 
 awk '{printf("mv %s %s\n",$9,tolower($9))
"/bin/sh"}'
  
  
  #
  # From Bill Homer
  #
  To rename each ordinary file in the current directory with at least
  one upper case letter in its name, so that all upper case letters
  are converted to lower case, unless a file with the proposed new name
  already exists:
  
  perl -e 'foreach $f (glob("*[A-Z]*"))
           { rename($f,lc($f)) if -f $f and not -e lc($f); }'
  
  ksh -c 'for i in *[A-Z]*; do typeset -l j=$i;
          [[ -f $i ]] && [[ ! -e $j ]] && mv $i $j; done'
  
  
  # 
  # Kurt Carlson
  #
  ls *[A-Z]* 
 awk '{ lo=tolower($1); print $1,lo; }' 

  while read UP LO; do if [ ! -e $LO ]; then mv $UP $LO; fi; done
  
  The 'if' clause is only there to prevent clobbering a
  file which already exists with the lowercase name.
  Issue the 'ls *[A-Z]*' when your done to see if
  something did not get renamed for that reason.
  If you don't really care:
  
  ls *[A-Z]* 
 awk '{ cmd="mv " $1 " " tolower($1); system(cmd); }'
  
  You can detect in awk if the file exists, too,
  but that requires more syntax than I can pop
  off top-of-head without my first cup of coffee.
  
  For anything like this, I tend to initially include
  an echo before the mv to ensure I haven't made an
  unfortunate typo (especially before that first cup
  of coffee).
  
  
  # 
  # Greg Newby
  #
  The "tr" command is made just for mapping one letter onto another.
  
  To rename all files (and subdirectories) in the current directory, 
  try one of these:
  
  sh/ksh syntax:
    for i in * ; do
      echo renaming $i to `echo $i 
 tr '[A-Z]' '[a-z]'`
      mv $i `echo $i 
 tr '[A-Z]' '[a-z]'`
    done
  
  csh/tcsh syntax:
    foreach i (*)
      echo renaming $i to `echo $i 
 tr '[A-Z]' '[a-z]'`
      mv $i `echo $i 
 tr '[A-Z]' '[a-z]'`
    end
  
  This isn't smart enough to skip over files that don't have any upper
  case letters, but such a test would be nearly as intensive as just
  trying the rename anyway.  On some systems, "mv" might complain about
  renaming a file to itself.
  
  
  #
  # Brad Havel
  #
  Perl to the rescue...
  -----
  [=> perl -e "opendir(DIRECTORY,'.'); @directory=readdir(DIRECTORY); \
  > closedir(DIRECTORY); foreach \$file (@directory) { \
  > (\$newname = \$file) =~tr /A-Z/a-z/; \
  > rename(\"\$file\",\"\$newname\"); }"
  -----
  Or the same thing nicely written out for a script rather than a
  continuous command line:
  -----
  #!/usr/bin/perl
  
  opendir(DIRECTORY,".");
  @directory=readdir(DIRECTORY);
  closedir(DIRECTORY);
  
  foreach $file (@directory) {
    ($newname = $file) =~tr /A-Z/a-z/;
    rename("$file","$newname");
  }
  
  
  #
  # Jed Brown
  #
  With the perl distribution, there is a `rename' (sometimes called
  `prename') which is my favorite for complicated renaming.  The `-n'
  option shows what would happen.
  
  $ rename '$_=lc' *.out
  
  
  #
  # Orion Lawlor
  #
  Rename files to lowercase names is one of my standard scripts:
  
  #!/bin/sh
  # Usage: rename_lowercase <files...>
  # Renames files with UpperCase names to all-lowercase.
  #
  for src in "$@"
  do
          dest=`echo "$src" 
 tr 'A-Z' 'a-z'`
          if [ "$src" != "$dest" ]
          then
                  echo " $src -> $dest "
                  mv -i "$src" "$dest" 

 exit 1
          fi
  done
  
  
  #
  # Alec Bennett
  #
  For a list of files, or directory, you can do something similar to
  the following in bash:
  
  for i in $( ls ); do mv $i `echo $i 
 tr "[:upper:]" "[:lower:]"`; done
  
  This should take care of an entire directory.  One thing to watch
  for on some Linux systems is that you may need to replace $( ls )
  with $( ls --color=none ) or something similar in order to avoid
  the colors interfering with the commands.




Q: I'm grepping for "Bachelor Bitter" (both words).  However, grep
   doesn't find it when there's a newline separating the two words.
   Can grep ignore newlines, somehow? Do you have any other ideas?

[[ Answers, Questions, and Tips Graciously Accepted ]]


Current Editors:
Ed Kornkven ARSC HPC Specialist ph: 907-450-8669
Kate Hedstrom ARSC Oceanographic Specialist ph: 907-450-8678
Arctic Region Supercomputing Center
University of Alaska Fairbanks
PO Box 756020
Fairbanks AK 99775-6020
E-mail Subscriptions: Archives:
    Back issues of the ASCII e-mail edition of the ARSC T3D/T3E/HPC Users' Newsletter are available by request. Please contact the editors.
Back to Top