[Menu Bar] Resourses at ARSC Science at ARSC Newsroom Support About ARSC ARSC Home

 

ARSC HPC Users' Newsletter 370, September 21, 2007

Newsletter Index Quick-Tip Index Search Newsletters

Contents

 

ParaView and ezVIZ Workshops at ARSC, Oct 11-12

ARSC is offering users and prospective users training on the following two visualization packages:

ParaView (Parallel Visualization Application)
Oct 11: 10 am - 5 pm

ParaView is an open-source, multi-platform application designed to visualize data sets of size varying from small to very large. See: http://www.paraview.org/

ezVIZ
Oct 12: 9 am - 12 pm

ezVIZ is an offline visualization tool meant for post-processing or covisualization of simulation outputs. See: http://daac.hpc.mil/ezviz

The instructor for both workshops is Randall Hand, a Visualization Scientist with the Unclassified Data Analysis and Assessment Center for the DoD High Performance Computing Modernization Program (HPCMP).

There is no charge for these workshops, but pre-registration is required.

 

Model Coupling Options

[By: Kate Hedstrom]

As someone who does ocean and sea ice modeling, I know that one often wants to couple the two models together. Like many things, there's more than one way to do it.

The code I'm most familiar with has a very tight coupling between the ocean and ice models. In fact, you could say the ice model is now part of the ocean model, since they use the same grid and time step, and the ice is simply a collection of subroutines within the ocean model.

This choice works very well - until someone goes and finds a better ice model. It also works in this case because using the same grid and time step isn't a terrible idea, but there can be situations where the time and space scales in your models don't naturally match, like say a glacier and the atmosphere. Maybe a glacier can have a time step of once per day or even longer while the natural timescales in the atmosphere are much shorter.

Dan Schaffer at NOAA once sent me a talk he'd given on model coupling choices. He points out that many choices are orthogonal and can be made independently. One of the choices is synchronous versus sequential: does the ocean run on one pool of processors at the same them the ice runs on another pool? Or do both use all the processors and take turns?

The National Center for Atmospheric Research (NCAR) has a community climate model (CCSM). Its structure is that of a coupler communicating with all the other components - land, ice, air, and ocean. Each component can be computed or read from disk for testing the individual components separately. In this case, the current version run synchronously, with the coupler knowing at startup time how many processors to give to each component. There is an effort to create a sequential version for use on IBM's Blue Gene system, in which each process has to be doing the same thing as all the rest.

There are a number of tools out there to help when coupling various geophysical models. I can't show you code, but I can give the URL's. CCSM now uses the Model Coupling Toolkit (MCT) from Argonne Labs, as does the Regional Ocean Modeling System (ROMS) when used with a wave model (SWAN). ROMS uses a variant written to look like WRF I/O calls, where WRF is the Weather & Research Forecasting Model.

http://www-unix.mcs.anl.gov/mct/

NASA has a multimillion dollar effort underway to develop a newer (better?) coupler called the Earth System Modeling Framework (ESMF).

http://www.esmf.ucar.edu/

And finally, you might take a look at the European PRogram for Integrated Earth System Modeling (PRISM).

http://www.prism.enes.org/

 

Advanced VIM

[By: Tom Baring]

Once you master vim, you can edit without touching a "mouse" and there's not a motion wasted.

This is not a complete list of advanced vim techniques, but it's a good start. If you don't already know vi or vim, this list, while not exhaustive, will definitely be exhaustING, and you should probably just skip the entire article.

VIM TIP #1: "Cursor movement perfection"

Challenge yourself to drop the cursor exactly where you want it with the fewest keystrokes.

VIM TIP #2: "Combine cursor movement with change/delete/pipe operations"

Once you master cursor movement, combine it with certain operations to increase your editing efficiency.

VIM TIP #3: "Use regular expressions"

Learn and use regular expression search and search/replace.

Some of the perl, awk, python, or other scripts you might write to perform batch edits can probably be eliminated. Perform them right in the editor, in ex-mode (:).

Vim regex's can do non-greedy matching, alternation, grouping, and replacement strings can include backrefernces. Unfortunately, most of the operators have to be escaped with a backslash.

For example, here's a command to swap the first two words on every line between 1000 and 2000:

:1000,2000s/\([^ ]*\) \([^ ]*\)\(.*\)/\2 \1\3/

Since complicated expressions are nearly impossible to type correctly without editing, don't even try to type them directly on the ex (:) prompt. Instead, edit them in the document, yank into a register, and execute as described below, under TIP #4. (Don't forget to remove them from the document afterwards!)

Here's a nice tutorial on vim regular expressions:
http://www.geocities.com/volontir/

VIM TIP #4: "Let VIM handle repetitive tasks"

Master the MANY ways to repeat what you've done before. VIM understands that editing tasks are often repetitive. It captures what you do, so you can do it again with minimal thinking, typing, or "mousing" around:

EXAMPLE: VIM REGISTER EXECUTION

vim maps, search/replace, register execution, and custom perl scripts are all alternatives for repetitive operations.

There are good times to prefer executed recordings. For instance:

For instance, say you're editing HTML, and decide to replace most (but not all) of the h2 headers with h4. You want to:

To do this, start a recording (to register "a" in this example), and then perform the steps, right in the file, and stop the recording: 

qa/<h2
f2r4/<\/h2>
f2r4/<h2
q

Oops! A mistake. We don't need to search twice. Let's skip the first search, but keep the last. Rerecord register "a" with this better idea: 

qaf2r4/<\/h2
f2r4/<h2
q

Now we can scan the rest of the file quickly:

Scan and process the rest of the file with a combination of @@ and n.

--

VIM or VI? With the exception of recordings, all of the above vim techniques work in standard vi, but there's no reason to use vi, as vim is available on all ARSC systems.

 

Quick-Tip Q & A

 
A:[[ i have a whole bunch of files with uppercase letters in the names.
  [[ unfortunately, i do not care for uppercase letters.  could you please
  [[ tell me how to rename the files in an automated fashion such that
  [[ there are no uppercase letters in my filenames?


  ## 
  ## Thanks to all eight (!) respondents... proving again that Great 
  ## Minds don't always think alike.
  ## 
  
  # 
  # Thanks to Ashwin Ramasubramaniam 
  # 
  ls -l * | awk '{printf("mv %s %s\n",$9,tolower($9))|"/bin/sh"}'
  
  
  #
  # From Bill Homer
  #
  To rename each ordinary file in the current directory with at least
  one upper case letter in its name, so that all upper case letters
  are converted to lower case, unless a file with the proposed new name
  already exists:
  
  perl -e 'foreach $f (glob("*[A-Z]*"))
           { rename($f,lc($f)) if -f $f and not -e lc($f); }'
  
  ksh -c 'for i in *[A-Z]*; do typeset -l j=$i;
          [[ -f $i ]] && [[ ! -e $j ]] && mv $i $j; done'
  
  
  # 
  # Kurt Carlson
  #
  ls *[A-Z]* | awk '{ lo=tolower($1); print $1,lo; }' |
  while read UP LO; do if [ ! -e $LO ]; then mv $UP $LO; fi; done
  
  The 'if' clause is only there to prevent clobbering a
  file which already exists with the lowercase name.
  Issue the 'ls *[A-Z]*' when your done to see if
  something did not get renamed for that reason.
  If you don't really care:
  
  ls *[A-Z]* | awk '{ cmd="mv " $1 " " tolower($1); system(cmd); }'
  
  You can detect in awk if the file exists, too,
  but that requires more syntax than I can pop
  off top-of-head without my first cup of coffee.
  
  For anything like this, I tend to initially include
  an echo before the mv to ensure I haven't made an
  unfortunate typo (especially before that first cup
  of coffee).
  
  
  # 
  # Greg Newby
  #
  The "tr" command is made just for mapping one letter onto another.
  
  To rename all files (and subdirectories) in the current directory, 
  try one of these:
  
  sh/ksh syntax:
    for i in * ; do
      echo renaming $i to `echo $i | tr '[A-Z]' '[a-z]'`
      mv $i `echo $i | tr '[A-Z]' '[a-z]'`
    done
  
  csh/tcsh syntax:
    foreach i (*)
      echo renaming $i to `echo $i | tr '[A-Z]' '[a-z]'`
      mv $i `echo $i | tr '[A-Z]' '[a-z]'`
    end
  
  This isn't smart enough to skip over files that don't have any upper
  case letters, but such a test would be nearly as intensive as just
  trying the rename anyway.  On some systems, "mv" might complain about
  renaming a file to itself.
  
  
  #
  # Brad Havel
  #
  Perl to the rescue...
  -----
  [=> perl -e "opendir(DIRECTORY,'.'); @directory=readdir(DIRECTORY); \
  > closedir(DIRECTORY); foreach \$file (@directory) { \
  > (\$newname = \$file) =~tr /A-Z/a-z/; \
  > rename(\"\$file\",\"\$newname\"); }"
  -----
  Or the same thing nicely written out for a script rather than a
  continuous command line:
  -----
  #!/usr/bin/perl
  
  opendir(DIRECTORY,".");
  @directory=readdir(DIRECTORY);
  closedir(DIRECTORY);
  
  foreach $file (@directory) {
    ($newname = $file) =~tr /A-Z/a-z/;
    rename("$file","$newname");
  }
  
  
  #
  # Jed Brown
  #
  With the perl distribution, there is a `rename' (sometimes called
  `prename') which is my favorite for complicated renaming.  The `-n'
  option shows what would happen.
  
  $ rename '$_=lc' *.out
  
  
  #
  # Orion Lawlor
  #
  Rename files to lowercase names is one of my standard scripts:
  
  #!/bin/sh
  # Usage: rename_lowercase <files...>
  # Renames files with UpperCase names to all-lowercase.
  #
  for src in "$@"
  do
          dest=`echo "$src" | tr 'A-Z' 'a-z'`
          if [ "$src" != "$dest" ]
          then
                  echo " $src -> $dest "
                  mv -i "$src" "$dest" || exit 1
          fi
  done
  
  
  #
  # Alec Bennett
  #
  For a list of files, or directory, you can do something similar to
  the following in bash:
  
  for i in $( ls ); do mv $i `echo $i | tr "[:upper:]" "[:lower:]"`; done
  
  This should take care of an entire directory.  One thing to watch
  for on some Linux systems is that you may need to replace $( ls )
  with $( ls --color=none ) or something similar in order to avoid
  the colors interfering with the commands.



Q: I'm grepping for "Bachelor Bitter" (both words).  However, grep
   doesn't find it when there's a newline separating the two words.
   Can grep ignore newlines, somehow? Do you have any other ideas?

[[ Answers, Questions, and Tips Graciously Accepted ]]

 

 

Current Editors:
Thomas J. Baring ARSC Web Specialist ph: 907-450-8619
Donald Bahls ARSC User Consultant ph: 907-450-8674
Arctic Region Supercomputing Center
University of Alaska Fairbanks
PO Box 756020
Fairbanks AK 99775-6020
Contact:
Send comments and questions to the current editors using this Contact Form.
Email Subscriptions: Archives:

 

Newsletter Index Quick-Tip Index Search Newsletters

 

Arctic Region Supercomputing Center
PO Box 756020, Fairbanks, AK 99775 | voice: 907-474-6935 | email:

home | search | about | support | news | science | resources