ARSC HPC Users' Newsletter 375, November 30, 2007

Midnight: Programming Environment Upgrades


[ By: Alec Bennett ]

Version 3.1 of the Pathscale compilers was recently installed and is now available on midnight. Additionally, The Sun Studio compilers have been updated to the 2007-06 release. Users are encouraged to test their code using the new compiler versions.

For Pathscale, if you are running the default PrgEnv, you can switch to the latest version with the following command:


  % module switch PrgEnv PrgEnv.path-3.1

Similarly, if you are using PrgEnv.sun, you can switch to the latest version of the Sun Studio Compilers using the following:


  % module switch PrgEnv.sun PrgEnv.sun-2007-06

If you intend to use the new Programming Environment by default, it would be a good idea to load the module from your ~/.profile or ~/.login to ensure that the same environment is used for your job runtime environment.

Alternatively you can load modules within your PBS scripts prior to running your executable. If you choose to load modules within your job scripts, please make sure to source the appropriate modules init file for your shell. For more details see "news modules" on midnight or read the news item online here:

    http://www.arsc.edu/support/news/systemnews/news.xml?system=midnight#midnight_modules

LaTeX Follow-up

[ By: Kate Hedstrom ]

From Martin Luethi in response to the Latex article:


> Some comments on your LaTeX praise:
>
> HPC Users Newsletter writes:
> > modernize with some whole new software or can I get away with using
> > LaTeX again?  Another reason to prefer LaTeX is that we're creating a
> > documentation wiki for the ocean model and the wiki can display LaTeX
> > equations directly, generating a png image for each. Equations I write
> > in the manual can be placed on the wiki with a simple cut and paste
> > operation and still look great.
>
> are you aware of jsMath? It renders TeX directly in any web browser.
>
> To include a graphic you don't need to use the picture environment
>
> \begin{figure}[tbhp]
> \begin{center}
>   \includegraphics[width=0.9\textwidth]{mickey.pdf}
> \end{center}
> \caption{Mickey Mouse}
> \label{fig:mickey}
> \end{figure}
> 
> Using Emacs as editor (with Auctex, reftex, cdlatex modes) makes
> writing latex, especially equations, references, indices a breeze.
>  
> One of the best references is the "not so short introduction" that
> comes with almost all Latex distributions.
> 
http://www.ctan.org/tex-archive/info/lshort/english/lshort.pdf

>  
> Best, Martin

Thanks much for the useful comments!

Martin is right that I did not need the picture environment for the example shown. It was there from my original document in which I inserted text over the included graphics with the full power of LaTeX ( $\xi$ , for instance, to get a Greek letter, xi ). I was making such mixed-mode figures with xfig, a vector drawing program that can export to LaTeX picture objects.

The not-so-short guide recommends these lines for seeing the pdf preview more clearly:


    \usepackage{lmodern}
    \usepackage[T1]{fontenc}
    \usepackage{textcomp}

The guide also talks about turning your internal references into pdf hyperlinks with:


    \usepackage[pdftex]{hyperref}

Also recommended is saving your images as both eps and pdf, then including them without an extension:


    \begin{figure}[tbhp]
      \begin{center}
        \includegraphics[width=0.9\textwidth]{mickey}
      \end{center}
      \caption{Mickey Mouse}
      \label{fig:mickey}
    \end{figure}

This allows the same source to work with both the dvi and the pdf methods (as long as they are all vector graphics).

Another LaTeX resource is "Guide to LaTeX" (fourth edition) by Helmut Kopka and Patrick W. Daly, ISBN 0-321-17385-6.

Kopka and Daly point out that one GUI option for writing LaTeX is Lyx ( www.lyx.org ) This should be an easy way to whip up simple documents, protecting you from having to remember all the header commands, etc. I plan to give it a whirl and perhaps report back here.

They also reminded me that part of the overhaul of LaTeX2e is a New Font Selection Scheme (NFSS). This allows the use of stroked fonts to replace the older bitmapped fonts from the original TeX distribution. Images can also be bitmapped or stroked (vector), which leads into some topics for another time - what little I know about graphics formats and free graphics programs.

LyX

[ By: Kate Hedstrom ]

LyX is a graphical user interface sitting on top of the LaTeX document preparation system and is available for free from www.lyx.org . LyX allows you to create an entire document without having to know the details of the underlying markup language. The intro documentation is written for people without any prior LaTeX background, emphasizing that a markup language separates style from content. Those with knowledge of html will know this already - content is an <h1> header and style comes from a stylesheet elsewhere, specifying the font and size of all <h1> headers. (Those of us from a LaTeX background boggle at the thought that something as mature and widely used as Word wouldn't behave this way.)

LyX comes with a very nice tutorial which leads you through some of the commonly used tasks such as creating section headings, links referring to other parts of the document, and equations. Equations can be created entirely from mouse clicks on the various symbols, hunting in the many symbol menus, or you can type them in yourself if you know the sequence. I found that extra spaces would dump me into the text following an equation, something I wasn't expecting.

It also takes care of those special characters like '&' and '"', doing the necessary transformations silently, behind the scenes. They note that you can set up your own templates, for instance, letters with your address up in the corner so that you don't need to type it in each time.

LyX creates documents with a .lyx extension, containing header information specific to LyX - you are expected to process the file from within the GUI, which will generate the dvi, pdf, or ps file for you from an export menu - with three pdf options! It can also export to plain or pdflatex for you, or even plain text. It can import documents in formats including LaTeX, text and even Word, though I haven't tried that yet.

My nits

  1. Many GUI "textbox" tools let you select a sentence or two and hit delete to remove the whole selection. This didn't work for me in my first attempts at LyX. I later tried again and discovered that I had been too ponderous in my selections. If you do it right, the highlighted text stays blue and you can delete the lot - or just hit delete before lifting the mouse button.
  2. Typing into the various parts of an equation takes a lot of precision - are you in the numerator? the equation next to the fraction? or after the equation entirely?

Overall Impression

I think LyX is well worth looking into for someone wanting to try LaTeX with immediate feedback on the look of the document, answering the most common complaint I've heard about LaTeX.

That said, I'm not going to be converting my current project to LyX. By the time I'll be done with the next round of changes, it will be 20 years old, in dozens of files. Call me old-fashioned, I don't care. For new projects, it's an easy way to whip out a letter or other document and I will be using it.

Nelchina and Iceflyer Retirement Reminder

This is a reminder to active users on Nelchina and Iceflyer that the retirement of those systems is coming soon.

Iceflyer Retirement

Iceflyer, the ARSC IBM Power4/Power5 system available to University users, will be taken out of production on December 28th, 2007. After December 28th, Iceflyer will only be available only to those who have made prior arrangements for special projects or to complete application migration. Some time between January 2008 and March 2008, Iceflyer will be powered down and retired.

As part of the retirement of Iceflyer, data on local filesystems will be removed. Local filesystem include $WORKDIR, $DATADIR, $SCRATCH and $HOME. Should you wish to keep data stored in these locations, please copy the data to $ARCHIVE before the retirement. If you need access to files on $ARCHIVE for use on Iceberg or Midnight, please contact the ARSC Help Desk for additional information.

We encourage you to begin moving your code to other ARSC resources such as Midnight. ARSC User consultants and specialists are available should you need assistance.

Again, unless you have made prior arrangements, accounts on Iceflyer will be placed in inactive state on the morning of December 28th.

Nelchina Retirement

Nelchina, the ARSC Cray XD1 system, will be retired on December 28th, 2007. We encourage you to begin moving your code to other ARSC resources such as Midnight or the Linux Workstations as soon as possible. ARSC User consultants and specialists are available should you need assistance.

For more information on available ARSC systems, see http://www.arsc.edu/resources/resources.html

As part of the retirement of Nelchina, data on local filesystems will be removed. Local filesystem include $WORKDIR, $DATADIR, $SCRATCH and $HOME. Should you wish to keep data stored in these locations, please copy the data to $ARCHIVE before December 28th. Files copied to $ARCHIVE on Nelchina will be accessible on Midnight.

Again, all accounts on Nelchina will be inactivated on the morning of December 28th.

If you need assistance moving your work from Nelchina or Iceflyer to another ARSC resource, please feel free to contact the ARSC Help Desk.

Quick-Tip Q & A


A:[[ I have a file which contains a long list of file names, one per line.
  [[ Every file name takes the format of a time-stamp followed by a name,
  [[ like this: YYYYMMDD.HHMM.<name>, and every file name is prefixed by
  [[ a full Unix path.  Here's a sampling of the file:
  [[  
  [[  /wrkdir/gizmo/STREAM_BENCHMARK/20071010.2322.stdout.txt
  [[  /archive/u1/uaf/gizmo/STREAM_BENCHMARK/CRAY-T3D/19940911.1023.stdout.txt
  [[  /archive/u1/uaf/gizmo/STREAM_BENCHMARK/CRAY-T3D/19940911.1023.stdout.txt
  [[  /u1/uaf/cranky/progs/nas/20050606.0730.run1.out
  [[  /u1/uaf/cranky/progs/nas/20050606.0838.run2.out
  [[  /scratch/stuff/20071106.1803.run1.out
  [[  
  [[ I need to sort the lines in this file, using the time-stamp as the
  [[ sort key.  
  [[ 
  [[ The character position of the time-stamp varies from line to line,
  [[ as does the number of forward slash delimiters.  Is there a way to
  [[ do this with the Unix "sort" command, or am I stuck writing a script?


  # 
  # Something for everybody this week: 
  #
  #   awk, perl
sed
sort
tr, Haskell, python, shell script, and in case 
  #   you're still bored, a "Schwartzian transform."
  #


  # 
  # Robert Linzell
  # 
  This works with the above example (assuming the file containing the
  list is named tst.txt):

  awk 'BEGIN {FS="/"};{print $NF, $0}' tst.txt 
 sort -k 1,1 
 awk '{print $2}'


  #
  # Thanks to Bill Homer
  #
  Pick a character, such as '@', not contained in the file names,
  use sed to split the file names into the relevant fields with that
  character as field separator, then pipe into sort, and finally into
  tr to eliminate the field separator.  Assume each line contains
  only one instance of a timestamp, and that duplicate lines should
  be eliminated.

    sed -r -e 's/([0-9]{8}[.][0-9]{4})/@&@/' file \
      
 sort -t@ -k2,2 -k1,1 -k3,3 -u 
 tr -d '@'

  If the sed on your system doesn't produce the expected result from
  the first line, it can be expressed in perl like this:

    perl -nle 'print join( "@", split( /(\d{8}[.]\d{4})/ ))'



  # 
  # Dan Stahlke
  # 
  The problem can be solved using a Schwartzian Transform in perl.  Just
  pipe the input through a perl script like this:

  cat file_list 
   perl -e 'print map {$_->[0]} sort {$a->[1] cmp $b-
  >[1]} map {($a)=/(\d{8}\.\d{4})/ or die "cannot parse line $_\n"; [$_,
  $a]} <>'

  Or save the above script into a file:
        print map {
                $_->[0]
        } sort {
                $a->[1] cmp $b->[1]
        } map {
                ($a) = /(\d{8}\.\d{4})/ or die "cannot parse line $_\n";
                [$_, $a]
        } <>

  And then run:
        cat file_list 
 perl script.pl


  # 
  # Martin Luthi
  # 
  Using Python this is easy:

  import datetime as DT
  fn = dict([(DT.datetime.strptime(line.split('/')[-1][:13],'%Y%m%d.%H%M'), line)
           for line in file('my-dates.dat')]))
  for k in sorted(fn.keys()):
    print fn[k]

  This works with Python 2.5, and 2.4. The same code written as for loop
  (instead of a list comprehension), and working on Python > 2.0:

  fn = {}                              # create a dict (dictionary or hashtable)
  for line in file('my-dates.dat'):    # loop over the file
                                       # parse the date after truncating the filename 
    date = datetime.datetime.strptime(line.split('/')[-1][:13],'%Y%m%d.%H%M')
      fn[date] = line                  # add to dict
  dates = fn.keys()                    # get the keys
  dates.sort()                         # sort them
  for d in dates:                      # loop over sorted keys
    print fn[d]


  # 
  # Greg Newby
  # 
  I don't think you can do it with 'sort' alone, since the field
  you're sorting on is not the same from line to line.  Here's a
  simple alternative.

  A shell script based on 'basename' should work.  The basename
  command is provided by GNU as free software, among other sources,
  and is installed on most Unix/Linux/BSD systems (including ARSC's
  OS X systems).

  The basename command is the opposite of 'dirname,' in case you
  ever want the directory portion rather than the file portion.  In a
  nutshell, basename returns the last portion of a directory/file path,
  the part after the final slash.

  So, for your example, let's say I have all the file names in a file:

  % cat file1.txt
    /wrkdir/gizmo/STREAM_BENCHMARK/20071010.2322.stdout.txt
    /archive/u1/uaf/gizmo/STREAM_BENCHMARK/CRAY-T3D/19940911.1023.stdout.txt
    /archive/u1/uaf/gizmo/STREAM_BENCHMARK/CRAY-T3D/19940911.1023.stdout.txt
    /u1/uaf/cranky/progs/nas/20050606.0730.run1.out
    /u1/uaf/cranky/progs/nas/20050606.0838.run2.out
    /scratch/stuff/20071106.1803.run1.out

  Since the sorting can (presumably) be based on just the numeric +
  period portions of the filename, and you're using dates that sort
  the same whether numerically or chronologically, try this:

    rm -f file2.txt
    for i in `cat file1.txt` ; do
    basename `echo $i` >> file2.txt
    done
    sort -n file2.txt

  Yielding:
    19940911.1023.stdout.txt
    19940911.1023.stdout.txt
    20050606.0730.run1.out
    20050606.0838.run2.out
    20071010.2322.stdout.txt
    20071106.1803.run1.out

  I'm using a temporary file (file2.txt) for simplicity and re-use.
  The rest, as they say, is features.  Here's a variation that keeps
  the full pathname, by saving the basename output next to the orginal
  path ($i in my example).  'sort' works the same, but I'll just grab
  the second field with 'cut':

    for i in `cat file1.txt` ; do
    echo `basename $i` $i >> file2.txt
    done
    sort -n file2.txt 
 cut -f2 -d' '

  Yielding:
    /archive/u1/uaf/gizmo/STREAM_BENCHMARK/CRAY-T3D/19940911.1023.stdout.txt
    /archive/u1/uaf/gizmo/STREAM_BENCHMARK/CRAY-T3D/19940911.1023.stdout.txt
    /u1/uaf/cranky/progs/nas/20050606.0730.run1.out
    /u1/uaf/cranky/progs/nas/20050606.0838.run2.out
    /wrkdir/gizmo/STREAM_BENCHMARK/20071010.2322.stdout.txt
    /scratch/stuff/20071106.1803.run1.out

  The -n for sorting does smallest-to-largest.  Omit that if you want
  the more recent dates sorted first.  I've just provided Borne shell
  syntax, but C shell is similar other than the loop construction.


  #
  # Jed Brown
  #
  The obligatory Haskell solution:

    $ ghc -e 'interact $ unlines . sortBy (Data.Ord.comparing 
        System.FilePath.takeBaseName) .  lines' < filename > sortedfile



Q:  The IBM compilers have some nice options for tracking down floating
    exceptions.  I took a look at the man page for both gcc and PathScale
    and don't see anything similar.  Is there a easy way to trap floating
    point exceptions with compilers on a Linux system?  Basically I want
    the code to produce a core dump as soon as a floating point exception 
    occurs.

[[ Answers, Questions, and Tips Graciously Accepted ]]


Current Editors:
Ed Kornkven ARSC HPC Specialist ph: 907-450-8669
Kate Hedstrom ARSC Oceanographic Specialist ph: 907-450-8678
Arctic Region Supercomputing Center
University of Alaska Fairbanks
PO Box 756020
Fairbanks AK 99775-6020
E-mail Subscriptions: Archives:
    Back issues of the ASCII e-mail edition of the ARSC T3D/T3E/HPC Users' Newsletter are available by request. Please contact the editors.
Back to Top