![[Menu Bar]](/images/header.gif)
ARSC system news for all systems
Menu to filter items by type
| Type | Downtime | News | |||
| Machine | All Systems | linuxws | midnight | pingo | seawolf |
Contents for all systems
News Items
"PET_HOME"
Last Updated: Wed, 20 Jan 2010 - Alec BennettMachines: pingo
PET_HOME ======== The PET-Computational Environment (CE) consists of a number of packages and libraries used in High Performance Computing. These tools are available in the $PET_HOME directory on Pingo. Current Versions in $PET_HOME ============================= Package Name Version CMake 2.6.4 FFTW-2 (sequential) 2.1.5 FFTW-3 (distributed) 3.1.3 GotoBLAS 1.26 GSL 1.12 HDF5 1.8.3 MatlabMPI 1.2 netCDF 4.0.1 Octave 3.0.5 PAPI 3.6.2 PETSc 2.3.3 ScaLAPACK 1.8.0 Scalasca 1.1 SPRNG 3.0 SuperLU 3.0 TAU 2.18.1 Trilinos 9.0.3 Valgrind 3.4.0
"PET_HOME"
Last Updated: Wed, 20 Jan 2010 - Alec BennettMachines: midnight
PET-Computational Environment (CE) ================================== The PET-Computational Environment (CE) consists of a number of packages and libraries used in High Performance Computing. These tools are available in the $PET_HOME directory on Midnight. Current Versions in $PET_HOME ============================= Package Name Version ARPACK 96 CMake 2.6.4 FFTW-2 (sequential) 2.1.5 FFTW-3 (distributed) 3.1.3 GotoBLAS 1.26 GSL 1.12 HDF5 1.8.3 MatlabMPI 1.2 netCDF 4.0.1 Octave 3.0.5 PAPI 3.6.2 PETSc 2.3.3 ScaLAPACK 1.8.0 Scalasca 1.1 SPRNG 3.0 SuperLU 3.0 TAU 2.18.1 Trilinos 9.0.3 Valgrind 3.4.0
"PrgEnv"
Last Updated: Wed, 22 Oct 2008 - Donald BahlsMachines: midnight
Programming Environments on midnight
====================================
Compiler and MPI Library versions on midnight are controlled via
the modules package. New accounts load the "PrgEnv" module by
default. This module adds the Pathscale compilers and Voltaire
MPI compilers to the PATH.
Should you experience problems with a compiler or library in
many cases a new programming environment may be available.
Below is a description of available Programming Environments:
Module Name Description
=============== ==============================================
PrgEnv Programming environment using Pathscale
compilers and MPI stack (default version).
PrgEnv.new programming environment using the latest available
version of the Pathscale compilers.
PrgEnv.path Alternate name for PrgEnv
PrgEnv.path.new Alternate name for PrgEnv.new
PrgEnv.gcc Programming environment using GNU compilers
and MPI stack.
PrgEnv.pgi Programming environment using PGI compilers
and MPI stack.
PrgEnv.sun Programming environment using Sun Studio compilers
and MPI stack.
For a list of the latest available Programming Environments, run:
mg56 % module avail PrgEnv
--------------------- /usr/local/pkg/modulefiles ----------------
PrgEnv PrgEnv.new PrgEnv.pgi
PrgEnv.gcc PrgEnv.path PrgEnv.sun
PrgEnv.gcc.new PrgEnv.path.new PrgEnv.sun.new
ARSC Policy for Maintaining Programming Environment Module Versions
===================================================================
ARSC maintains a "current" programming environment, the last
programming environment and the most recently released version using
the "modules" facility. These policies explain the naming scheme
for the programming environment modules as well as the schedule for
changing the versions referenced by those modules.
- These policies pertain to the PathScale, PGI, and SunStudio compilers.
PathScale is our "official" compiler, hence it is pointed to by the
special names PrgEnv, PrgEnv.old and PrgEnv.new reflecting the
"current", "old" and "new" versions respectively. These are aliases for:
PrgEnv == PrgEnv.path
PrgEnv.old == PrgEnv.path.old
PrgEnv.new == PrgEnv.path.new
There will be "current", "old" and "new" versions for the other
compilers as well
PrgEnv.pgi, PrgEnv.pgi.old, PrgEnv.pgi.new
PrgEnv.sun, PrgEnv.sun.old, PrgEnv.sun.new
and in the discussion that follows, it is understood that policies that
apply to "PrgEnv", "PrgEnv.old" and "PrgEnv.new" apply to all three
compilers.
- The above module names are aliases for modules named by their version,
e.g., PrgEnv.path-3.0.
- When versions newer than "PrgEnv.new" are made available, we will
assign them a version-specific module name and notify users of their
availability. We will install new Programming Environment versions as
soon as practical, balanced against the benefits of the new version as
judged by ARSC staff.
- Under normal circumstances, i.e., no "extreme" need to act sooner, we
will rotate "PrgEnv", "PrgEnv.old" and "PrgEnv.new" no more frequently
than twice annually, on or about October 1 and April 1. This rotation
will be done only if "PrgEnv.new" is actually a newer version than
"PrgEnv" and only if "PrgEnv.new" has been available for testing for at
least one month. The rotation keeps the old version and updates the
current version as follows:
"PrgEnv.old" <-- "PrgEnv"
"PrgEnv" <-- "PrgEnv.new"
"PrgEnv.new" <-- a newer version (if it exists)
- Under extreme circumstances (e.g., a version that we cannot tolerate
being used or a significant system upgrade), ARSC staff may elect to
rotate these versions sooner or remove a version entirely.
- Insofar as it is feasible, we will maintain old versions (older than
"PrgEnv.old") as long as possible and will always maintain at least the
"PrgEnv.old" version.
- Any changes to "PrgEnv.old", "PrgEnv" and "PrgEnv.new" will be
preceded by at least two weeks notice to users.
Programming Environment Changes
================================
The following is a table of recent additions and changes to the
Programming Environment on midnight.
Date Module Name Description
---------- --------------------- -----------------------------------
2009-10-21 PrgEnv.path Fall updates to Programming
PrgEnv.path.old Environment.
PrgEnv.pgi.new
PrgEnv.pgi.old
PrgEnv.pgi
PrgEnv.pgi.new
2008-10-23 PrgEnv.path.new Fall updates to Programming
PrgEnv.path Environment.
PrgEnv.path.old
PrgEnv.pgi.old
PrgEnv.pgi
PrgEnv.pgi.new
2008-05-05 PrgEnv.pgi.new Latest release of the PGI compiler
PrgEnv.pgi-7.1.6 Suite.
2007-12-05 PrgEnv.pgi New Programming Environment using
PrgEnv.pgi-7.0.2 PGI Compiler Suite
2007-11-19 PrgEnv.path-3.1 New Programming Environment using
PathScale 3.1 compilers.
2007-09-10 PrgEnv.sun-2007-06 New Programming Environment using
Sun Studio 2007-06 compilers.
2009-10-21 Updates
===================
During system maintenance on October 21, 2009 the PathScale and
Portland Group Programming Environments will be updated as follows
Module Name Now Aliased To Previously Aliased To
--------------- ------------------- ----------------------
PrgEnv.path.old PrgEnv.path-3.1 PrgEnv.path-3.0
PrgEnv.path PrgEnv.path-3.2 PrgEnv.path-3.1
PrgEnv.pgi.old PrgEnv.pgi-7.1.6 PrgEnv.pgi-7.0.2
PrgEnv.pgi PrgEnv.pgi-7.2.2 PrgEnv.pgi-7.1.6
PrgEnv.pgi.new PrgEnv.pgi-8.0.6 PrgEnv.pgi-7.2.2
2008-10-23 Updates
===================
During system maintenance October 22, the PathScale and Portland Group
Programming Environments were updated as follows:
Module Name Now Aliased To Previously Aliased To
--------------- ------------------- ----------------------
PrgEnv.old PrgEnv.path.old PrgEnv.path-2.5
PrgEnv.path.old PrgEnv.path-3.0 PrgEnv.path-2.5
PrgEnv PrgEnv.path PrgEnv.path-3.0
PrgEnv.path PrgEnv.path-3.1 PrgEnv.path-3.0
PrgEnv.new PrgEnv.path.new PrgEnv.path-3.1
PrgEnv.path.new PrgEnv.path-3.2 PrgEnv.path-3.1
PrgEnv.pgi.old PrgEnv.pgi-7.0.2 nothing -- new module
PrgEnv.pgi PrgEnv.pgi-7.1.6 PrgEnv.pgi-7.0.2
PrgEnv.pgi.new PrgEnv.pgi-7.2.2 PrgEnv.pgi-7.1.6
"PrgEnv"
Last Updated: Wed, 22 Oct 2008 - Donald BahlsMachines: pingo
Programming Environments on pingo
========================================
Compilers on pingo are controlled via the modules package. All
accounts load the "PrgEnv-pgi" module by default. This module
adds the PGI compilers with MPI, OpenMP and SHMEM support into
the PATH.
Should you experience problems with a compiler or library a newer
version may be available.
Below is a description of available Programming Environments:
Module Name Description
================= ==============================================
PrgEnv-pgi Programming environment using the PGI
compilers and MPI stack (default module).
PrgEnv-pathscale Programming environment using the PathScale
compilers and MPI stack
PrgEnv-gcc Programming environment using the GNU compilers
and MPI stack.
For a list of the latest available Programming Environments, run:
pingo1 % module avail PrgEnv
---------------------- /opt/modulefiles --------------------------
PrgEnv-gnu/2.1.24HD PrgEnv-pathscale/2.1.24HD
PrgEnv-pgi/2.1.24HD PrgEnv-gnu/2.1.27HD(default)
PrgEnv-pathscale/2.1.27HD(default) PrgEnv-pgi/2.1.27HD(default)
Available Compute Node Compilers
=================================
Several compilers are available on pingo. The following compilers can
be used to compile MPI, OpenMP, serial and SHMEM applications for use
on compute nodes.
Module C C++ Fortran Notes
================= ====== ====== ======== ============================
PrgEnv-pgi cc CC ftn Supports MPI, OpenMP, Serial
PrgEnv-pathscale cc CC ftn Supports MPI, OpenMP, Serial
PrgEnv-gnu cc CC ftn Supports MPI, OpenMP, Serial
Pre and Post processing applications which will be run on login nodes
must be compile with one of the following compilers.
Module C C++ Fortran Notes
================= ====== ====== ======== ============================
PrgEnv-pgi pgcc pgCC pgf90 Supports Serial
PrgEnv-pathscale pathcc pathCC pathf90 Supports Serial
PrgEnv-gnu gcc g++ gfortran Supports Serial
Programming Environment Changes
================================
The following is a table of recent additions and changes to the
Programming Environment on pingo.
Date Module Name Description
---------- --------------------- -----------------------------------
"archive"
Last Updated: Tue, 28 Aug 2008 - Oralee NudsonMachines: linuxws seawolf
Pending $ARCHIVE Changes
========================================
During the first week of January 2009 all data currently in $ARCHIVE on
nanook.arsc.edu will be moved to seawolf.arsc.edu in order to
consolidate the ARSC long term storage infrastructure.
There is a possibility that identically
named files and/or directories could exist on both long term storage
systems. In cases where identically named files and/or directories
are present, we are asking each user to determine which copy they wish
to keep and to remove the copy they do not need. If there are naming
conflicts in the user's $ARCHIVE directory that are unresolved when
data is moved in January, data with naming conflicts will be moved
to a subdirectory called $ARCHIVE/nanook.
NOTE: This change only affects users with data in the /archive/u1/uaf directory.
Data in other directories has already been moved to seawolf.
Frequently asked questions:
1) Which systems use/used nanook.arsc.edu for the $ARCHIVE server?
* SGI systems (retired)
* Iceflyer (retired)
* Linux Workstations - list of systems:
http://www.arsc.edu/cgi/status.cgi?Page_mode=1
2) Which systems use/used seawolf.arsc.edu for the $ARCHIVE server?
* klondike.arsc.edu (retired)
* iceberg.arsc.edu (retired)
* cerebro.arsc.edu (retired)
* nelchina.arsc.edu (retired)
* midnight.arsc.edu
* ognip.arsc.edu
* pingo.arsc.edu
3) Do I need to move my files from nanook.arsc.edu to seawolf.arsc.edu
myself?
No, we can perform this move in an automated fashion if there are
no filename conflicts. When possible, we prefer to move
data in an automated fashion.
4) How will I know if I have identicially named files on both $ARCHIVE
filesystems?
The ARSC help desk will contact users with identically named
files and/or directories. We will request that the naming conflicted
be resolved as soon as possible.
5) What will happen if I have identically named files and/or directories
when the $ARCHIVE move occurs in January?
Any naming conflicts still present in January will be resolved by
ARSC. We will create a "nanook" subdirectory in the user's $ARCHIVE
on seawolf. The files and/or directories will be moved into the new
"nanook" directory.
E.g.
If you had a directory called $ARCHIVE/data on both nanook and
seawolf, the data directory formerly on nanook would be available
on seawolf as $ARCHIVE/nanook/data. The existing directory on
seawolf would continue to be available as $ARCHIVE/data.
6) How do I know whether or not my $ARCHIVE directory will be moved?
If your $ARCHIVE directory is in the /archive/u1/uaf filesystem,
any contents in your $ARCHIVE directory will be moved. You can
echo the value of $ARCHIVE to determine whether or not your $ARCHIVE
directory is in the /archive/u1/uaf filesystem.
e.g.
mallard 3% echo $ARCHIVE
/archive/u1/uaf/username
If you have questions about this pending change, please contact the
ARSC Help Desk.
"compilers"
Last Updated: Wed, 21 Jun 2006 - Donald BahlsMachines: linuxws
Compilers
========================================
The ARSC Linux Workstations have two suites of compilers available:
* The GNU Compiler suite version 4.0 including:
- gcc C compiler
- g++ C++ compiler
- f95 Fortran 95 compiler
* The Portland Group (PGI) compiler suite version 6.1 including:
- pgcc C compiler
- pgCC C++ compiler
- pgf90 Fortran 90 compiler
- pgf95 Fortran 95 compiler
The PGI compilers require several environment variables to be set:
For ksh/bash users:
===================
export PGI=/usr/local/pkg/pgi/pgi-6.1
pgibase=${PGI}/linux86-64/6.1
export PATH=$PATH:${pgibase}/bin
if [ -z "$MANPATH" ]; then
export MANPATH=${pgibase}/man
else
export MANPATH=${pgibase}/man:$MANPATH
fi
if [ -z "$LD_LIBRARY_PATH" ]; then
export LD_LIBRARY_PATH=${pgibase}/lib
else
export LD_LIBRARY_PATH=${pgibase}/lib:$LD_LIBRARY_PATH
fi
unset pgibase
For csh/tcsh users:
===================
setenv PGI /usr/local/pkg/pgi/pgi-6.1/
set pgibase=${PGI}/linux86-64/6.1
setenv PATH ${PATH}:${pgibase}/bin
if ( ! ${?MANPATH} ) then
setenv MANPATH ${pgibase}/man
else
setenv MANPATH ${pgibase}/man:${MANPATH}
endif
if ( ! ${?LD_LIBRARY_PATH} ) then
setenv LD_LIBRARY_PATH ${pgibase}/lib
else
setenv LD_LIBRARY_PATH ${pgibase}/lib:${LD_LIBRARY_PATH}
endif
unset pgibase
"eslogin"
Last Updated: Mon, 28 Dec 2009 - Craig StephensonMachines: pingo
eslogin Nodes ============= The esLogin nodes (external service Login) provide additional functionality for users needing resources beyond what is available on the standard login nodes. The configuration of these nodes is as follows: - 128GB shared memory - 4x quad core AMD Opteron processors (for a total of 16 cores) - 1x 10 Gigabit Ethernet Interconnect Currently there are two esLogin nodes available, and they can be accessed with the following addresses: - pingob.arsc.edu - pingoc.arsc.edu Also, due to the increased resources available on these nodes, it has allowed for the increase of available limits on the nodes. Memory limits on these nodes are currently set at 4GB for soft limits, and 16GB for hard limits. If you require additional resources, please contact ARSC Support. Limitations =========== Currently, there are a couple of differences between the esLogin nodes and the standard login nodes. The esLogin nodes do not have the following capabilities: - The Pathscale compiler suite is unavailable on esLogin nodes - Totalview debugging is currently unavailable from the esLogin nodes - Some library versions differ. If you have dynamically linked executables, there may be some inconsistency between the node types. - Some packages available on the standard login nodes may not be available. If there is a package you require to complete work, and it is not available, please contact us. Questions ========= If you have any questions regarding the capability of the esLogin nodes, or experience any difficulty while using them, please contact the ARSC Support staff and we will do our best to work through any issues you may experience.
"file_striping"
Last Updated: Wed, 03 Mar 2010 - Oralee NudsonMachines: pingo
File Striping ======================================== The $WORKDIR Lustre filesystem on midnight is composed of multiple object storage targets (OSTs). The default filesystem settings place each file on one OST. These settings are appropriate for many applications, however some applications may benefit by enabling striping. Striping places a file on multiple OSTs. This can: * Provide better bandwidth to files accessed by many tasks simultaneously. * Increase read and write performance for I/O intensive codes. * More evenly distribute large files across OSTs. Striping should be considered for any file larger than 50 GB. The following command shows how to enable striping across 6 OSTs for an empty directory: lfs setstripe $WORKDIR/case1 -s 0 -i -1 -c 6 NOTE: All new files created in the $WORKDIR/case1/ directory will take on these striping attributes. Files moved within the same filesystem or files that already exist within the directory will maintain their existing striping characteristics. See "lfs setstripe" for a description of striping parameters. The following set of commands shows how to stripe a directory across 6 OSTs and how to copy an existing large file into that directory (so the large file will also be striped.) cd $WORKDIR/case1 mkdir striped lfs setstripe striped -s 0 -i -1 -c 6 cp largeFile.txt striped/largeFile.txt # confirm largeFile.txt data looks normal rm largeFile.txt To confirm the file striping was successful, check the particular file or directory with the following commands: lfs getstripe filename lfs getstripe directoryname
"file_striping"
Last Updated: Mon, 26 Jan 2009 - Oralee NudsonMachines: midnight
File Striping ======================================== The $WORKDIR Lustre filesystem on midnight is composed of multiple object storage targets (OSTs). The default filesystem settings place each file on one OST. These settings are appropriate for many applications, however some applications may benefit by enabling striping. Striping places a file on multiple OSTs. This can: * Provide better bandwidth to files accessed by many tasks simultaneously. * Increase read and write performance for I/O intensive codes. * More evenly distribute large files across OSTs. Striping should be considered for any file larger than 50 GB. The following command shows how to enable striping across 6 OSTs for an empty directory: lfs setstripe $WORKDIR/case1 0 -1 6 NOTE: All new files created in the $WORKDIR/case1/ directory will take on these striping attributes. Files moved within the same filesystem or files that already exist within the directory will maintain their existing striping characteristics. See "lfs setstripe" for a description of striping parameters. The following set of commands shows how to stripe a directory across 6 OSTs and how to copy an existing large file into that directory (so the large file will also be striped.) cd $WORKDIR/case1 mkdir striped lfs setstripe striped 0 -1 6 cp largeFile.txt striped/largeFile.txt # confirm largeFile.txt data looks normal rm largeFile.txt To confirm the file striping was successful, check the particular file or directory with the following commands: lfs getstripe filename lfs getstripe directoryname
"filezilla"
Last Updated: Mon, 28 Dec 2009 - Craig StephensonMachines: pingo
Filezilla ========= This news item is for users of the Filezilla ftp client which is included with the HPCMP Kerberos kit for Windows. Some users have reported problems connecting to the Linux Workstations with the Filezilla client. Should you experience these problems with Filezilla, try the following: Option 1) Set the transfer mode to passive: a) Select the menus "Edit | Settings" b) Select the tree option "Firewall Settings " c) Then check the passive mode check box Option 2) Set the ftp protocol to sftp: a) Select the menus "File | Site Manager" b) Click "New Site" button. Enter the machine name. c) Fill in the "host" (e.g. iceberg.arsc.edu) d) Set "Servertype" to "SFTP using SSH2" e) Set "Logontype" to "Normal" f) Set the "User" to your username. g) Click the "Connect" button to connect. Sites entered in the Site Manager can be also be selected using the leftmost icon on the Filezilla toolbar. If you experience problems with the Filezilla client using these recommendations, please report them to the ARSC help desk (consult@arsc.edu).
"filezilla"
Last Updated: Mon, 28 Dec 2009 - Craig StephensonMachines: linuxws
Filezilla ========= This news item is for users of the Filezilla ftp client which is included with the HPCMP Kerberos kit for Windows. Some users have reported problems connecting to the Linux Workstations with the Filezilla client. Should you experience these problems with Filezilla, try the following: Option 1) Set the transfer mode to passive: a) Select the menus "Edit | Settings" b) Select the tree option "Firewall Settings " c) Then check the passive mode check box Option 2) Set the ftp protocol to sftp: a) Select the menus "File | Site Manager" b) Click "New Site" button. Enter the machine name. c) Fill in the "host" (e.g. iceberg.arsc.edu) d) Set "Servertype" to "SFTP using SSH2" e) Set "Logontype" to "Normal" f) Set the "User" to your username. g) Click the "Connect" button to connect. Sites entered in the Site Manager can be also be selected using the leftmost icon on the Filezilla toolbar. If you experience problems with the Filezilla client using these recommendations, please report them to the ARSC help desk (consult@arsc.edu).
"filezilla"
Last Updated: Thu, 14 May 2009 - Craig StephensonMachines: midnight
Filezilla ========= This news item is for users of the Filezilla ftp client which is included with the HPCMP Kerberos kit for Windows. Some users have reported problems connecting to Midnight with the Filezilla client. Should you experience these problems with Filezilla, try the following: Option 1) Set the transfer mode to passive: a) Select the menus "Edit | Settings" b) Select the tree option "Firewall Settings " c) Then check the passive mode check box Option 2) Set the ftp protocol to sftp: a) Select the menus "File | Site Manager" b) Click "New Site" button. Enter the machine name. c) Fill in the "host" (e.g. midnight.arsc.edu) d) Set "Servertype" to "SFTP using SSH2" e) Set "Logontype" to "Normal" f) Set the "User" to your username. g) Click the "Connect" button to connect. Sites entered in the Site Manager can be also be selected using the leftmost icon on the Filezilla toolbar. If you experience problems with the Filezilla client using these recommendations, please report them to the ARSC help desk (consult@arsc.edu).
"idl"
Last Updated: Tue, 25 Apr 2006 - Andrew LeeMachines: linuxws
IDL at ARSC
========================================
IDL is installed at ARSC on the Linux workstations and iceberg.
To run, you must first make sure the following two steps are done:
1) Files named idl_setup and idl_setup.ksh are
automatically created by the installation script. These
files contain appropriate definitions for IDL-related
environment variables and aliases. Users must set up their
accounts as shown below:
C-shell users add the line:
source /usr/local/pkg/idl/idl-6.2/idl/bin/idl_setup
source /usr/local/pkg/idl/idl-6.2/envi/bin/envi_setup
to your .cshrc files.
Korn shell users add the line:
. /usr/local/pkg/idl/idl-6.2/idl/bin/idl_setup.ksh
. /usr/local/pkg/idl/idl-6.2/envi/bin/envi_setup.ksh
to your .profile files.
Bash shell users add the line:
. /usr/local/pkg/idl/idl-6.2/idl/bin/idl_setup.bash
. /usr/local/pkg/idl/idl-6.2/envi/bin/envi_setup.bash
to your .bashrc files.
2) Make sure the license variable is set correctly, if not add this
line to the same file as listed above for your shell:
C-shell (csh, tcsh)
setenv LM_LICENSE_FILE 7111@license1.arsc.edu:7111@license.arsc.edu
Korn shell (ksh)
export LM_LICENSE_FILE=7111@license1.arsc.edu:7111@license.arsc.edu
Bash shell (bash)
export LM_LICENSE_FILE=7111@license1.arsc.edu:7111@license.arsc.edu
3) Some users have reported malloc errors with the 64 bit version
on envi and idl. We are currently investigating these errors.
Should you encounter a malloc error try using the 32 bit version of
idl or envi.
The "-32" flag will invoke the 32 bit executable:
e.g.
% idl -32
"idle"
Last Updated: Wed, 01 Feb 2006 - Andrew LeeMachines: seawolf
Idle Time-outs ======================================== As of February 1, 2006, ARSC has instituted time limits for login sessions to many of our systems. In order to minimize the exposure of your login sessions to 'session hijacking' vulnerabilities, we are limiting the total login time to 24 hours. All logins will has been open for more than 6 hours, it will be terminated when it has been idle for more than 4 consecutive hours within that 24 hr period. All login sessions will be terminated once they have reached 24 hours -- regardless of if there is activity at the 24 hour mark or not. Please see: http://www.arsc.edu/support/policy/secpolicy.html or contact the ARSC Help Desk (consult@arsc.edu, 907-450-8602) for more information.
"kerberos"
Last Updated: Wed, 22 Apr 2009 - Donald BahlsMachines: linuxws midnight pingo seawolf
Required Kerberos Updates
========================================
Recently MIT announced two vulnerablities in Kerberos. Due to the
nature of these vulnerablities, ARSC and HPCMP are requiring all
kerberos clients to be updated to the latest release by May 22nd, 2009.
The following versions of Kerberos or newer will be the only supported
versions beyond May 22nd:
* HPCMP_RELEASE_20090331 (Linux / Unix / Mac OS X)
and
* HPCMP_RELEASE_20090422 (Windows).
Updated clients are available from the HPCMP Kerberos website.
Precompiled clients can be found under the "Software" link on the
left of this page.
https://www.hpcmo.hpc.mil/security/kerberos/
The known vulnerability also affects HPCMP OpenSSH client kits.
All versions older than OpenSSH-5.0p1c will not be allowed after
May 22nd. Updated versions of the HPCMP OpenSSH client kits are also
available at the HPCMP Kerberos website.
Frequently Asked Questions:
1) How can I tell which version of Kerberos I have installed?
a) For Linux, Solaris and Mac OS X:
# run the krlogin command with the -V option.
% krlogin -V
usage: rlogin host [-option] [-option...] [-k realm ] [-t ttytype] [-l username]
where option is e, 7, 8, noflow, n, a, x, f, F, c, PO, or PN
version: @(#) $NRL: HPCMP_RELEASE_20081006 20081006 $
The version of this kit is HPCMP_RELEASE_20081006.
b) For Windows:
Start the PuTTY client, then select the "About" button in the lower
left corner. The HPCMP_RELEASE_20090409 release should indicate
"PuTTY HPCMP Release 0.60" or newer is being used.
2) How can I check the version of ssh I have installed?
a) For Linux, Solaris and Mac OS X
% ssh -V
OpenSSH_5.2p1a, OpenSSL 0.9.8k 25 Mar 2009
b) For Windows
(See question 1.b above)
3) I'm not using Linux, Solaris, Mac OS X or Windows, how can I
obtain a kit for my Operating System?
Client kits for operating systems not listed above must be
compiled from source. Kerberos source code is available from:
https://www.hpcmo.hpc.mil/security/kerberos/
4) Are there a direct links to the client kits available?
Yes.
a) Linux and Solaris
https://www.hpcmo.hpc.mil/security/kerberos/software/clients/kerberos/index.html
b) Windows
https://www.hpcmo.hpc.mil/security/kerberos/software/clients/kerberos/indexWin.html
c) Macintosh
https://www.hpcmo.hpc.mil/security/kerberos/software/clients/kerberos/indexMac.html
"largefile"
Last Updated: Sat, 13 Feb 2010 - Don BahlsMachines: seawolf
Large File Support ======================================== Some commands on seawolf do not have support for files larger that 2 GB. See "man largefile" for a list of commands which have large file support. In particular the "gzip" and "bzip2" are 32 bit executables on seawolf and therefore may truncate large files. If you need to manipulate files larger than 2 GB, it is recommended that an HPC system or Linux Workstation be used. If you have questions, please contact User Support.
"matlab"
Last Updated: Wed, 20 Jan 2010 - Don BahlsMachines: midnight
Matlab License Changes
========================================
Several changes to matlab licensing are occurring in order to provide
better support for licensed product through Advanced Reservations.
These changes will require users of matlab to make changes to their
normal workflow as described below.
This change in policy will officially go into effect on January
26th, 2010.
Jobs Run Through PBS:
=====================
On January 20th, 2010 a PBS resource will be added for matlab.
This resource will ensure that licenses are available prior to the
start of a PBS job. Matlab requires one license per node used by
the job using the matlab resource (e.g. #PBS -l matlab=1):
#!/bin/bash
#PBS -l walltime=1:00:00
#PBS -l matlab=1
#PBS -l select=1:ncpus=4:node_type=4way
#PBS -q standard
#PBS -j oe
cd $PBS_O_WORKDIR
matlab < input.m
Matlab Changes for Login Nodes:
================================
All matlab work must request license reservations via the
"matlab_license_check" script. This script requires that you specify
the length of time you intend to run matlab. Any work requesting
more than 1 hour should be run through PBS as described above.
e.g.
# Request a 60 minute reservation
mg56 % /usr/local/bin/matlab_license_check -w 60
Reservation successfully created!
module load matlab-7.8.0
matlab < input.m
Running Matlab On Compute Nodes:
=================================
Matlab can be run on compute nodes with full access to the GUI by using the
following technique:
A) Log into midnight1 or midnight2 with X11 forwarding enabled.
% ssh -X -Y username@midnight.arsc.edu
B) Run the "tunnelx" command to setup X11 forwarding for the
compute nodes.
midnight% tunnelx
C) Start an interactive PBS job requesting matlab:
midnight% qsub -l select=1:ncpus=1:node_type=4way -lmatlab=1 -I
D) Once the interactive PBS job starts, rerun the "tunnelx"
command to initialize the X11 settings for your shell.
mt101% tunnelx
Next, source the file ~/.tx_local which was generated by the tunnelx
command.
# bash/ksh syntax
mt101% . ~/.tx_local
# csh/tcsh syntax
mt101% source ~/.tx_local
E) You should be able to run matlab with X11 support.
mt101% module load matlab-7.6.0
mt101% matlab
F) When you are done using matlab, be sure to run "exit" to end the interactive
PBS job.
"modules"
Last Updated: Mon, 28 Dec 2009 - Craig StephensonMachines: pingo
Using the Modules Package ========================= The modules package is used to prepare the environment for various applications before they are run. Loading a module will set the environment variables required for a program to execute properly. Conversely, unloading a module will unset all environment variables that had been previously set. This functionality is ideal for switching between different versions of the same application, keeping differences in file paths transparent to the user. Sourcing the Module Init Files --------------------------------------------------------------------- For some jobs, it may be necessary to source these files, as they may not be sourced automatically as with login shells. Before the modules package can be used, its init file must first be sourced. To do this using tcsh or csh, type: source /opt/modules/default/init/To do this using bash, ksh, or sh, type: . /opt/modules/default/init/ For either case, replace with the shell you are using. If your shell is bash, for example: . /opt/modules/default/init/bash Once the modules init file has been sourced, the following commands become available: Command Purpose --------------------------------------------------------------------- module avail - list all available modules module load - load a module file from environment module unload - unload a module file from environment module list - display modules currently loaded module switch - replace module with module module purge - unload all modules
"modules"
Last Updated: Mon, 28 Dec 2009 - Craig StephensonMachines: linuxws
Using the Modules Package ========================= The modules package is used to prepare the environment for various applications before they are run. Loading a module will set the environment variables required for a program to execute properly. Conversely, unloading a module will unset all environment variables that had been previously set. This functionality is ideal for switching between different versions of the same application, keeping differences in file paths transparent to the user. The following modules commands are available: module avail - list all available modules module load- load a module file from environment module unload - unload a module file from environment module list - display modules currently loaded module switch - replace module with module module purge - unload all modules Before the modules package can be used in a script, its init file may need to be sourced. To do this using tcsh or csh, type: source /usr/local/pkg/modules/init/ To do this using bash, ksh, or sh, type: . /usr/local/pkg/modules/init/ For either case, replace with the shell you are using. If your shell is bash, for example: . /usr/local/pkg/modules/init/bash Known Issues: ============= 2009-09-24 Accounts using bash that were created before 9/24/2009 are missing the default ~/.bashrc file. This may cause the module command to be unavailable in some instances. Should you experience this issue run the following: # copy the template .bashrc to your account. [ ! -f ~/.bashrc ] && cp /etc/skel/.bashrc ~ If you continue to experience issues, please contact the ARSC Help Desk.
"modules"
Last Updated: Fri, 30 Oct 2008 - Donald BahlsMachines: midnight
Using the Modules Package ========================= The modules package is used to prepare the environment for various applications before they are run. Loading a module will set the environment variables required for a program to execute properly. Conversely, unloading a module will unset all environment variables that had been previously set. This functionality is ideal for switching between different versions of the same application, keeping differences in file paths transparent to the user. Sourcing the Module Init Files --------------------------------------------------------------------- For some jobs, it may be necessary to source these files, as they may not be auto-sourced as with login shells. Before the modules package can be used, its init file must first be sourced. To do this using tcsh or csh, type: source /usr/share/modules/init/To do this using bash, ksh, or sh, type: . /usr/share/modules/init/ For either case, replace with the shell you are using. If your shell is bash, for example: . /usr/share/modules/init/bash Once the modules init file has been sourced, the following commands become available: Command Purpose --------------------------------------------------------------------- module avail - list all available modules module load - load a module file from environment module unload - unload a module file from environment module list - display modules currently loaded module switch - replace module with module module purge - unload all modules
"mpi"
Last Updated: Wed, 30 Apr 2008 - Donald BahlsMachines: midnight
MPI Environment
========================================
Midnight uses the Voltaire versions of the MVAPICH MPI stack. A list of
MVAPICH specific features is available on the ARSC webpage here:
http://www.arsc.edu/support/howtos/usingsun.html#mpi
Changes to MPI Stack
====================
The following are a list of changes to the MPI stack on midnight.
2008-05-19:
Task affinity is now enabled by default for MPI jobs. It can be disabled
by using "-noaff". See 2008-04-30 changes for details.
2008-04-30:
The mpirun command now includes an option to enable task affinity (-aff).
This option helps decrease the variability in run time for many applications,
however it should not be used for hybrid applications (e.g. MPI/OpenMP).
The "-noaff" option can be used do disable task affinity. This option should
be used for hybrid MPI/OpenMP or MPI/pthread applications.
Task affinity will be enabled for all MPI applications not specify "-noaff"
during maintenance on May 17-18th.
MPI Feature Support
===================
The following MPI features are not currently supported on midnight.
1) MPI-2 and MPI-IO. MPI-2 features including MPI-IO are not currently
available on midnight.
2) MPMD Support. The MVAPICH stack does not support the multiple program
multiple data (MPMD) programming model used by some codes (e.g. CCSM).
"pbs_10"
Last Updated: Wed, 13 May 2009 - Donald BahlsMachines: midnight
PBS Pro 10 Upgrade
========================================
During scheduled maintenance on May 16th - 18th, PBS Pro will
be upgraded from version 7.1.3 to version 10.0. Most users will not
need to make changes to job scripts to ensure unaltered operation of
those scripts. There are several exceptions listed below.
1) Node Placement:
Jobs using 1/2 of the cores on a node or less will need to
include a placement statement to ensure the proper number of cores
per node are used.
e.g.,
# use 4 nodes with 2 cores per node
#PBS -l select=4:ncpus=2:node_type=4way
# place only one set of 2 tasks per node.
#PBS -l place=scatter:excl
...
2) Backfill:
The PBS scheduler will be updated to use job based priorities and
eligible time with backfill. Jobs will gain priority based on the time
the job is in the queue and eligible to run. If a job can run without
interfering with the start time of the highest priority job, that job
will be eligible for backfill. Jobs specifying shorter walltimes
and smaller processor counts generally have more opportunities to
take advantage of backfill windows. Significant overestimates in
walltime requests decrease the likelihood backfill can occur.
e.g.,
# Example of limits which may backfill nicely
#PBS -l select=4:ncpus=4:node_type=4way
#PBS -l walltime=2:00:00
...
3) Use of the -V flag:
Following the upgrade to PBS Pro 10, it was determined that the -V
option can cause PBS to work improperly under certain circumstances.
The following workarounds can be used:
a) Use the -v (small v) option to set individual environment variables
during job submission:
e.g.
qsub -v VARIABLE=value myscript.pbs
b) If the variable value is static, you can add the required
environment variables to your ~/.profile or ~/.login
c) Unset unneeded environment variables prior to submitting your PBS
script:
# bash/ksh
unset LM_LICENSE_FILE #unset license file string
qsub -V myscript.pbs
# csh/tcsh
unsetenv LM_LICENSE_FILE #unset license file string
qsub -V myscript.pbs
4) Script names not starting with a letter.
Script names not starting with a letter may not function properly
at runtime.
The following workarounds can be used:
Rename the script so the first character is a letter rather than a
number or symbol.
e.g.
mv 2test.pbs a2test.pbs
If you experience issues not described by this news item, please
contact User Support. The following command will provide contact
information:
% support
"pin"
Last Updated: Fri, 27 Aug 2007 - Derek BastilleMachines: midnight seawolf
New SecurID card PINs to be 5 digit ======================================== In order to improve the security and effectiveness of our authentication system, ARSC will begin issuing 5 digit SecurID card PINs as of 1 May, 2007. Existing SecurID cards will NOT have their PINs updated at this time. However, if a PIN reset is needed, the replacement PIN will be 5 digit. All new SecurID cards issued after 1 May will be set to 5 digit PINs. If you have any questions please contact the ARSC Help Desk.
"projects"
Last Updated: Wed, 17 Dec 2008 - Donald BahlsMachines: midnight
Instructions for Users with Multiple Projects
========================================
This news item is intended for users that are members of more than one
project. Users in a single project will automatically have use
charged against the allocation for their primary group (i.e. project).
Users in more than one project can select an alternate project to
charge use to by using the "-W group_list" PBS option. If the
"-W group_list" option is not specified the account number will default
to your primary group (i.e. project).
Below is an example "-W group_list" statement.
e.g.
#PBS -W group_list=proja
The "-W group_list" option can also be used on the command line.
e.g.
mg56 % qsub -Wgroup_list=proja script.bat
Each project has a corresponding UNIX group, therefore the groups
command will show all projects and other groups of which you are a
member.
e.g.
mg56 % groups
proja projb
In this case use would be charged to proja by default, but could be
charged to projb by setting "-W group_list=projb" in the PBS script.
If you have questions about this news item, please contact the ARSC
help desk (consult@arsc.edu).
'show_usage' Available
=======================
Project utilization information is now available via the BCT Compliant 'show_usage'
command. 'show_usage' with no parameters will display the remaining
allocation for each project of which you are a member.
e.g.
mg56 % show_usage
ARSC - Subproject Usage Information (in CPU Hours)
As of 01:11:21 hours ADT 11 Nov 2007
For Fiscal Year 2008 (01 October 2007 - 30 September 2008)
Percentage of Fiscal Year Remaining: 88.77%
Hours Hours Hours Percent Background
System Subproject Allocated Used Remaining Remaining Hours Used
========== ============== ========== ========== ========== ========= ==========
midnight projectA 100000.00 39663.48 60336.52 60.34% 0.00
midnight projectB 5000.00 0.00 5000.00 100.00% 0.00
Project utilization for all allocated systems can be obtained by using 'show_usage'
with the '-s' option as in 'show_usage -s'.
mg56 % show_usage -s
ARSC - Subproject Usage Information (in CPU Hours)
As of 13:00:00 hours ADT 11 Nov 2007
For Fiscal Year 2008 (01 October 2007 - 30 September 2008)
Percentage of Fiscal Year Remaining: 88.77%
Hours Hours Hours Percent Background
System Subproject Allocated Used Remaining Remaining Hours Used
========== ============== ========== ========== ========== ========= ==========
iceberg projectA 30000.00 309.84 29690.16 98.97% 0.00
iceberg projectB 10000.00 3873.46 6126.54 61.27% 0.00
midnight projectA 100000.00 39663.48 60336.52 60.34% 0.00
midnight projectB 5000.00 0.00 5000.00 100.00% 0.00
"projects"
Last Updated: Wed, 17 Dec 2008 - Donald BahlsMachines: pingo
Instructions for Users with Multiple Projects
========================================
This news item is intended for users that are members of more than one
project. Users in a single project will automatically have use
charged against the allocation for their primary group (i.e. project).
Users in more than one project can select an alternate project to
charge use to by using the "-W group_list" PBS option. If the "-W
group_list" option is not specified the account number will default
to your primary group (i.e. project).
Below is an example "-W group_list" statement.
e.g.
#PBS -W group_list=proja
The "-W group_list" option can also be used on the command line.
e.g.
pingo4 % qsub -Wgroup_list=proja script.bat
Each project has a corresponding UNIX group, therefore the groups
command will show all projects and other groups of which you are a
member.
e.g.
pingo4 % groups
proja projb
In this case use would be charged to proja by default, but could be
charged to projb by setting "-W group_list=projb" in the PBS script.
If you have questions about this news item, please contact user
support.
'show_usage' Available
=======================
Project utilization information is now available via the BC Compliant
'show_usage' command. 'show_usage' with no parameters will display
the remaining allocation for each project of which you are a member.
e.g.
pingo4 % show_usage
ARSC - Subproject Usage Information (in CPU Hours)
As of 01:11:21 hours ADT 11 Nov 2007
For Fiscal Year 2008 (01 October 2007 - 30 September 2008)
Percentage of Fiscal Year Remaining: 88.77%
Hours Hours Hours Percent Background
System Subproject Allocated Used Remaining Remaining Hours Used
========== ============== ========== ========== ========== ========= ==========
pingo projectA 100000.00 39663.48 60336.52 60.34% 0.00
pingo projectB 5000.00 0.00 5000.00 100.00% 0.00
Project utilization for all allocated systems can be obtained by using
'show_usage' with the '-s' option as in 'show_usage -s'.
pingo4 % show_usage -s
ARSC - Subproject Usage Information (in CPU Hours)
As of 13:00:00 hours ADT 11 Nov 2007
For Fiscal Year 2008 (01 October 2007 - 30 September 2008)
Percentage of Fiscal Year Remaining: 88.77%
Hours Hours Hours Percent Background
System Subproject Allocated Used Remaining Remaining Hours Used
========== ============== ========== ========== ========== ========= ==========
pingo projectA 30000.00 309.84 29690.16 98.97% 0.00
pingo projectB 10000.00 3873.46 6126.54 61.27% 0.00
midnight projectA 100000.00 39663.48 60336.52 60.34% 0.00
midnight projectB 5000.00 0.00 5000.00 100.00% 0.00
"queues"
Last Updated: Fri, 02 Mar 2009 - Oralee NudsonMachines: midnight
Midnight Queues
===============
The queue configuration is as described below. It is subject to
review and further updates.
Login Nodes Use:
=================
Login nodes are a shared resource and are not intended for
computationally or memory intensive work. Processes using more
than 30 minutes of CPU time on login nodes may be killed by ARSC
without warning. Please use compute nodes for computationally or
memory intensive work.
Queues:
===============
Specify one of the following queues in your PBS qsub script
(e.g., "#PBS -q standard"):
Queue Name Purpose of queue
------------- ------------------------------
standard General use by all allocated users.
debug Quick turnaround queue for debugging work.
data Provides access to long term storage (i.e. $ARCHIVE_HOME)
background For projects with little or no remaining allocation.
This queue has the lowest priority, however projects
running jobs in this queue do not have allocation deducted.
The number of running jobs or processors available to this
queue may be altered based on system load.
special For jobs which do not fit into normal queue limits. (Limited Access)
high For jobs requiring higher priority than normal. (Special Access Only)
urgent For jobs requiring highest priority. (Special Access Only)
challenge For challenge projects only.
See 'qstat -q' for a complete list of system queues. Note, some
queues are not intended for general use.
Maximum Walltimes:
===================
The maximum allowed walltime for a job is dependant on the number of
processors requested. The table below describes maximum walltimes for
each queue.
Queue Min Max Max
Procs Procs Walltime Notes
--------------- ----- ----- --------- ------------
standard
1 16 84:00:00 See (A)
17 256 16:00:00
257 512 12:00:00
challenge
1 16 96:00:00 See (B)
17 256 48:00:00 See (F)
257 516 48:00:00 See (G)
background 1 512 12:00:00
debug 1 32 01:00:00 See (D)
data 1 1 12:00:00 See (E)
NOTES:
(A) The max walltime for standard_sm jobs (i.e. 1 to 16 processors) was
increased from 42 hours to 84 hours on October 5th 2007.
(B) The max walltime for challenge_sm jobs (i.e. 1 to 16 processors) was
increased from 48 hours to 96 hours on October 5th 2007.
(C) The max walltime for challenge_med jobs (i.e. 17 to 256 processors) was
increased from 24 hours to 96 hours on November 28th 2007.
(D) As of January 2009 the debug queue now has 4 dedicated nodes for
quick turn around debugging work. 2- 4way nodes and 4- 16way nodes.
(E) Do not specify a node_type when submitting jobs to the data queue.
Please refer to the sample PBS data staging script below for an example.
(F) The max walltime for challenge_med jobs (i.e. 17 to 256 processors) was
decreased from 96 hours to 48 hours on September 25, 2008.
(G) The max number of CPUs was increased from 512 to 516 on August 19, 2009.
The max walltime was increased from 12 hours to 48 hours on August 21, 2009.
PBS Commands:
=============
Below is a list of common PBS commands. Additional information is
available in the man pages for each command.
Command Purpose
-------------- -----------------------------------------
qsub submit jobs to a queue
qdel delete a job from the queue
qsig send a signal to a running job
qmap display a map of the system (ARSC only)
Running a Job:
==============
To run a batch job, create a qsub script which, in addition to
running your commands, specifies the processor resources and time
required. Submit the job to PBS with the following command. (For
more PBS directives, type "man qsub".)
qsub <script file>
Sample PBS scripts:
--------------
#### Beginning of MPI Example Script using Sun Fire x2200 nodes ############
#!/bin/bash
#PBS -q standard
#PBS -l select=8:ncpus=4:node_type=4way
#PBS -l walltime=8:00:00
#PBS -j oe
cd $PBS_O_WORKDIR
mpirun -np 32 ./myprog
#### End of Sample Script ##################
#### Beginning of MPI Example Script using Sun Fire x4600 nodes ############
#!/bin/bash
#PBS -q standard
#PBS -l walltime=8:00:00
#PBS -l select=2:ncpus=16:node_type=16way
#PBS -j oe
cd $PBS_O_WORKDIR
mpirun -np 32 ./myprog
#### End of Sample Script ##################
#### Beginning of OpenMP Example Script using Sun Fire x4600 nodes ############
#!/bin/bash
#PBS -q standard
#PBS -l select=1:ncpus=16:node_type=16way
#PBS -l walltime=8:00:00
#PBS -j oe
cd $PBS_O_WORKDIR
export OMP_NUM_THREADS=16
export PSC_OMP_AFFINITY=TRUE
./myprog
#### End of Sample Script ##################
#### Beginning of Data Staging Example Script ############
#!/bin/bash
#PBS -q data
#PBS -l walltime=4:00:00
#PBS -l select=1:ncpus=1
#PBS -j oe
cd $PBS_O_WORKDIR
cp -r $ARCHIVE_HOME/mydataset/* . || exit 1
qsub mpi_job.pbs
#### End of Sample Script ##################
Resource Limits:
==================
The only resource limits users should specify are walltimes and select,
ncpus, and node_type limits. The "select" statement requests a job be
allocated a number of chunks with the given "ncpus" size. The "node_type"
limit is used to explicitly request a job be placed on x2200 or x4600 nodes.
Tracking Your Job:
==================
To see which jobs are queued and/or running, execute this
command:
qstat -a
Current Queue Limits:
=====================
Queue limits are subject to change and this news item is not always
updated immediately. For a current list of all queues, execute:
qstat -Q
For all limits on a particular queue:
qstat -Q -f <queue-name>
Maintenance
============
Scheduled maintenance activities on Midnight use the Dedicated Time
functionality of PBS to reserve all available nodes on the system.
This reservation keeps PBS from scheduling jobs which would still be
running during maintenance. This allows the queues to be left running
until maintenance. Because walltime is used to determine whether
or not a job will complete prior to maintenance, using a shorter
walltime in your job script may allow your job to be run sooner.
e.g.
If maintenance begins at 6PM and it is currently 4PM, jobs specifying
walltimes of 2 hours or less will start if there are available nodes.
CPU Usage
==========
Only one job may run per node on midnight (i.e. jobs may not share nodes).
If your job uses fewer than the number of available processors on a node
the job will be charged for all processors on the node.
* A job running on a X2200 node (i.e. node_type=4way) consumes 4 CPU hours
of allocation per hour regardless of the number of tasks running on each
node.
* A job running on a X4600 node (i.e. node_type=16way) consumes 16 CPU hours
of allocation per hour regardless of the number of tasks running on each
node.
PBS Scripts - Default Walltimes
===============================
As of January 22, 2008, any PBS script submitted to the scheduler not
including a walltime for the job will be given a default walltime of 8
hours. The only exception to this is the debug queue, which will default
to 1 hour. To indicate a walltime, add the following line to your
PBS script:
#PBS -l walltime=1:30:00
This command will request a walltime of 1 hour and 30 minutes.
"queues"
Last Updated: Wed, 17 Dec 2008 - Donald BahlsMachines: pingo
Pingo Queues
========================================
The queue configuration is as described below. It is subject to
review and further updates.
Queues:
===============
Specify one of the following queues in your PBS qsub script
(e.g., "#PBS -q standard"):
Queue Name Purpose of queue
------------- ------------------------------
standard General use by all allocated users.
challenge For challenge projects only.
background For projects with little or no remaining allocation.
This queue has the lowest priority, however
projects running jobs in this queue do not have
allocation deducted. The number of running jobs or
processors available to this queue may be altered
based on system load.
debug Quick turn around debugging queue.
data Provides access to long term storage
(i.e. $ARCHIVE_HOME)
high For jobs requiring higher priority than
normal. (Special Access Only)
special For jobs which do not fit into normal queue
limits. (Limited Access)
urgent For jobs requiring highest priority. (Special
Access Only)
See 'qstat -q' for a complete list of system queues. Note, some
queues are not intended for general use.
Maximum Walltimes:
===================
The maximum allowed walltime for a job is dependent on the number of
processors requested. The table below describes maximum walltimes
for each queue.
Queue Min Max Max
Procs Procs Walltime Notes
--------------- ----- ----- --------- ------------
standard
1 16 84:00:00 routes to standard_sm
17 1024 16:00:00 routes to standard_lg
challenge
1 16 96:00:00 routes to challenge_sm
17 1536 48:00:00 routes to challenge_lg
background 1 512 8:00:00
debug 1 128 1:00:00
data 1 1 12:00:00
PBS Commands:
=============
Below is a list of common PBS commands. Additional information is
available in the man pages for each command.
Command Purpose
-------------- -----------------------------------------
qsub submit jobs to a queue
qdel delete a job from the queue
qsig send a signal to a running job
Running a Job:
==============
To run a batch job first create a qsub script which, in addition
to running your commands, specifies the processor resources and
time required. Submit the job to PBS with the following command.
(For more PBS directives, type "man qsub".)
qsub <script file>
Sample PBS scripts:
--------------------
Example: MPI-based Application
#!/bin/bash
#PBS -q standard
#PBS -l mppwidth=32
#PBS -l walltime=8:00:00
#PBS -j oe
cd $PBS_O_WORKDIR
aprun -n 32 ./myprog
Example: OpenMP-based Application
#!/bin/bash
#PBS -q standard
#PBS -l mppwidth=8
#PBS -l walltime=8:00:00
#PBS -j oe
cd $PBS_O_WORKDIR
export OMP_NUM_THREADS=8
aprun -n 1 -d 8 ./myprog
Example: Long Term Data Staging
#!/bin/bash
#PBS -q data
#PBS -l walltime=4:00:00
#PBS -j oe
cd $PBS_O_WORKDIR
cp -r $ARCHIVE_HOME/mydataset/* . || exit 1
qsub mpi_job.pbs
Resource Limits:
==================
The only resource limits users should specify are walltimes
and mppwidth requests.
Tracking Your Job:
==================
To see which jobs are queued and/or running, execute this
command:
qstat -a
Current Queue Limits:
=====================
Queue limits are subject to change and this news item is not always
updated immediately. For a current list of all queues, execute:
qstat -Q
For all limits on a particular queue:
qstat -Q -f <queue-name>
Maintenance
============
Scheduled maintenance activities on pingo use the Dedicated Time
functionality of PBS to reserve all available nodes on the system.
This reservation prevents PBS from scheduling jobs which would still
be running during maintenance, and enables the queues to run until
maintenance begins. Because walltime is used to determine whether
or not a job will complete prior to maintenance, using a shorter
walltime in your job script may allow your job to run sooner.
e.g.
If maintenance begins at 6PM and it is currently 4PM, jobs specifying
walltimes of 2 hours or less will start if nodes are available.
CPU Usage
==========
Only one job may run per node on pingo (i.e. jobs may not share nodes).
If your job uses fewer than the number of available processors on a node
the job will be charged for all processors on the node.
* A job running on a node consumes 8 CPU hours of allocation per hour
regardless of the number of tasks running on each node.
Queue Changes
=============
2009-07-10 Maximum walltime for challenge_lg was increased from
24 hours to 48 hours.
2009-07-28 Maximum mppwidth for standard_lg was increased from
768 to 1024.
"retirement"
Last Updated: Wed, 10 Feb 2010 - Don BahlsMachines: midnight
Midnight Retirement ======================================== On June 1st 2010, midnight.arsc.edu will be taken out of regular production status. Existing allocations on the system will be honored through June 1st. Following June 1st and up through September 30, 2010, midnight.arsc.edu will continue to operate and will remain available to existing users, but without full vendor support. If certain hardware and/or software issues arise between June 1st and September 30th, the system may become unavailable for unscheduled prolonged outages. If you have questions regarding these planned changes, please contact User Support.
"samples_home"
Last Updated: Thu, 01 Jan 2009 - ARSC Help DeskMachines: midnight
Baseline Configuration Sample Code Repository
=============================================
Filename: INDEX.txt
Description: This file contains the name,location, and brief
explanation of "samples" included in this Sample
Code Repository. There are several subdirectories within
this code repository containing frequently-used procedures,
routines, scripts, and code used on this allocated system,
midnight. This sample code repository can be
accessed from midnight by changing directories to
$SAMPLES_HOME, or changing directories to the following
location: mg56% /usr/local/pkg/samples. This particular
file can be viewed from the internet at:
http://www.arsc.edu/support/news/systemnews/midnightnews.xml#samples_home
Contents: AVAILABLE_TOOLS.txt
applications
dataManagement
debugging
jobSubmission
libraries
parallelEnvironment
******************************************************************************
Directory: applications
Description: This directory includes example uses of the software
packages available on midnight. Each directory name
corresponds to the name of the software package example.
The contents of each subdirectory may include the
following information:
* sample input files
* how to access a software license (if required)
* instructions for executing a sample test using both
PBS and the interactive job option
* applicable module and/or environment
information (if required)
* sample scripts for submitting job types using the
software
* additional resources and references available
Contents: abaqus
cobalt
gaussian
gaussian_09
gaussian_linda_09
ls-dyna
matlab
nwchem
*****************************************************************************
Directory: dataManagement
Description: This directory contains information about data management
techniques including use of the data queue, storing files in
$WORKDIR, $ARCHIVE, and proper data retrieval methods.
Contents: storageManagement
storagePolicies
usingArchive
*****************************************************************************
Directory: debugging
Description: This directory contains basic information on how to start up
and use the available debuggers on midnight.
Contents: totalview
*****************************************************************************
Directory: jobSubmission
Description: This directory contains sample PBS batch scripts
and helpful commands for monitoring job progress.
Examples include options to submit a jobs such as
declaring which group membership you belong to
(for allocation accounting), how to request a particular
software license, etc.
Contents: dataStaging
MPI_OpenMP_scripts
MPI_scripts
OpenMP_scripts
*****************************************************************************
Directory: libraries
Description: This directory contains a summary of the various libraries
available and examples of how to compile a program linking to
those libraries. Example source code, Makefiles,
and data files may be included in this directory.
Contents: scalapack
papi
*****************************************************************************
Directory: parallelEnvironment
Description: This directory contains sample code and scripts containing
compiler options for common parallel programming practices
including code profiling.
Contents: auto_tau_pi
autoloops_tau_pi
hello_world_mpi
manual_tau_pi
tau_papi_counters
*****************************************************************************
"samples_home"
Last Updated: Wed, 23 Dec 2009 - ARSC Help DeskMachines: pingo
Baseline Configuration Sample Code Repository
=============================================
Filename: INDEX.txt
Description: This file contains the name,location, and brief
explanation of "samples" included in this Sample
Code Repository. There are several subdirectories within
this code repository containing frequently-used procedures,
routines, scripts, and code used on this allocated system,
midnight. This sample code repository can be
accessed from pingo by changing directories to
$SAMPLES_HOME, or changing directories to the following
location: pingo1% /usr/local/pkg/samples. This particular
file can be viewed from the internet at:
http://www.arsc.edu/support/news/systemnews/pingonews.xml#samples_home
+ AVAILABLE_TOOLS.txt (list of locally developed utilities)
Directory Description
===================== =======================================================
applications This directory includes example uses of the software
packages available on pingo
+ cobalt
+ gaussian
+ gaussian_09
+ namd
+ nwchem
-----------------------------------------------------------------------------
dataManagement This directory contains information about data
management techniques including use of the data queue,
storing files in $WORKDIR, $ARCHIVE, and proper data
retrieval methods.
+ storageManagement
+ storagePolicies
-----------------------------------------------------------------------------
debugging This directory contains basic information on how to
start up and use the available debuggers on pingo.
-----------------------------------------------------------------------------
jobSubmission This directory contains sample PBS batch scripts
and helpful commands for monitoring job progress.
Examples include options to submit a jobs such as
declaring which group membership you belong to
(for allocation accounting), how to request a
particular software license, etc.
+ dataStaging
+ MPI_OpenMP_scripts
+ MPI_scripts
+ OpenMP_scripts
-----------------------------------------------------------------------------
libraries This directory contains a summary of the various
libraries available and examples of how to compile a
program linking to those libraries. Example source
code, Makefiles, and data files may be included in this
directory.
+ scalapack
+ papi
-----------------------------------------------------------------------------
parallelEnvironment This directory contains sample code and scripts
containing compiler options for common parallel
programming practices including code profiling.
+ hello_world_mpi
*****************************************************************************
"software"
Last Updated: Tue, 17 Feb 2009 - Oralee NudsonMachines: pingo
Software
========
ferret: ferret-6.3 (2010-02-12)
ferret 6.3 is now available on pingo via the "ferret/6.3" module
nco: nco-3.9.9 (2009-11-11)
nco 3.9.9 is now available on pingo and can be loaded with
the following module command:
module load nco/3.9.9
git: git-1.5.6 (2009-10-29)
git 1.5.6 is now available on pingo, and a module has been
created for use (git/1.5.6)
paraview: paraview-3.4.0 (2009-10-13)
paraview 3.4.0 is now available on pingo and can be loaded with
the following module command:
module load paraview/3.4.0
ncl: ncl-5.1.1 (2009-08-05)
ncl 5.1.1 is now available on pingo and can be loaded with
the following module command:
module load ncl/5.1.1
subversion: subversion-1.6.3 (2009-07-31)
The latest version of subversion is available via the
subversion/1.6.3 module.
gaussian: gaussian-09.A.01 (2009-07-31)
Gaussian 09 vA.01 is now available in /usr/local/pkg/gaussian.
Examples are located in:
$SAMPLES_HOME/applications/gaussian_09
git: git-1.6.1.3 (2009-07-28)
git 1.6.1.3 is now available on pingo, and is located in
/usr/local/pkg/git/git-1.6.1.3.gnu.snl It is also available
by loading the module, with the following:
module load git/1.6.1.3
ncl: ncl-5.1.0 (2009-07-21)
ncl 5.1.0 is now available on pingo and can be loaded with
the following module:
module load ncl/5.1.0
python: python-2.6.2 (2009-05-20)
Python 2.6.2 is now available on pingo with the following
modules:
basemap-0.99.3, gnuplot-1.8, Imaging-1.1.6,
matplotlib-0.98.5.2, netCDF4-0.8, numpy-1.3.0,
pycdf-0.6-3b, pytools-9, scipy-0.7.0
It is available by running the following:
module load python/2.6.2
python
geos: geos-2.2.3 (2009-05-19)
GEOS is an implementation of GIS in C++. It is located in
/usr/local/pkg/geos/geos-2.2.3.gnu
cobalt: cobalt-4.2.1 (2009-05-01)
Version 4.2.1 of cobalt is now available on pingo. You must
be in the cobaltcg group to access this application. An
example cobalt job is available in:
$SAMPLES_HOME/applications/cobalt
mpt: mpt-3.1.1 (2009-04-21) (Upgrade from 3.0.2 on 2009-04-29)
The default MPT will be upgraded to 3.1.1 on April 29th during
system maintenance. This will not affect previously compiled
applications, but will affect future compiles using the default
compiler environment. This fix should resolve a memory leak related
to user defined types. Users can access the old MPT version by running
"module switch mpt mpt/3.0.2" if you experience problems with the
new default version.
gaussian: gaussian-03.E.01 (2009-04-10)
Gaussian 03 is now available. It can be accessed in the following
directory:
/usr/local/pkg/gaussian/gaussian-03.E.01
This software is group controlled. Please contact consult@arsc.edu
if you are interested in using it.
hdf: hdf-5-1.8.1 (2009-04-03)
HDF 5-1.8.1 is now available with shared libraries for the GNU
compiler. It is located in /usr/local/pkg/hdf/hdf-5-1.8.1.gnu.gnl
nwchem: nwchem-5.1.1 (2009-03-30)
The latest version of NWChem has been installed on pingo.
Contact user support to get access to this package.
An example PBS Script and job file are available in
$SAMPLES_HOME/applications/nwchem
namd: namd-2.7b1 (2009-03-30)
NAMD 2.7b1 has been installed on pingo in the directory:
/usr/local/pkg/namd/namd-2.7b1/NAMD_2.7b1_Source/CRAY-XT-g++/
An example PBS Script and job file are available in
$SAMPLES_HOME/applications/namd
netcdf: netcdf-3.6.3 (2009-03-23)
NetCDF 3.6.3 is now available for Pathscale compilers on pingo.
The libraries are available in:
/usr/local/pkg/netcdf/netcdf-3.6.3.path
mpscp: mpscp-1.3a (2009-03-17)
mpscp 1.3a is now available. This package has also been made
the default version on the system, and is available as either mpscp
or in /usr/local/pkg/mpscp/mpscp-1.3a
silo: silo-4.6.2.gnu.snl (2009-02-17)
The silo software is now available on pingo
and is located in the following directory:
/usr/local/pkg/silo/silo-4.6.2.gnu.snl
Please note that silo has been installed with
pingo login node support. Compute node
applications using the silo library may not
operate correctly.
"software"
Last Updated: Tue, 17 Feb 2009 - Alec BennettMachines: linuxws
Software on Linuxws
========================================
gsl-1.13: GNU Scientific Library 1.13 (2010-01-14)
The newest version of GSL is now available. This version
was compiled using the GNU compiler and is now available
on the Workstations in the following location:
/usr/local/pkg/gsl/gsl-1.13/
pgi: pgi-9.0.4 (2010-01-11)
The pgi 9.0.4 compiler is now available via the
pgi-9.0.4 module.
paraview: paraview-3.6.2 (2010-01-11)
Paraview 3.6.2 is now available via the paraview-3.6.2
module.
vapor: vapor-1.5.0 (2009-10-21)
Vapor 1.5.0 is now available via the vapor-1.5.0 module.
paraview: paraview-3.6.1 (2009-10-20)
Paraview 3.6.1 is now available via the paraview-3.6.1
module.
idv: idv-2.7u2 (2009-10-20)
idv 2.7 update 2 is now available via the idv-2.7u2 module.
subversion: subversion-1.6.3 (2009-09-03)
The latest version of subversion is available via the
subversion-1.6.3 module.
google-earth: google-earth-5.0 (2009-08-31)
Google Earth 5.0 is now available on the Workstations in the
following location:
/usr/local/pkg/google-earth/google-earth-5.0
ncl: ncl-5.1.1 (2009-08-05)
The latest version of ncl is now available via the "ncl-5.1.1"
module.
git: git-1.6.1.3 (2009-07-28)
git 1.6.1.3 is now available. This package is available
by loading the "git-1.6.1.3" module.
abaqus: Abaqus 6.9 (2009-07-09)
Abaqus 6.9 is now available. This package is available
by loading the "abaqus-6.9" module.
matlab: matlab-7.8.0 (2009-06-19)
Matlab 7.8.0 is now available. A module is
available for this software and can be loaded with:
module load matlab-7.8.0
After loading the module, type 'matlab' at the prompt
to open a new matlab session.
cmake: cmake-2.6.4
The latest version of cmake is available via the cmake-2.6.4
module.
blender: blender-2.48a
Blender 2.48a is now available on the Workstations. It is
available in a module, as blender-2.48a
avizo: avizo-6.0 (2009-04-30)
Avizo 6.0 is now available in the following directory:
/usr/local/pkg/avizo/avizo60
visit: visit-1.11.2 (2009-04-29)
VisIt 1-11.2 is now available via the "visit-1.11.2" module.
ncl: ncl-5.1.0 (2009-04-29)
The latest version of ncl is now available via the "ncl-5.1.0"
module.
paraview: paraview-3.4.0 (2009-03-10)
Paraview 3.4.0 is now available. It can be accessed with the
paraview-3.4.0 module file and is located in the
/usr/local/pkg/paraview/paraview-3.4.0 directory.
visit: visit-1.11.1 (2009-03-02)
VisIt is now available in /usr/local/pkg/visit/visit-1.11.1
Module files are available as both "visit" and "visit-1.11.1"
comsol: comsol-3.5a (2009-02-06)
The newest version of comsol is now available in
/usr/local/pkg/comsol/comsol-3.5a. This version appears
to resolve the previous errors when starting the software.
matlab: matlab-7.7.0 (2009-02-05)
The latest version of Matlab is available for
use by loading the matlab-7.7.0 module.
matlab: matlab-7.6.0 (2008-07-24)
The latest version of Matlab is available for
use by loading the matlab-7.6.0 module.
paraview: paraview-3.0.2 (2007-09-19)
Paraview version 3.0.2 has been installed into
/usr/local/pkg/paraview/paraview-3.0.2. It is
available via a module (paraview-3.0.2).
acml: acml-3.6.0 & acml-4.0.0 AMD Core Math Library (2007-09-14)
The ACML has been installed and is available in
/usr/local/pkg/acml. The following versions were installed:
acml-3.6.0.gcc
acml-3.6.0.pgi
acml-4.0.0.gcc
These libraries are available as of Sep 14th, 2007 and
the current link was set to point to acml-3.6.0.gcc.
idv: idv-2.2: Integrated Data Viewer (2007-07-28)
The new version of idv(2.2) has been installed
in /usr/local/pkg/idv/idv-2.2 and will be made
the default version on July 12th, 2007.
"software"
Last Updated: Fri, 04 Jan 2008 - Donald BahlsMachines: seawolf
Software on Seawolf
========================================
mpscp: mpscp-1.3a (2009-03-20)
The default version of mpscp was updated from 1.2b
to 1.3a.
mpscp: mpscp-1.2b (2008-01-04)
On January 7th, 2008 the default version of mpscp on
seawolf will be upgraded from version 1.2 to 1.2b.
"software"
Last Updated: Tue, 17 Feb 2009 - Alec BennettMachines: midnight
Software on Midnight
========================================
nco: nco-3.9.9 (2009-11-11)
nco 3.9.9 is now available via the nco-3.9.9 module.
gsl: gsl 1.13 (2009-11-05)
The newest version of GSL is now available. This version
was compiled using the GNU compiler and is located
in /usr/local/pkg/gsl/gsl-1.13.gcc/
pgi: PGI 9.0.4 (2009-10-08)
The latest version of the PGI compiler is available for
serial or parallel codes by loading the PrgEnv.pgi-9.0.4
module.
subversion: subversion-1.6.3 (2009-09-02)
The latest version of subversion is available via the
subversion-1.6.3 module.
ncl: ncl-5.1.1 (2009-08-05)
ncl 5.1.1 is now available (without OpenDAP support). This
software is accessible by loading the ncl-5.1.1 module.
gaussian: gaussian-09.A.01 (2009-07-14)
Gaussian 09 vA.01 is now available in /usr/local/pkg/gaussian.
Examples are located in:
$SAMPLES_HOME/applications/gaussian_09
$SAMPLES_HOME/applications/gaussian_linda_09
git: git-1.6.1.3 (2009-07-09)
git-1.6.1.3 is now available on midnight and is available
in /usr/local/pkg/git/git-1.6.1.3
abaqus: Abaqus 6.9 (2009-07-09)
Abaqus 6.9 is now available. This package is available
by loading the "abaqus-6.9" module.
matlab: matlab-7.8.0 (2009-06-17)
Matlab 7.8.0 is now available. A module is
available for this software and can be loaded with:
module load matlab-7.8.0
After loading the module, type 'matlab' at the prompt
to open a new matlab session.
pgi: PGI 8.0.6 (2009-06-10)
The latest version of the PGI compiler is available for
serial or parallel codes by loading the PrgEnv.pgi-8.0.6
module.
nwchem: nwchem-5.1.1 (2009-05-29)
The latest release of nwchem is now available on midnight.
Contact user support to request access to this package.
netcdf: netcdf-4.0.1 (2009-05-21)
netCDF 4.0.1 is now available, located in:
/usr/local/pkg/netcdf/netcdf-4.0.1.gcc.shared
python: python-2.6.2 (2009-05-20)
Python 2.6.2 is now available on pingo with the following
modules:
basemap-0.99.3, gnuplot-1.8, Imaging-1.1.6,
matplotlib-0.98.5.2, netCDF4-0.8, numpy-1.3.0,
pycdf-0.6-3b, pytools-9, scipy-0.7.0
It is available by running the following:
module load python-2.6.2
python
geos: geos-2.2.3 (2009-05-20)
GEOS is an implementation of GIS in C++. It is located in
/usr/local/pkg/geos/geos-2.2.3.gcc
ncl: ncl-5.1.0 (2009-04-09)
ncl 5.1.0 with OPeNDAP support is now available. This software
is accessible by loading the ncl-5.1.0 module.
boost: boost-1.38.0 (2009-04-09)
boost 1.38.0 is now available. Boost is an extension to C++ and
includes a number of high levels libraries. It is available in
/usr/local/pkg/boost/boost-1.38.0
curl: curl-7.19.4 (2009-04-09)
curl 7.19.4 is now available in /usr/local/pkg/curl/curl-7.19.4.
This install is to supprt NCL 5.1, and may not have all
functionality included in the default system curl package.
libnc-dap: libnc-dap-3.7.3 (2009-04-08)
libnc-dap has been installed in order to support NCL OPeNDAP
support. It is located in:
/usr/local/pkg/libnc-dap/libnc-dap-3.7.3.gcc
libdap: libdap-3.8.2 (2009-04-08)
libdap has been installed in order to support NCL OPeNDAP
support. It is located in:
/usr/local/pkg/libdap/libdap-3.8.2.gcc
fftw: fftw-2.1.5 w/ shared libs / OpenMP (2009-04-02)
FFTW 2.1.5 has been installed with additional support
added for OpenMP and Shared Libraries for Pathscale. This
version is available in the following directory:
- /usr/local/pkg/fftw/fftw-2.1.5.path.shared
mpscp: mpscp-1.3a (2009-01-28)
mpscp has been updated to the newest available version.
mexnc: mexnc-2.9.4 (2008-12-19)
The mexnc software is now available for use with
matlab, and is located in the /usr/local/pkg/mexnc
directory.
To use this version of mexnc, "module load matlab-7.7.0"
then enter the following at the matlab command prompt:
"addpath /usr/local/pkg/mexnc/mexnc-2.9.4/mexcdf/mexnc"
matlab: matlab-7.7.0 (2008-12-19)
Matlab 7.7.0 is now available. A module is
available for this software and can be loaded with
module load matlab-7.7.0
and running 'matlab'
mpscp: mpscp-1.3a (2008-12-15)
The latest version of mpscp is available. To access this
version run "module load mpscp-1.3a".
pgi: PGI 8.0.1 (2008-12-12)
The latest version of the PGI compiler is available for
serial or parallel codes by loading the PrgEnv.pgi-8.0.1
module.
show_queues: (2008-12-08)
The show_queues command is now available on midnight.
mexnc: mexnc-2.0.31-2 (2008-10-01)
The mexnc software is now available for use with
matlab, and is located in the /usr/local/pkg/mexnc
directory.
python: Python 2.5.2 (2008-09-05)
Python 2.5.2 is available with additional modules:
matplotlib,numpy,PIL,pycdf,scipy
This package is available by loading the python-2.5.2
module.
abaqus: Abaqus 6.8 (2008-07-28)
Abaqus 6.8 is now available. This package is available
by loading the "abaqus-6.8" module.
gnuplot: gnuplot-4.2.3 (2008-07-25)
GNUPlot is now available. The executables is located
in /usr/local/pkg/gnuplot/gnuplot-4.2.3/bin as gnuplot.
pgi: PGI 7.2.2 (2008-07-24)
The latest version of the PGI compiler is available for
serial or parallel codes by loading the PrgEnv.pgi-7.2.2
module.
pathscale: PathScale 3.2 (2008-07-23)
The latest version of the PathScale compiler has been
installed. Use the "PrgEnv.path-3.2" module to access this
environment.
git: git-1.5.6 (2008-07-10)
git (revision control) has been installed on midnight.
This software is available in /usr/local/pkg/git
matlab: matlab-7.6.0 (2008-07-10)
Matlab 7.6.0 is now available. A module is also
available for this software and can be loaded with
module load matlab-7.6.0
and running 'matlab'
mexnc: mexnc-2.0.29 (2008-07-01)
The mexnc software is now available for use with
matlab, and is located in the /usr/local/pkg/mexnc
directory.
cobalt: cobalt-4.2.1 (2008-06-25)
Cobalt 4.2 was rebuilt to include HDF5 support. This
version is otherwise the same as cobalt-4.2
ncl: ncl-5.0.0 (2008-06-19)
The latest release of the NCAR Command Language is
now available on midnight. To use the latest release
run:
module load ncl-5.0.0
This version includes both NCL and NCAR Graphics.
idv: idv-2.5 (2008-06-19)
The Integrated Data Viewer (IDV) has been installed on
midnight in /usr/local/pkg/idv/idv-2.5 This software is
available by loading the idv module with the following:
module load idv-2.5
and then running the command "IDV".
pgi: PGI 7.1.6 (2008-05-05)
The latest version of the PGI compiler is available for
serial or parallel codes by loading the PrgEnv.pgi.new
module (or PrgEnv.pgi-7.1.6).
cobalt 4.2:
The latest release of cobalt is now available on midnight.
NCAR Graphics 4.4.1 (PathScale):
The PathScale version of NCAR Graphics library is now
available in /usr/local/pathscale/lib.
ezViz:
ezViz has been available on midnight for some time
now. There is now a module available which will
add the ezViz tools to your PATH. To load ezViz,
run "module load ezViz". For more informations on
ezViz, see:
https://visualization.hpc.mil/wiki/EzViz_Batch_Mode
gaussian: gaussian-03.E.01 & gaussview-4.1 (2008-03-03)
Gaussian 03 vE.01 is now available in
/usr/local/pkg/gaussian. Additionally, GaussView
has been installed in /usr/local/pkg/gaussview and
is available by running "module load gaussview" and
"gview". This software is group controlled, but please
contact consult@arsc.edu if you would like to use this
software. Also, there is now a sample file in the
$SAMPLES_HOME/applications/gaussian directory.
totalview: totalview-8.4.0-0 (2008-02-26)
The latest version of the totalview debugger is now
available via the modules environment. To use the
latest version run "module load totalview-8.4.0-0"
mpscp: mpscp-1.2b (2008-01-04)
During scheduled maintenance on January 9, 2008, the
default version of mpscp on midnight will be upgraded
from version 1.2 to 1.2b.
pgi: PGI 7.0.2 (2007-12-05)
The PGI compilers have been installed on midnight.
Additionally, the following packages have been built to
support this compiler suite:
acml-3.6.0, blacs-1.1.3, fftw-2.1.5, fftw-3.1.2,
hdf-4.2r1, ncarg-4.4.1, netcdf-3.6.1, scalapack-1.8.0
Additionally, modules "PrgEnv.pgi" and "PrgEnv.pgi-7.0.2" are
available for ease of use.
pathscale: PathScale 3.1 (2007-11-19)
The latest version of the PathScale compiler has been
installed. Use the "PrgEnv.path-3.1" module to access this
environment.
ncl-4.3.1: NCAR Command Language 4.3.1 (2007-08-28)
New version of ncl is available via the "ncl-4.3.1" module.
sunstudio: Sun Studio 2007-06 (2007-09-14)
The latest release of the Sun Studio Compiler suite
is available as of Sept 14, 2007. The new version is
available via the "PrgEnv.sun-2007-06" module.
pathscale: PathScale (2.5 & 3.0) (2007-08-09)
The PathScale compiler defaults file will be installed
and active as of Aug 23rd at 8:00am AST. This sets defaults
relating to include/lib paths as well as search path, and
shared library paths. This should only affect new builds,
and should not impact executables and objects compiled
prior to Aug 23rd, 2007.
ncl-4.3.0: NCAR Command Language 4.3.0 (2007-06-27)
New version of ncl is available via the "ncl-4.3.0" module.
svn: Subversion 1.4.3 installed (2007-06-01)
subversion (svn) is a versioning control systems. See
'man svn' or 'svn help' for more information. This
version was built with SSL enabled to allow access to
https servers as well as standard http and ssh.
"ssh"
Last Updated: Wed, 11 Jul 2008 - Donald BahlsMachines: seawolf
SSH Allowed Versions Change ======================================== The following two releases of the kerberized ssh packages will be disallowed after July 28, 2008: OpenSSH_4.7p1b and OpenSSH_5.0p1a These versions will no longer be allowed to connect to ARSC systems after this date. You can run "ssh -V" to determine the version of ssh on your system. E.g.: iceberg % ssh -V OpenSSH_5.0p1b, OpenSSL 0.9.8h 28 May 2008 Version OpenSSH_5.0p1b of the ssh kit is available here: https://www.hpcmo.hpc.mil/security/kerberos/
"storage"
Last Updated: Fri, 19 Jun 2009 - Oralee NudsonMachines: pingo
Pingo Storage
=============
Long Term Storage Use
======================
batch_stage:
------------
Files saved in $ARCHIVE_HOME can potentially be offline (i.e. not
on disk). When accessing multiple files in $ARCHIVE_HOME, the
"batch_stage" can significantly speed the process of retrieving
files from tape.
e.g.
cd $ARCHIVE_HOME/somedirectory
find . -type f | batch_stage -i
See "man batch_stage" for additional examples.
/usr/bin/rcp:
-------------
While $ARCHIVE_HOME is available as an NFS filesystem, higher
transfer rates can be obtained by using the "rcp" command for
large transfers to and from $ARCHIVE_HOME.
The non-kerberosized version of rcp may be used to transfer files to
$ARCHIVE_HOME using the "seanfs" hostname.
e.g.
/usr/bin/rcp results.tar "seanfs:$ARCHIVE_HOME"
NOTE: The full path to rcp (i.e. /usr/bin/rcp) must be used to
make transfers without a ticket.
See http://www.arsc.edu/support/howtos/storage.html for more information
on storage best practices at ARSC.
$HOME quotas
============
On July 6th, 2009 ARSC will begin enforcing $HOME quotas on pingo.
The default $HOME quota will be set to 1GB.
If your current $HOME disk usage is larger 1 GB, we will be contacting
you directly to request that you either delete or move data to the long
term $ARCHIVE_HOME storage server to reduce the overall size of your $HOME.
If you require an exception to this 1GB $HOME quota, please communicate
your specific circumstances to the ARSC Consultants and we will work with
you to find a solution.
Please contact User Support if you have any questions.
"storage"
Last Updated: Mon, 19 Jun 2006 - Donald BahlsMachines: linuxws
Linux Workstation Storage
========================================
The environment variables listed below represent paths. They are
expanded to their actual value by the shell, and can be used in commands
(i.e. ls $ARCHIVE). From the command prompt the value and the variable
are usually interchangeable. However, in non-shell settings like ftp
you will need to use the actual path, not the variable.
In the listing below, $USER is an environment variable holding your
ARSC username.
Filesystem Purpose Purged Backed Up Quota
------------- ---------------------- ------- ---------- ------
$HOME shared filesystem No Yes 256 MB
$WRKDIR temp filesystem Yes No None (1)
$WORKDIR
$SCRATCH
$ARCHIVE long term storage No Yes None
$ARCHIVE_HOME
NOTES:
(1) Use is limited by the available space on the disk.
Environment Variable Definitions
=================================
Variable Definition
-------------- ---------------------
$HOME /u1/uaf/$USER or
/u2/wes/$USER or
/u2/red/$USER
$WRKDIR /scratch/$USER
$WORKDIR /scratch/$USER
$SCRATCH /scratch/$USER
$ARCHIVE /archive/$HOME
$ARCHIVE_HOME /archive/$HOME
-- Home directories are intended primarily for basic account info
(e.g. dotfiles). Please use $WRKDIR (your /scratch/$USER directory)
for compiles, inputs, outputs, etc.
* The 'qcheck' command will show quota information for your $HOME
directory. This command requires that you have an account on
nanook, please contact the help desk should you need your account
activated.
* "qcheck -k" will show quota information in kilobytes rather than
blocks.
-- The $WRKDIR or $SCRATCH directories are local to each machine. (On the
Linux Workstations these two variables both point to the same
location.) When moving to another machine you will also need to move
your files. This file system not backed up, files not accessed in
over 30 days are purged (deleted).
-- Your $SCRATCH directory is not created by default. If one does
not exist on the machine you are using, type 'mkdir $SCRATCH' to
create one.
-- Purging: Files not accessed in over 30 days in $WRKDIR ($SCRATCH)
directories are purged, and these directories are not backed up.
Please store what you want to keep in $ARCHIVE.
-- Long-term backed up storage is only available in your $ARCHIVE
directory. No other directories are backed up. As this is an
NFS-mounted filesystem from nanook, files will be temporarily
unavailable when nanook goes down for maintenance. I/O performance
in this directory will be much slower. Compiles and runs in $ARCHIVE
are not recommended.
See http://www.arsc.edu/support/howtos/storage.html for more information
on storage policies at ARSC.
"storage"
Last Updated: Thu, 12 Apr 2007 - Donald BahlsMachines: midnight
Midnight Storage
========================================
The environment variables listed below represent paths. They are
expanded to their actual value by the shell, and can be used in
commands (i.e. ls $ARCHIVE_HOME). From the command prompt, the
expanded path and the variable are usually interchangeable. However,
in non-shell settings like ftp, you will need to use the actual path,
not the variable.
In the listing below, $USER is an environment variable holding
your ARSC username.
Filesystem Purpose Quota
------------------ ------------------------ -----------
$HOME dotfiles, sm. files 500 MB
/u1/uaf/$USER
/u2/wes/$USER
/u2/red/$USER
$WORKDIR do work here 100 GB (Initial)
/wrkdir/$USER (Purged)
$ARCHIVE_HOME long-term remote storage no quota
/archive/$HOME
$SCRATCH local drive each node no quota
/scratch/$USER (Purged)
-- $HOME: Home directories are intended primarily for basic account
info (e.g. dotfiles). Please use $WORKDIR (your /wrkdir/$USER
directory) for compiles, inputs, outputs, etc. Files in the
$HOME are backed up periodically. Quotas are enabled on this
filesystem. Use the command "quota -v" to show your current
$HOME use.
-- $ARCHIVE_HOME: Long-term backed up storage is only available in
your $ARCHIVE_HOME directory. As this is an NFS-mounted
filesystem from seawolf, files will be temporarily unavailable
when seawolf goes down for maintenance. I/O performance in this
directory will be much slower. Compiles in $ARCHIVE_HOME are not
recommended. $ARCHIVE_HOME is not available from compute nodes.
The "data" queue provides access to $ARCHIVE_HOME for copying
data to and from $WORKDIR. See the "Long Term Storage Use"
section below for additional Long Term Storage recommendations.
-- $WORKDIR: Short term, not backed up, purged filesystem. This is
a large fast local disk. The $WORKDIR filesystem is available
to all nodes on midnight. This is the recommended location
for input, output, and temporary files. The $ARCHIVE_HOME
filesystem is available for long term storage.
-- $SCRATCH: drives are local to each node. For example, the
$SCRATCH on midnight1 is only available to that system and is
not available to any other compute node in the midnight cluster.
You must copy any data you wish to keep from $SCRATCH to $WORKDIR
at the completion of your job. When possible $WORKDIR should
be preferred over $SCRATCH as files in $WORKDIR are available
from all compute nodes.
The $SCRATCH directory is not created by default. If you wish to
use $SCRATCH, you must create the directory on each compute node
that you will be using it on. Please contact the ARSC Help desk
for further details.
Long Term Storage Use
======================
batch_stage:
------------
Files saved in $ARCHIVE_HOME can potentially be offline (i.e. not
on disk). When accessing multiple files in $ARCHIVE_HOME, the
"batch_stage" can significantly speed the process of retrieving
files from tape.
e.g.
cd $ARCHIVE_HOME/somedirectory
find . -type f | batch_stage -i
See "man batch_stage" for additional examples.
/usr/bin/rcp:
-------------
While $ARCHIVE_HOME is available as an NFS filesystem, higher
transfer rates can be obtained by using the "rcp" command for
large transfers to and from $ARCHIVE_HOME.
The non-kerberosized version of rsh may be used to transfer files to
$ARCHIVE_HOME using the "seanfs" hostname.
e.g.
/usr/bin/rcp results.tar "seanfs:$ARCHIVE_HOME"
NOTE: The full path to rcp (i.e. /usr/bin/rcp) must be used to
make transfers without a ticket.
See http://www.arsc.edu/support/howtos/storage.html for more information
on storage best practices at ARSC.
"support"
Last Updated: Thu, 07 Aug 2008 - Oralee NudsonMachines: linuxws midnight pingo seawolf
User Support Information ======================================== ARSC hosts accounts for two sets of users: HPCMP and academic. To receive the most prompt support, it is important to contact the appropriate support channel for your account. HPCMP users should contact the Consolidated Customer Assistance Center (CCAC) as their first point of contact. The ARSC Help Desk should be the the first point of contact for academic users. The support command will display the appropriate support email address and phone number for your account type: % support
"totalview"
Last Updated: Mon, 02 Jun 2008 - Donald BahlsMachines: midnight
Totalview
========================================
Totalview is available on midnight and can be used to debug MPI, OpenMP and
serial applications. Generally debugging should occur on compute nodes
through the use of an interactive PBS job. Totalview may be run on login
nodes to debug short serial applications or to inspect core files.
The instructions below are prefaced by a prompt corresponding to a system
name where the command should be run.
+ midnight% corresponds to a midnight login node
(i.e. midnight1 or midnight2).
+ midnight-compute% corresponds to a midnight compute node.
+ local% corresponds to the name of your local workstation.
I. Starting an interactive job with X11 forwarding enabled.
A) Log into midnight1 or midnight2 with X11 forwarding enabled.
local% ssh -X -Y username@midnight.arsc.edu
B) Run the "tunnelx" command to setup X11 forwarding for the
compute nodes.
midnight1% tunnelx
C) Start an interactive PBS job requesting the number of processors
required for your job.
# for an 8 task MPI job on 4way (i.e. X2200) nodes
midnight% qsub -l select=2:ncpus=4:node_type=4way -I
# for a 16 task OpenMP job on 16way (i.e. X4600) nodes
midnight% qsub -l select=1:ncpus=16:node_type=16way -I
# for a serial job
midnight% qsub -l select=1:ncpus=1:node_type=4way -I
When there are a sufficient number of nodes available, PBS will
start the job.
D) Once the interactive PBS job starts, rerun the "tunnelx"
command to initialize the X11 settings for your shell.
midnight-compute% tunnelx
Next, source the file ~/.tx_local which was generated by the tunnelx
command.
# bash/ksh syntax
midnight-compute% . ~/.tx_local
# csh/tcsh syntax
midnight-compute% source ~/.tx_local
You should be able to run X11 applications on the compute nodes.
E) For additional details on the "tunnelx" command see "man tunnelx"
II. Running totalview.
A) For MPI applications, start the application using the "-tv" and
"-timeout" flags.
midnight-compute% mpirun -tv -timeout 600 ./a.out
The "-tv" flag instructs mpirun to start the executable (a.out)
under the control of totalview. The "-timeout" flag sets the
maximum time the MPI environment should wait before generating
a timeout error. Setting the timeout limit makes it easier to
set up breakpoints, etc without totalview.
B) For OpenMP and serial applications, start the application using
totalview.
midnight-compute% totalview ./a.out
Additional hints:
1) Code should be compiled with -g. This makes it possible for
totalview to refer back to the source code. Code compiled without
-g will appear as assembly and you will not have meaningful access
to variable values.
2) You can view core files with totalview by passing the executable
and core file to totalview. A core file from an MPI application
can be viewed without using mpirun.
midnight% totalview ./a.out core.1234
3) The totalview command line version (i.e. totalviewcli) is also
available. This version can be used with MPI environment by
setting the TOTALVIEW environment variable before issuing the
mpirun command.
# bash/ksh syntax
midnight-compute% export TOTALVIEW=/usr/local/bin/totalviewcli
midnight-compute% mpirun -tv -timeout 600 ./a.out
# csh/tcsh syntax
midnight-compute% setenv TOTALVIEW /usr/local/bin/totalviewcli
midnight-compute% mpirun -tv -timeout 600 ./a.out
For more information, see http://www.totalviewtech.com
"totalview"
Last Updated: Tue, 20 Jan 2009 - Donald BahlsMachines: pingo
Totalview
========================================
Totalview is available on pingo and can be used to debug MPI, OpenMP
and serial applications. Generally debugging should occur on compute
nodes through the use of an interactive PBS job. Totalview may be
run on login nodes to debug short serial applications or to inspect
core files, however the majority of debugging work should occur on
compute nodes.
The instructions below are prefaced by a prompt corresponding to a
system name where the command should be run.
+ pingo4% corresponds to a pingo4.arsc.edu login node
+ pingo4-pbs% corresponds to a pingo4 node assigned by PBS
+ local% corresponds to the name of your local workstation.
NOTE: At this time, totalview is only supported on pingo4.arsc.edu.
I. Starting an interactive job with X11 forwarding enabled.
A) Log into pingo4 with X11 forwarding enabled.
local% ssh -X -Y username@pingo4.arsc.edu
B) Start an interactive PBS job requesting the number of processors
required for your job.
# for an 16 task MPI job
pingo4% qsub -l select=1:host=nid00075 -q debug -l mppwidth=16 -v DISPLAY -I
Here's a breakdown of this command:
a) -l select=1:host=nid00075
requests that the job be placed on pingo4 (a.k.a. nid00075)
b) -q debug
requests that the job run in the debug queue
c) -l mppwidth=16
requests 16 cores
d) -v DISPLAY
export the DISPLAY environment variable
e) -I
runs a PBS interactive job.
When there are a sufficient number of nodes available, PBS
will start the job. The default and maximum walltime for
"debug" queue jobs is 1 hour. Should you need to run a longer
debugging sessions, please use the "standard" queue or contact
User Support for additional assistance.
C) Once the interactive PBS job starts, run the "xt-totalview" module
if it's not already loaded.
pingo4-pbs% module load xt-totalview
II. Running totalview.
A) For MPI applications, start the application using the following:
pingo4-pbs% totalview aprun -a -n 16 ./a.out
This starts the aprun executable under the control of totalview.
The aprun executable will pass control of the application to
totalview. Here the "-a" flag indicates that all options after
that flag should be passed to "aprun" rather than totalview.
B) For OpenMP and serial applications, start the application using
totalview.
# csh/tcsh syntax
pingo4-pbs% setenv OMP_NUM_THREADS 8
pingo4-pbs% totalview aprun -a -n 1 -d 8 ./a.out
# sh/bash syntax
pingo4-pbs% export OMP_NUM_THREADS=8
pingo4-pbs% totalview aprun -a -n 1 -d 8 ./a.out
Additional hints:
1) Code should be compiled with -g. This makes it possible for
totalview to refer back to the source code. Code compiled without
-g will appear as assembly and may be difficult to debug.
2) You can view core files with totalview by passing the executable
and core file to totalview. A core file from an MPI application
can be viewed without using mpirun.
pingo% totalview ./a.out core.1234
For more information, see http://www.totalviewtech.com
Arctic Region Supercomputing Center
PO Box 756020, Fairbanks, AK 99775 | voice: 907-450-8600 | email: 
home | search | about | support | news | science | resources