![[Menu Bar]](/images/header.gif)
ARSC news items for midnight
Contents
- downtime
- file_striping
- filezilla
- kerberos
- matlab
- modules
- mpi
- pbs_10
- PET_HOME
- pin
- PrgEnv
- projects
- queues
- retirement
- samples_home
- software
- storage
- support
- totalview
News Items
"news downtime" on midnight
ARSC system downtime for midnight (all)
======================================
This file contains a description of the next scheduled downtime
for preventive maintenance, and the most recent system downtime
and/or network interruption. All times are Alaska Time. Contact
consult@arsc.edu or 907-450-8602 for more information.
============================================================
17 Mar 2010 Scheduled Downtime for midnight
----------------------------------------------------
Start Time: 03/17/2010 -- 09:00
End Time: 03/17/2010 -- 22:00
Reason: General system maintenance. PBS Pro will be upgraded
from version 10.0 to version 10.2.
08 Mar 2010 Unscheduled Downtime for linuxws midnight pingo seawolf
----------------------------------------------------
Start Time: 03/08/2010 -- 12:30
End Time: 03/08/2010 -- 13:30
Reason: Unscheduled file system problems on seawolf ($ARCHIVE)
06 Mar 2010 Unscheduled Downtime for midnight
----------------------------------------------------
Start Time: 03/06/2010 -- 20:15
End Time: 03/06/2010 -- 21:09
Reason: There was a hardware issue with the $WORKDIR file system.
01 Mar 2010 Unscheduled Downtime for midnight
----------------------------------------------------
Start Time: 03/01/2010 -- 18:45
End Time: 03/01/2010 -- 19:30
Reason: There was an issue with a component of the $WORKDIR file system.
01 Mar 2010 Unscheduled Downtime for midnight
----------------------------------------------------
Start Time: 03/01/2010 -- 12:00
End Time: 03/01/2010 -- 13:00
Reason: Unscheduled $WORKDIR problems
24 Feb 2010 Unscheduled Downtime for linuxws midnight pingo seawolf
----------------------------------------------------
Start Time: 02/24/2010 -- 11:00
End Time: 02/24/2010 -- 15:00
Reason: The controller for the seawolf silo is experiencing
unexpected problems. Admins are working to resolve the problem,
but there may be intermittent problems accessing and writing
to $ARCHIVE until it has been resolved.
17 Feb 2010 Scheduled Downtime for midnight
----------------------------------------------------
Start Time: 02/17/2010 -- 09:00
End Time: 02/17/2010 -- 22:00
Reason: General system maintenance.
10 Feb 2010 Scheduled Downtime for midnight pingo seawolf
----------------------------------------------------
Start Time: 02/09/2010 -- 18:00
End Time: 02/09/2010 -- 19:00
Reason: The $ARCHIVE_HOME file system for HPCMP users needs to be taken down
for repair. This affects users in the "u2" file system only. You
can determine if this outage will affect you by looking for "u2" in
the $ARCHIVE_HOME variable:
% echo $ARCHIVE_HOME
/archive/u2/wes/username
03 Feb 2010 Unscheduled Downtime for linuxws midnight pingo seawolf
----------------------------------------------------
Start Time: 02/03/2010 -- 09:20
End Time: 02/03/2010 -- 10:00
Reason: Network Issue -- There was an issue with the local router which
resulted in a loss of connectivity to systems outside of ARSC.
This outage did not affect running jobs.
22 Dec 2009 Scheduled Downtime for midnight pingo seawolf
----------------------------------------------------
Start Time: 12/22/2009 -- 20:00
End Time: 12/23/2009 -- 04:00
Reason: The DREN network connection will be unavailable for 30 minutes
during this downtime.
09 Dec 2009 Scheduled Downtime for midnight
----------------------------------------------------
Start Time: 12/09/2009 -- 15:00
End Time: 12/09/2009 -- 16:10
Reason: The $WORKDIR filesystem experienced a hang. Jobs running during
this outage should be reviewed to ensure the output looks correct.
Users with jobs running during the outage have been notified.
02 Dec 2009 Scheduled Downtime for midnight
----------------------------------------------------
Start Time: 12/02/2009 -- 08:30
End Time: 12/02/2009 -- 22:00
Reason: General system maintenance.
"news file_striping" on midnight
file_striping
========================
File Striping
========================================
The $WORKDIR Lustre filesystem on midnight is composed of multiple object
storage targets (OSTs). The default filesystem settings place each file on
one OST. These settings are appropriate for many applications, however some
applications may benefit by enabling striping. Striping places a file on
multiple OSTs. This can:
* Provide better bandwidth to files accessed by many tasks simultaneously.
* Increase read and write performance for I/O intensive codes.
* More evenly distribute large files across OSTs. Striping should be
considered for any file larger than 50 GB.
The following command shows how to enable striping across 6 OSTs for an empty
directory:
lfs setstripe $WORKDIR/case1 0 -1 6
NOTE: All new files created in the $WORKDIR/case1/ directory will take on
these striping attributes. Files moved within the same filesystem or files
that already exist within the directory will maintain their existing striping
characteristics.
See "lfs setstripe" for a description of striping parameters.
The following set of commands shows how to stripe a directory across 6 OSTs
and how to copy an existing large file into that directory (so the large file
will also be striped.)
cd $WORKDIR/case1
mkdir striped
lfs setstripe striped 0 -1 6
cp largeFile.txt striped/largeFile.txt
# confirm largeFile.txt data looks normal
rm largeFile.txt
To confirm the file striping was successful, check the particular file
or directory with the following commands:
lfs getstripe filename
lfs getstripe directoryname
-Oralee Nudson
Mon, 26 Jan 2009 09:49:30 -0800
"news filezilla" on midnight
filezilla
========================
Filezilla
=========
This news item is for users of the Filezilla ftp client which is
included with the HPCMP Kerberos kit for Windows.
Some users have reported problems connecting to Midnight
with the Filezilla client. Should you experience these problems
with Filezilla, try the following:
Option 1) Set the transfer mode to passive:
a) Select the menus "Edit | Settings"
b) Select the tree option "Firewall Settings "
c) Then check the passive mode check box
Option 2) Set the ftp protocol to sftp:
a) Select the menus "File | Site Manager"
b) Click "New Site" button. Enter the machine name.
c) Fill in the "host" (e.g. midnight.arsc.edu)
d) Set "Servertype" to "SFTP using SSH2"
e) Set "Logontype" to "Normal"
f) Set the "User" to your username.
g) Click the "Connect" button to connect.
Sites entered in the Site Manager can be also be selected using the
leftmost icon on the Filezilla toolbar.
If you experience problems with the Filezilla client using these
recommendations, please report them to the ARSC help desk
(consult@arsc.edu).
-Craig Stephenson
Thu, 14 May 2009 11:00:00 -0800
"news kerberos" on midnight
kerberos
========================
Required Kerberos Updates
========================================
Recently MIT announced two vulnerablities in Kerberos. Due to the
nature of these vulnerablities, ARSC and HPCMP are requiring all
kerberos clients to be updated to the latest release by May 22nd, 2009.
The following versions of Kerberos or newer will be the only supported
versions beyond May 22nd:
* HPCMP_RELEASE_20090331 (Linux / Unix / Mac OS X)
and
* HPCMP_RELEASE_20090422 (Windows).
Updated clients are available from the HPCMP Kerberos website.
Precompiled clients can be found under the "Software" link on the
left of this page.
https://www.hpcmo.hpc.mil/security/kerberos/
The known vulnerability also affects HPCMP OpenSSH client kits.
All versions older than OpenSSH-5.0p1c will not be allowed after
May 22nd. Updated versions of the HPCMP OpenSSH client kits are also
available at the HPCMP Kerberos website.
Frequently Asked Questions:
1) How can I tell which version of Kerberos I have installed?
a) For Linux, Solaris and Mac OS X:
# run the krlogin command with the -V option.
% krlogin -V
usage: rlogin host [-option] [-option...] [-k realm ] [-t ttytype] [-l username]
where option is e, 7, 8, noflow, n, a, x, f, F, c, PO, or PN
version: @(#) $NRL: HPCMP_RELEASE_20081006 20081006 $
The version of this kit is HPCMP_RELEASE_20081006.
b) For Windows:
Start the PuTTY client, then select the "About" button in the lower
left corner. The HPCMP_RELEASE_20090409 release should indicate
"PuTTY HPCMP Release 0.60" or newer is being used.
2) How can I check the version of ssh I have installed?
a) For Linux, Solaris and Mac OS X
% ssh -V
OpenSSH_5.2p1a, OpenSSL 0.9.8k 25 Mar 2009
b) For Windows
(See question 1.b above)
3) I'm not using Linux, Solaris, Mac OS X or Windows, how can I
obtain a kit for my Operating System?
Client kits for operating systems not listed above must be
compiled from source. Kerberos source code is available from:
https://www.hpcmo.hpc.mil/security/kerberos/
4) Are there a direct links to the client kits available?
Yes.
a) Linux and Solaris
https://www.hpcmo.hpc.mil/security/kerberos/software/clients/kerberos/index.html
b) Windows
https://www.hpcmo.hpc.mil/security/kerberos/software/clients/kerberos/indexWin.html
c) Macintosh
https://www.hpcmo.hpc.mil/security/kerberos/software/clients/kerberos/indexMac.html
-Donald Bahls
Wed, 22 Apr 2009 12:00:00 -0800
"news matlab" on midnight
matlab
========================
Matlab License Changes
========================================
Several changes to matlab licensing are occurring in order to provide
better support for licensed product through Advanced Reservations.
These changes will require users of matlab to make changes to their
normal workflow as described below.
This change in policy will officially go into effect on January
26th, 2010.
Jobs Run Through PBS:
=====================
On January 20th, 2010 a PBS resource will be added for matlab.
This resource will ensure that licenses are available prior to the
start of a PBS job. Matlab requires one license per node used by
the job using the matlab resource (e.g. #PBS -l matlab=1):
#!/bin/bash
#PBS -l walltime=1:00:00
#PBS -l matlab=1
#PBS -l select=1:ncpus=4:node_type=4way
#PBS -q standard
#PBS -j oe
cd $PBS_O_WORKDIR
matlab < input.m
Matlab Changes for Login Nodes:
================================
All matlab work must request license reservations via the
"matlab_license_check" script. This script requires that you specify
the length of time you intend to run matlab. Any work requesting
more than 1 hour should be run through PBS as described above.
e.g.
# Request a 60 minute reservation
mg56 % /usr/local/bin/matlab_license_check -w 60
Reservation successfully created!
module load matlab-7.8.0
matlab < input.m
Running Matlab On Compute Nodes:
=================================
Matlab can be run on compute nodes with full access to the GUI by using the
following technique:
A) Log into midnight1 or midnight2 with X11 forwarding enabled.
% ssh -X -Y username@midnight.arsc.edu
B) Run the "tunnelx" command to setup X11 forwarding for the
compute nodes.
midnight% tunnelx
C) Start an interactive PBS job requesting matlab:
midnight% qsub -l select=1:ncpus=1:node_type=4way -lmatlab=1 -I
D) Once the interactive PBS job starts, rerun the "tunnelx"
command to initialize the X11 settings for your shell.
mt101% tunnelx
Next, source the file ~/.tx_local which was generated by the tunnelx
command.
# bash/ksh syntax
mt101% . ~/.tx_local
# csh/tcsh syntax
mt101% source ~/.tx_local
E) You should be able to run matlab with X11 support.
mt101% module load matlab-7.6.0
mt101% matlab
F) When you are done using matlab, be sure to run "exit" to end the interactive
PBS job.
-Don Bahls
Wed, 20 Jan 2010 11:11:11 -0900
"news modules" on midnight
modules
========================
Using the Modules Package
=========================
The modules package is used to prepare the environment for various
applications before they are run. Loading a module will set the
environment variables required for a program to execute properly.
Conversely, unloading a module will unset all environment variables
that had been previously set. This functionality is ideal for
switching between different versions of the same application, keeping
differences in file paths transparent to the user.
Sourcing the Module Init Files
---------------------------------------------------------------------
For some jobs, it may be necessary to source these files, as they
may not be auto-sourced as with login shells.
Before the modules package can be used, its init file must first be
sourced.
To do this using tcsh or csh, type:
source /usr/share/modules/init/<shell>
To do this using bash, ksh, or sh, type:
. /usr/share/modules/init/<shell>
For either case, replace <shell> with the shell you are using.
If your shell is bash, for example:
. /usr/share/modules/init/bash
Once the modules init file has been sourced, the following commands
become available:
Command Purpose
---------------------------------------------------------------------
module avail - list all available modules
module load <pkg> - load a module file from environment
module unload <pkg> - unload a module file from environment
module list - display modules currently loaded
module switch <old> <new> - replace module <old> with module <new>
module purge - unload all modules
-Donald Bahls
Fri, 30 Oct 2008 09:10:46 -0800
"news mpi" on midnight
mpi
========================
MPI Environment
========================================
Midnight uses the Voltaire versions of the MVAPICH MPI stack. A list of
MVAPICH specific features is available on the ARSC webpage here:
http://www.arsc.edu/support/howtos/usingsun.html#mpi
Changes to MPI Stack
====================
The following are a list of changes to the MPI stack on midnight.
2008-05-19:
Task affinity is now enabled by default for MPI jobs. It can be disabled
by using "-noaff". See 2008-04-30 changes for details.
2008-04-30:
The mpirun command now includes an option to enable task affinity (-aff).
This option helps decrease the variability in run time for many applications,
however it should not be used for hybrid applications (e.g. MPI/OpenMP).
The "-noaff" option can be used do disable task affinity. This option should
be used for hybrid MPI/OpenMP or MPI/pthread applications.
Task affinity will be enabled for all MPI applications not specify "-noaff"
during maintenance on May 17-18th.
MPI Feature Support
===================
The following MPI features are not currently supported on midnight.
1) MPI-2 and MPI-IO. MPI-2 features including MPI-IO are not currently
available on midnight.
2) MPMD Support. The MVAPICH stack does not support the multiple program
multiple data (MPMD) programming model used by some codes (e.g. CCSM).
-Donald Bahls
Wed, 30 Apr 2008 09:55:06 -0800
"news pbs_10" on midnight
pbs_10
========================
PBS Pro 10 Upgrade
========================================
During scheduled maintenance on May 16th - 18th, PBS Pro will
be upgraded from version 7.1.3 to version 10.0. Most users will not
need to make changes to job scripts to ensure unaltered operation of
those scripts. There are several exceptions listed below.
1) Node Placement:
Jobs using 1/2 of the cores on a node or less will need to
include a placement statement to ensure the proper number of cores
per node are used.
e.g.,
# use 4 nodes with 2 cores per node
#PBS -l select=4:ncpus=2:node_type=4way
# place only one set of 2 tasks per node.
#PBS -l place=scatter:excl
...
2) Backfill:
The PBS scheduler will be updated to use job based priorities and
eligible time with backfill. Jobs will gain priority based on the time
the job is in the queue and eligible to run. If a job can run without
interfering with the start time of the highest priority job, that job
will be eligible for backfill. Jobs specifying shorter walltimes
and smaller processor counts generally have more opportunities to
take advantage of backfill windows. Significant overestimates in
walltime requests decrease the likelihood backfill can occur.
e.g.,
# Example of limits which may backfill nicely
#PBS -l select=4:ncpus=4:node_type=4way
#PBS -l walltime=2:00:00
...
3) Use of the -V flag:
Following the upgrade to PBS Pro 10, it was determined that the -V
option can cause PBS to work improperly under certain circumstances.
The following workarounds can be used:
a) Use the -v (small v) option to set individual environment variables
during job submission:
e.g.
qsub -v VARIABLE=value myscript.pbs
b) If the variable value is static, you can add the required
environment variables to your ~/.profile or ~/.login
c) Unset unneeded environment variables prior to submitting your PBS
script:
# bash/ksh
unset LM_LICENSE_FILE #unset license file string
qsub -V myscript.pbs
# csh/tcsh
unsetenv LM_LICENSE_FILE #unset license file string
qsub -V myscript.pbs
4) Script names not starting with a letter.
Script names not starting with a letter may not function properly
at runtime.
The following workarounds can be used:
Rename the script so the first character is a letter rather than a
number or symbol.
e.g.
mv 2test.pbs a2test.pbs
If you experience issues not described by this news item, please
contact User Support. The following command will provide contact
information:
% support
-Donald Bahls
Wed, 13 May 2009 09:57:07 -0800
"news PET_HOME" on midnight
PET_HOME
========================
PET-Computational Environment (CE)
==================================
The PET-Computational Environment (CE) consists of a number of packages and
libraries used in High Performance Computing. These tools are available in
the $PET_HOME directory on Midnight.
Current Versions in $PET_HOME
=============================
Package Name Version
ARPACK 96
CMake 2.6.4
FFTW-2 (sequential) 2.1.5
FFTW-3 (distributed) 3.1.3
GotoBLAS 1.26
GSL 1.12
HDF5 1.8.3
MatlabMPI 1.2
netCDF 4.0.1
Octave 3.0.5
PAPI 3.6.2
PETSc 2.3.3
ScaLAPACK 1.8.0
Scalasca 1.1
SPRNG 3.0
SuperLU 3.0
TAU 2.18.1
Trilinos 9.0.3
Valgrind 3.4.0
-Alec Bennett
Wed, 20 Jan 2010 15:36:07 -0900
"news pin" on midnight
pin
========================
New SecurID card PINs to be 5 digit
========================================
In order to improve the security and effectiveness of our authentication
system, ARSC will begin issuing 5 digit SecurID card PINs as of 1
May, 2007. Existing SecurID cards will NOT have their PINs updated
at this time. However, if a PIN reset is needed, the replacement
PIN will be 5 digit. All new SecurID cards issued after 1 May will
be set to 5 digit PINs. If you have any questions please contact
the ARSC Help Desk.
-Derek Bastille
Fri, 27 Aug 2007 09:58:48 -0800
"news PrgEnv" on midnight
PrgEnv
========================
Programming Environments on midnight
====================================
Compiler and MPI Library versions on midnight are controlled via
the modules package. New accounts load the "PrgEnv" module by
default. This module adds the Pathscale compilers and Voltaire
MPI compilers to the PATH.
Should you experience problems with a compiler or library in
many cases a new programming environment may be available.
Below is a description of available Programming Environments:
Module Name Description
=============== ==============================================
PrgEnv Programming environment using Pathscale
compilers and MPI stack (default version).
PrgEnv.new programming environment using the latest available
version of the Pathscale compilers.
PrgEnv.path Alternate name for PrgEnv
PrgEnv.path.new Alternate name for PrgEnv.new
PrgEnv.gcc Programming environment using GNU compilers
and MPI stack.
PrgEnv.pgi Programming environment using PGI compilers
and MPI stack.
PrgEnv.sun Programming environment using Sun Studio compilers
and MPI stack.
For a list of the latest available Programming Environments, run:
mg56 % module avail PrgEnv
--------------------- /usr/local/pkg/modulefiles ----------------
PrgEnv PrgEnv.new PrgEnv.pgi
PrgEnv.gcc PrgEnv.path PrgEnv.sun
PrgEnv.gcc.new PrgEnv.path.new PrgEnv.sun.new
ARSC Policy for Maintaining Programming Environment Module Versions
===================================================================
ARSC maintains a "current" programming environment, the last
programming environment and the most recently released version using
the "modules" facility. These policies explain the naming scheme
for the programming environment modules as well as the schedule for
changing the versions referenced by those modules.
- These policies pertain to the PathScale, PGI, and SunStudio compilers.
PathScale is our "official" compiler, hence it is pointed to by the
special names PrgEnv, PrgEnv.old and PrgEnv.new reflecting the
"current", "old" and "new" versions respectively. These are aliases for:
PrgEnv == PrgEnv.path
PrgEnv.old == PrgEnv.path.old
PrgEnv.new == PrgEnv.path.new
There will be "current", "old" and "new" versions for the other
compilers as well
PrgEnv.pgi, PrgEnv.pgi.old, PrgEnv.pgi.new
PrgEnv.sun, PrgEnv.sun.old, PrgEnv.sun.new
and in the discussion that follows, it is understood that policies that
apply to "PrgEnv", "PrgEnv.old" and "PrgEnv.new" apply to all three
compilers.
- The above module names are aliases for modules named by their version,
e.g., PrgEnv.path-3.0.
- When versions newer than "PrgEnv.new" are made available, we will
assign them a version-specific module name and notify users of their
availability. We will install new Programming Environment versions as
soon as practical, balanced against the benefits of the new version as
judged by ARSC staff.
- Under normal circumstances, i.e., no "extreme" need to act sooner, we
will rotate "PrgEnv", "PrgEnv.old" and "PrgEnv.new" no more frequently
than twice annually, on or about October 1 and April 1. This rotation
will be done only if "PrgEnv.new" is actually a newer version than
"PrgEnv" and only if "PrgEnv.new" has been available for testing for at
least one month. The rotation keeps the old version and updates the
current version as follows:
"PrgEnv.old" <-- "PrgEnv"
"PrgEnv" <-- "PrgEnv.new"
"PrgEnv.new" <-- a newer version (if it exists)
- Under extreme circumstances (e.g., a version that we cannot tolerate
being used or a significant system upgrade), ARSC staff may elect to
rotate these versions sooner or remove a version entirely.
- Insofar as it is feasible, we will maintain old versions (older than
"PrgEnv.old") as long as possible and will always maintain at least the
"PrgEnv.old" version.
- Any changes to "PrgEnv.old", "PrgEnv" and "PrgEnv.new" will be
preceded by at least two weeks notice to users.
Programming Environment Changes
================================
The following is a table of recent additions and changes to the
Programming Environment on midnight.
Date Module Name Description
---------- --------------------- -----------------------------------
2009-10-21 PrgEnv.path Fall updates to Programming
PrgEnv.path.old Environment.
PrgEnv.pgi.new
PrgEnv.pgi.old
PrgEnv.pgi
PrgEnv.pgi.new
2008-10-23 PrgEnv.path.new Fall updates to Programming
PrgEnv.path Environment.
PrgEnv.path.old
PrgEnv.pgi.old
PrgEnv.pgi
PrgEnv.pgi.new
2008-05-05 PrgEnv.pgi.new Latest release of the PGI compiler
PrgEnv.pgi-7.1.6 Suite.
2007-12-05 PrgEnv.pgi New Programming Environment using
PrgEnv.pgi-7.0.2 PGI Compiler Suite
2007-11-19 PrgEnv.path-3.1 New Programming Environment using
PathScale 3.1 compilers.
2007-09-10 PrgEnv.sun-2007-06 New Programming Environment using
Sun Studio 2007-06 compilers.
2009-10-21 Updates
===================
During system maintenance on October 21, 2009 the PathScale and
Portland Group Programming Environments will be updated as follows
Module Name Now Aliased To Previously Aliased To
--------------- ------------------- ----------------------
PrgEnv.path.old PrgEnv.path-3.1 PrgEnv.path-3.0
PrgEnv.path PrgEnv.path-3.2 PrgEnv.path-3.1
PrgEnv.pgi.old PrgEnv.pgi-7.1.6 PrgEnv.pgi-7.0.2
PrgEnv.pgi PrgEnv.pgi-7.2.2 PrgEnv.pgi-7.1.6
PrgEnv.pgi.new PrgEnv.pgi-8.0.6 PrgEnv.pgi-7.2.2
2008-10-23 Updates
===================
During system maintenance October 22, the PathScale and Portland Group
Programming Environments were updated as follows:
Module Name Now Aliased To Previously Aliased To
--------------- ------------------- ----------------------
PrgEnv.old PrgEnv.path.old PrgEnv.path-2.5
PrgEnv.path.old PrgEnv.path-3.0 PrgEnv.path-2.5
PrgEnv PrgEnv.path PrgEnv.path-3.0
PrgEnv.path PrgEnv.path-3.1 PrgEnv.path-3.0
PrgEnv.new PrgEnv.path.new PrgEnv.path-3.1
PrgEnv.path.new PrgEnv.path-3.2 PrgEnv.path-3.1
PrgEnv.pgi.old PrgEnv.pgi-7.0.2 nothing -- new module
PrgEnv.pgi PrgEnv.pgi-7.1.6 PrgEnv.pgi-7.0.2
PrgEnv.pgi.new PrgEnv.pgi-7.2.2 PrgEnv.pgi-7.1.6
-Donald Bahls
Wed, 22 Oct 2008 12:00:00 -0800
"news projects" on midnight
projects
========================
Instructions for Users with Multiple Projects
========================================
This news item is intended for users that are members of more than one
project. Users in a single project will automatically have use
charged against the allocation for their primary group (i.e. project).
Users in more than one project can select an alternate project to
charge use to by using the "-W group_list" PBS option. If the
"-W group_list" option is not specified the account number will default
to your primary group (i.e. project).
Below is an example "-W group_list" statement.
e.g.
#PBS -W group_list=proja
The "-W group_list" option can also be used on the command line.
e.g.
mg56 % qsub -Wgroup_list=proja script.bat
Each project has a corresponding UNIX group, therefore the groups
command will show all projects and other groups of which you are a
member.
e.g.
mg56 % groups
proja projb
In this case use would be charged to proja by default, but could be
charged to projb by setting "-W group_list=projb" in the PBS script.
If you have questions about this news item, please contact the ARSC
help desk (consult@arsc.edu).
'show_usage' Available
=======================
Project utilization information is now available via the BCT Compliant 'show_usage'
command. 'show_usage' with no parameters will display the remaining
allocation for each project of which you are a member.
e.g.
mg56 % show_usage
ARSC - Subproject Usage Information (in CPU Hours)
As of 01:11:21 hours ADT 11 Nov 2007
For Fiscal Year 2008 (01 October 2007 - 30 September 2008)
Percentage of Fiscal Year Remaining: 88.77%
Hours Hours Hours Percent Background
System Subproject Allocated Used Remaining Remaining Hours Used
========== ============== ========== ========== ========== ========= ==========
midnight projectA 100000.00 39663.48 60336.52 60.34% 0.00
midnight projectB 5000.00 0.00 5000.00 100.00% 0.00
Project utilization for all allocated systems can be obtained by using 'show_usage'
with the '-s' option as in 'show_usage -s'.
mg56 % show_usage -s
ARSC - Subproject Usage Information (in CPU Hours)
As of 13:00:00 hours ADT 11 Nov 2007
For Fiscal Year 2008 (01 October 2007 - 30 September 2008)
Percentage of Fiscal Year Remaining: 88.77%
Hours Hours Hours Percent Background
System Subproject Allocated Used Remaining Remaining Hours Used
========== ============== ========== ========== ========== ========= ==========
iceberg projectA 30000.00 309.84 29690.16 98.97% 0.00
iceberg projectB 10000.00 3873.46 6126.54 61.27% 0.00
midnight projectA 100000.00 39663.48 60336.52 60.34% 0.00
midnight projectB 5000.00 0.00 5000.00 100.00% 0.00
-Donald Bahls
Wed, 17 Dec 2008 12:00:00 -0800
"news queues" on midnight
queues
========================
Midnight Queues
===============
The queue configuration is as described below. It is subject to
review and further updates.
Login Nodes Use:
=================
Login nodes are a shared resource and are not intended for
computationally or memory intensive work. Processes using more
than 30 minutes of CPU time on login nodes may be killed by ARSC
without warning. Please use compute nodes for computationally or
memory intensive work.
Queues:
===============
Specify one of the following queues in your PBS qsub script
(e.g., "#PBS -q standard"):
Queue Name Purpose of queue
------------- ------------------------------
standard General use by all allocated users.
debug Quick turnaround queue for debugging work.
data Provides access to long term storage (i.e. $ARCHIVE_HOME)
background For projects with little or no remaining allocation.
This queue has the lowest priority, however projects
running jobs in this queue do not have allocation deducted.
The number of running jobs or processors available to this
queue may be altered based on system load.
special For jobs which do not fit into normal queue limits. (Limited Access)
high For jobs requiring higher priority than normal. (Special Access Only)
urgent For jobs requiring highest priority. (Special Access Only)
challenge For challenge projects only.
See 'qstat -q' for a complete list of system queues. Note, some
queues are not intended for general use.
Maximum Walltimes:
===================
The maximum allowed walltime for a job is dependant on the number of
processors requested. The table below describes maximum walltimes for
each queue.
Queue Min Max Max
Procs Procs Walltime Notes
--------------- ----- ----- --------- ------------
standard
1 16 84:00:00 See (A)
17 256 16:00:00
257 512 12:00:00
challenge
1 16 96:00:00 See (B)
17 256 48:00:00 See (F)
257 516 48:00:00 See (G)
background 1 512 12:00:00
debug 1 32 01:00:00 See (D)
data 1 1 12:00:00 See (E)
NOTES:
(A) The max walltime for standard_sm jobs (i.e. 1 to 16 processors) was
increased from 42 hours to 84 hours on October 5th 2007.
(B) The max walltime for challenge_sm jobs (i.e. 1 to 16 processors) was
increased from 48 hours to 96 hours on October 5th 2007.
(C) The max walltime for challenge_med jobs (i.e. 17 to 256 processors) was
increased from 24 hours to 96 hours on November 28th 2007.
(D) As of January 2009 the debug queue now has 4 dedicated nodes for
quick turn around debugging work. 2- 4way nodes and 4- 16way nodes.
(E) Do not specify a node_type when submitting jobs to the data queue.
Please refer to the sample PBS data staging script below for an example.
(F) The max walltime for challenge_med jobs (i.e. 17 to 256 processors) was
decreased from 96 hours to 48 hours on September 25, 2008.
(G) The max number of CPUs was increased from 512 to 516 on August 19, 2009.
The max walltime was increased from 12 hours to 48 hours on August 21, 2009.
PBS Commands:
=============
Below is a list of common PBS commands. Additional information is
available in the man pages for each command.
Command Purpose
-------------- -----------------------------------------
qsub submit jobs to a queue
qdel delete a job from the queue
qsig send a signal to a running job
qmap display a map of the system (ARSC only)
Running a Job:
==============
To run a batch job, create a qsub script which, in addition to
running your commands, specifies the processor resources and time
required. Submit the job to PBS with the following command. (For
more PBS directives, type "man qsub".)
qsub <script file>
Sample PBS scripts:
--------------
#### Beginning of MPI Example Script using Sun Fire x2200 nodes ############
#!/bin/bash
#PBS -q standard
#PBS -l select=8:ncpus=4:node_type=4way
#PBS -l walltime=8:00:00
#PBS -j oe
cd $PBS_O_WORKDIR
mpirun -np 32 ./myprog
#### End of Sample Script ##################
#### Beginning of MPI Example Script using Sun Fire x4600 nodes ############
#!/bin/bash
#PBS -q standard
#PBS -l walltime=8:00:00
#PBS -l select=2:ncpus=16:node_type=16way
#PBS -j oe
cd $PBS_O_WORKDIR
mpirun -np 32 ./myprog
#### End of Sample Script ##################
#### Beginning of OpenMP Example Script using Sun Fire x4600 nodes ############
#!/bin/bash
#PBS -q standard
#PBS -l select=1:ncpus=16:node_type=16way
#PBS -l walltime=8:00:00
#PBS -j oe
cd $PBS_O_WORKDIR
export OMP_NUM_THREADS=16
export PSC_OMP_AFFINITY=TRUE
./myprog
#### End of Sample Script ##################
#### Beginning of Data Staging Example Script ############
#!/bin/bash
#PBS -q data
#PBS -l walltime=4:00:00
#PBS -l select=1:ncpus=1
#PBS -j oe
cd $PBS_O_WORKDIR
cp -r $ARCHIVE_HOME/mydataset/* . || exit 1
qsub mpi_job.pbs
#### End of Sample Script ##################
Resource Limits:
==================
The only resource limits users should specify are walltimes and select,
ncpus, and node_type limits. The "select" statement requests a job be
allocated a number of chunks with the given "ncpus" size. The "node_type"
limit is used to explicitly request a job be placed on x2200 or x4600 nodes.
Tracking Your Job:
==================
To see which jobs are queued and/or running, execute this
command:
qstat -a
Current Queue Limits:
=====================
Queue limits are subject to change and this news item is not always
updated immediately. For a current list of all queues, execute:
qstat -Q
For all limits on a particular queue:
qstat -Q -f <queue-name>
Maintenance
============
Scheduled maintenance activities on Midnight use the Dedicated Time
functionality of PBS to reserve all available nodes on the system.
This reservation keeps PBS from scheduling jobs which would still be
running during maintenance. This allows the queues to be left running
until maintenance. Because walltime is used to determine whether
or not a job will complete prior to maintenance, using a shorter
walltime in your job script may allow your job to be run sooner.
e.g.
If maintenance begins at 6PM and it is currently 4PM, jobs specifying
walltimes of 2 hours or less will start if there are available nodes.
CPU Usage
==========
Only one job may run per node on midnight (i.e. jobs may not share nodes).
If your job uses fewer than the number of available processors on a node
the job will be charged for all processors on the node.
* A job running on a X2200 node (i.e. node_type=4way) consumes 4 CPU hours
of allocation per hour regardless of the number of tasks running on each
node.
* A job running on a X4600 node (i.e. node_type=16way) consumes 16 CPU hours
of allocation per hour regardless of the number of tasks running on each
node.
PBS Scripts - Default Walltimes
===============================
As of January 22, 2008, any PBS script submitted to the scheduler not
including a walltime for the job will be given a default walltime of 8
hours. The only exception to this is the debug queue, which will default
to 1 hour. To indicate a walltime, add the following line to your
PBS script:
#PBS -l walltime=1:30:00
This command will request a walltime of 1 hour and 30 minutes.
-Oralee Nudson
Fri, 02 Mar 2009 12:00:00 -0800
"news retirement" on midnight
retirement
========================
Midnight Retirement
========================================
On June 1st 2010, midnight.arsc.edu will be taken out of regular
production status. Existing allocations on the system will be honored
through June 1st.
Following June 1st and up through September 30, 2010, midnight.arsc.edu
will continue to operate and will remain available to existing users,
but without full vendor support. If certain hardware and/or software
issues arise between June 1st and September 30th, the system may
become unavailable for unscheduled prolonged outages.
If you have questions regarding these planned changes, please contact
/support/support.html.
-Don Bahls
Wed, 10 Feb 2010 15:57:56 -0900
"news samples_home" on midnight
samples_home
========================
Baseline Configuration Sample Code Repository
=============================================
Filename: INDEX.txt
Description: This file contains the name,location, and brief
explanation of "samples" included in this Sample
Code Repository. There are several subdirectories within
this code repository containing frequently-used procedures,
routines, scripts, and code used on this allocated system,
midnight. This sample code repository can be
accessed from midnight by changing directories to
$SAMPLES_HOME, or changing directories to the following
location: mg56% /usr/local/pkg/samples. This particular
file can be viewed from the internet at:
http://www.arsc.edu/support/news/systemnews/midnightnews.xml#samples_home
Contents: AVAILABLE_TOOLS.txt
applications
dataManagement
debugging
jobSubmission
libraries
parallelEnvironment
******************************************************************************
Directory: applications
Description: This directory includes example uses of the software
packages available on midnight. Each directory name
corresponds to the name of the software package example.
The contents of each subdirectory may include the
following information:
* sample input files
* how to access a software license (if required)
* instructions for executing a sample test using both
PBS and the interactive job option
* applicable module and/or environment
information (if required)
* sample scripts for submitting job types using the
software
* additional resources and references available
Contents: abaqus
cobalt
gaussian
gaussian_09
gaussian_linda_09
ls-dyna
matlab
nwchem
*****************************************************************************
Directory: dataManagement
Description: This directory contains information about data management
techniques including use of the data queue, storing files in
$WORKDIR, $ARCHIVE, and proper data retrieval methods.
Contents: storageManagement
storagePolicies
usingArchive
*****************************************************************************
Directory: debugging
Description: This directory contains basic information on how to start up
and use the available debuggers on midnight.
Contents: totalview
*****************************************************************************
Directory: jobSubmission
Description: This directory contains sample PBS batch scripts
and helpful commands for monitoring job progress.
Examples include options to submit a jobs such as
declaring which group membership you belong to
(for allocation accounting), how to request a particular
software license, etc.
Contents: dataStaging
MPI_OpenMP_scripts
MPI_scripts
OpenMP_scripts
*****************************************************************************
Directory: libraries
Description: This directory contains a summary of the various libraries
available and examples of how to compile a program linking to
those libraries. Example source code, Makefiles,
and data files may be included in this directory.
Contents: scalapack
papi
*****************************************************************************
Directory: parallelEnvironment
Description: This directory contains sample code and scripts containing
compiler options for common parallel programming practices
including code profiling.
Contents: auto_tau_pi
autoloops_tau_pi
hello_world_mpi
manual_tau_pi
tau_papi_counters
*****************************************************************************
-ARSC Help Desk
Thu, 01 Jan 2009 12:00:00 -0800
"news software" on midnight
software
========================
Software on Midnight
========================================
nco: nco-3.9.9 (2009-11-11)
nco 3.9.9 is now available via the nco-3.9.9 module.
gsl: gsl 1.13 (2009-11-05)
The newest version of GSL is now available. This version
was compiled using the GNU compiler and is located
in /usr/local/pkg/gsl/gsl-1.13.gcc/
pgi: PGI 9.0.4 (2009-10-08)
The latest version of the PGI compiler is available for
serial or parallel codes by loading the PrgEnv.pgi-9.0.4
module.
subversion: subversion-1.6.3 (2009-09-02)
The latest version of subversion is available via the
subversion-1.6.3 module.
ncl: ncl-5.1.1 (2009-08-05)
ncl 5.1.1 is now available (without OpenDAP support). This
software is accessible by loading the ncl-5.1.1 module.
gaussian: gaussian-09.A.01 (2009-07-14)
Gaussian 09 vA.01 is now available in /usr/local/pkg/gaussian.
Examples are located in:
$SAMPLES_HOME/applications/gaussian_09
$SAMPLES_HOME/applications/gaussian_linda_09
git: git-1.6.1.3 (2009-07-09)
git-1.6.1.3 is now available on midnight and is available
in /usr/local/pkg/git/git-1.6.1.3
abaqus: Abaqus 6.9 (2009-07-09)
Abaqus 6.9 is now available. This package is available
by loading the "abaqus-6.9" module.
matlab: matlab-7.8.0 (2009-06-17)
Matlab 7.8.0 is now available. A module is
available for this software and can be loaded with:
module load matlab-7.8.0
After loading the module, type 'matlab' at the prompt
to open a new matlab session.
pgi: PGI 8.0.6 (2009-06-10)
The latest version of the PGI compiler is available for
serial or parallel codes by loading the PrgEnv.pgi-8.0.6
module.
nwchem: nwchem-5.1.1 (2009-05-29)
The latest release of nwchem is now available on midnight.
Contact user support to request access to this package.
netcdf: netcdf-4.0.1 (2009-05-21)
netCDF 4.0.1 is now available, located in:
/usr/local/pkg/netcdf/netcdf-4.0.1.gcc.shared
python: python-2.6.2 (2009-05-20)
Python 2.6.2 is now available on pingo with the following
modules:
basemap-0.99.3, gnuplot-1.8, Imaging-1.1.6,
matplotlib-0.98.5.2, netCDF4-0.8, numpy-1.3.0,
pycdf-0.6-3b, pytools-9, scipy-0.7.0
It is available by running the following:
module load python-2.6.2
python
geos: geos-2.2.3 (2009-05-20)
GEOS is an implementation of GIS in C++. It is located in
/usr/local/pkg/geos/geos-2.2.3.gcc
ncl: ncl-5.1.0 (2009-04-09)
ncl 5.1.0 with OPeNDAP support is now available. This software
is accessible by loading the ncl-5.1.0 module.
boost: boost-1.38.0 (2009-04-09)
boost 1.38.0 is now available. Boost is an extension to C++ and
includes a number of high levels libraries. It is available in
/usr/local/pkg/boost/boost-1.38.0
curl: curl-7.19.4 (2009-04-09)
curl 7.19.4 is now available in /usr/local/pkg/curl/curl-7.19.4.
This install is to supprt NCL 5.1, and may not have all
functionality included in the default system curl package.
libnc-dap: libnc-dap-3.7.3 (2009-04-08)
libnc-dap has been installed in order to support NCL OPeNDAP
support. It is located in:
/usr/local/pkg/libnc-dap/libnc-dap-3.7.3.gcc
libdap: libdap-3.8.2 (2009-04-08)
libdap has been installed in order to support NCL OPeNDAP
support. It is located in:
/usr/local/pkg/libdap/libdap-3.8.2.gcc
fftw: fftw-2.1.5 w/ shared libs / OpenMP (2009-04-02)
FFTW 2.1.5 has been installed with additional support
added for OpenMP and Shared Libraries for Pathscale. This
version is available in the following directory:
- /usr/local/pkg/fftw/fftw-2.1.5.path.shared
mpscp: mpscp-1.3a (2009-01-28)
mpscp has been updated to the newest available version.
mexnc: mexnc-2.9.4 (2008-12-19)
The mexnc software is now available for use with
matlab, and is located in the /usr/local/pkg/mexnc
directory.
To use this version of mexnc, "module load matlab-7.7.0"
then enter the following at the matlab command prompt:
"addpath /usr/local/pkg/mexnc/mexnc-2.9.4/mexcdf/mexnc"
matlab: matlab-7.7.0 (2008-12-19)
Matlab 7.7.0 is now available. A module is
available for this software and can be loaded with
module load matlab-7.7.0
and running 'matlab'
mpscp: mpscp-1.3a (2008-12-15)
The latest version of mpscp is available. To access this
version run "module load mpscp-1.3a".
pgi: PGI 8.0.1 (2008-12-12)
The latest version of the PGI compiler is available for
serial or parallel codes by loading the PrgEnv.pgi-8.0.1
module.
show_queues: (2008-12-08)
The show_queues command is now available on midnight.
mexnc: mexnc-2.0.31-2 (2008-10-01)
The mexnc software is now available for use with
matlab, and is located in the /usr/local/pkg/mexnc
directory.
python: Python 2.5.2 (2008-09-05)
Python 2.5.2 is available with additional modules:
matplotlib,numpy,PIL,pycdf,scipy
This package is available by loading the python-2.5.2
module.
abaqus: Abaqus 6.8 (2008-07-28)
Abaqus 6.8 is now available. This package is available
by loading the "abaqus-6.8" module.
gnuplot: gnuplot-4.2.3 (2008-07-25)
GNUPlot is now available. The executables is located
in /usr/local/pkg/gnuplot/gnuplot-4.2.3/bin as gnuplot.
pgi: PGI 7.2.2 (2008-07-24)
The latest version of the PGI compiler is available for
serial or parallel codes by loading the PrgEnv.pgi-7.2.2
module.
pathscale: PathScale 3.2 (2008-07-23)
The latest version of the PathScale compiler has been
installed. Use the "PrgEnv.path-3.2" module to access this
environment.
git: git-1.5.6 (2008-07-10)
git (revision control) has been installed on midnight.
This software is available in /usr/local/pkg/git
matlab: matlab-7.6.0 (2008-07-10)
Matlab 7.6.0 is now available. A module is also
available for this software and can be loaded with
module load matlab-7.6.0
and running 'matlab'
mexnc: mexnc-2.0.29 (2008-07-01)
The mexnc software is now available for use with
matlab, and is located in the /usr/local/pkg/mexnc
directory.
cobalt: cobalt-4.2.1 (2008-06-25)
Cobalt 4.2 was rebuilt to include HDF5 support. This
version is otherwise the same as cobalt-4.2
ncl: ncl-5.0.0 (2008-06-19)
The latest release of the NCAR Command Language is
now available on midnight. To use the latest release
run:
module load ncl-5.0.0
This version includes both NCL and NCAR Graphics.
idv: idv-2.5 (2008-06-19)
The Integrated Data Viewer (IDV) has been installed on
midnight in /usr/local/pkg/idv/idv-2.5 This software is
available by loading the idv module with the following:
module load idv-2.5
and then running the command "IDV".
pgi: PGI 7.1.6 (2008-05-05)
The latest version of the PGI compiler is available for
serial or parallel codes by loading the PrgEnv.pgi.new
module (or PrgEnv.pgi-7.1.6).
cobalt 4.2:
The latest release of cobalt is now available on midnight.
NCAR Graphics 4.4.1 (PathScale):
The PathScale version of NCAR Graphics library is now
available in /usr/local/pathscale/lib.
ezViz:
ezViz has been available on midnight for some time
now. There is now a module available which will
add the ezViz tools to your PATH. To load ezViz,
run "module load ezViz". For more informations on
ezViz, see:
https://visualization.hpc.mil/wiki/EzViz_Batch_Mode
gaussian: gaussian-03.E.01 & gaussview-4.1 (2008-03-03)
Gaussian 03 vE.01 is now available in
/usr/local/pkg/gaussian. Additionally, GaussView
has been installed in /usr/local/pkg/gaussview and
is available by running "module load gaussview" and
"gview". This software is group controlled, but please
contact consult@arsc.edu if you would like to use this
software. Also, there is now a sample file in the
$SAMPLES_HOME/applications/gaussian directory.
totalview: totalview-8.4.0-0 (2008-02-26)
The latest version of the totalview debugger is now
available via the modules environment. To use the
latest version run "module load totalview-8.4.0-0"
mpscp: mpscp-1.2b (2008-01-04)
During scheduled maintenance on January 9, 2008, the
default version of mpscp on midnight will be upgraded
from version 1.2 to 1.2b.
pgi: PGI 7.0.2 (2007-12-05)
The PGI compilers have been installed on midnight.
Additionally, the following packages have been built to
support this compiler suite:
acml-3.6.0, blacs-1.1.3, fftw-2.1.5, fftw-3.1.2,
hdf-4.2r1, ncarg-4.4.1, netcdf-3.6.1, scalapack-1.8.0
Additionally, modules "PrgEnv.pgi" and "PrgEnv.pgi-7.0.2" are
available for ease of use.
pathscale: PathScale 3.1 (2007-11-19)
The latest version of the PathScale compiler has been
installed. Use the "PrgEnv.path-3.1" module to access this
environment.
ncl-4.3.1: NCAR Command Language 4.3.1 (2007-08-28)
New version of ncl is available via the "ncl-4.3.1" module.
sunstudio: Sun Studio 2007-06 (2007-09-14)
The latest release of the Sun Studio Compiler suite
is available as of Sept 14, 2007. The new version is
available via the "PrgEnv.sun-2007-06" module.
pathscale: PathScale (2.5 & 3.0) (2007-08-09)
The PathScale compiler defaults file will be installed
and active as of Aug 23rd at 8:00am AST. This sets defaults
relating to include/lib paths as well as search path, and
shared library paths. This should only affect new builds,
and should not impact executables and objects compiled
prior to Aug 23rd, 2007.
ncl-4.3.0: NCAR Command Language 4.3.0 (2007-06-27)
New version of ncl is available via the "ncl-4.3.0" module.
svn: Subversion 1.4.3 installed (2007-06-01)
subversion (svn) is a versioning control systems. See
'man svn' or 'svn help' for more information. This
version was built with SSL enabled to allow access to
https servers as well as standard http and ssh.
-Alec Bennett
Tue, 17 Feb 2009 09:33:32 -0800
"news storage" on midnight
storage
========================
Midnight Storage
========================================
The environment variables listed below represent paths. They are
expanded to their actual value by the shell, and can be used in
commands (i.e. ls $ARCHIVE_HOME). From the command prompt, the
expanded path and the variable are usually interchangeable. However,
in non-shell settings like ftp, you will need to use the actual path,
not the variable.
In the listing below, $USER is an environment variable holding
your ARSC username.
Filesystem Purpose Quota
------------------ ------------------------ -----------
$HOME dotfiles, sm. files 500 MB
/u1/uaf/$USER
/u2/wes/$USER
/u2/red/$USER
$WORKDIR do work here 100 GB (Initial)
/wrkdir/$USER (Purged)
$ARCHIVE_HOME long-term remote storage no quota
/archive/$HOME
$SCRATCH local drive each node no quota
/scratch/$USER (Purged)
-- $HOME: Home directories are intended primarily for basic account
info (e.g. dotfiles). Please use $WORKDIR (your /wrkdir/$USER
directory) for compiles, inputs, outputs, etc. Files in the
$HOME are backed up periodically. Quotas are enabled on this
filesystem. Use the command "quota -v" to show your current
$HOME use.
-- $ARCHIVE_HOME: Long-term backed up storage is only available in
your $ARCHIVE_HOME directory. As this is an NFS-mounted
filesystem from seawolf, files will be temporarily unavailable
when seawolf goes down for maintenance. I/O performance in this
directory will be much slower. Compiles in $ARCHIVE_HOME are not
recommended. $ARCHIVE_HOME is not available from compute nodes.
The "data" queue provides access to $ARCHIVE_HOME for copying
data to and from $WORKDIR. See the "Long Term Storage Use"
section below for additional Long Term Storage recommendations.
-- $WORKDIR: Short term, not backed up, purged filesystem. This is
a large fast local disk. The $WORKDIR filesystem is available
to all nodes on midnight. This is the recommended location
for input, output, and temporary files. The $ARCHIVE_HOME
filesystem is available for long term storage.
-- $SCRATCH: drives are local to each node. For example, the
$SCRATCH on midnight1 is only available to that system and is
not available to any other compute node in the midnight cluster.
You must copy any data you wish to keep from $SCRATCH to $WORKDIR
at the completion of your job. When possible $WORKDIR should
be preferred over $SCRATCH as files in $WORKDIR are available
from all compute nodes.
The $SCRATCH directory is not created by default. If you wish to
use $SCRATCH, you must create the directory on each compute node
that you will be using it on. Please contact the ARSC Help desk
for further details.
Long Term Storage Use
======================
batch_stage:
------------
Files saved in $ARCHIVE_HOME can potentially be offline (i.e. not
on disk). When accessing multiple files in $ARCHIVE_HOME, the
"batch_stage" can significantly speed the process of retrieving
files from tape.
e.g.
cd $ARCHIVE_HOME/somedirectory
find . -type f | batch_stage -i
See "man batch_stage" for additional examples.
/usr/bin/rcp:
-------------
While $ARCHIVE_HOME is available as an NFS filesystem, higher
transfer rates can be obtained by using the "rcp" command for
large transfers to and from $ARCHIVE_HOME.
The non-kerberosized version of rsh may be used to transfer files to
$ARCHIVE_HOME using the "seanfs" hostname.
e.g.
/usr/bin/rcp results.tar "seanfs:$ARCHIVE_HOME"
NOTE: The full path to rcp (i.e. /usr/bin/rcp) must be used to
make transfers without a ticket.
See http://www.arsc.edu/support/howtos/storage.html for more information
on storage best practices at ARSC.
-Donald Bahls
Thu, 12 Apr 2007 09:39:38 -0800
"news support" on midnight
support
========================
User Support Information
========================================
ARSC hosts accounts for two sets of users: HPCMP and academic. To
receive the most prompt support, it is important to contact the
appropriate support channel for your account.
HPCMP users should contact the Consolidated Customer Assistance
Center (CCAC) as their first point of contact.
The ARSC Help Desk should be the the first point of contact
for academic users.
The support command will display the appropriate support email
address and phone number for your account type:
% support
-Oralee Nudson
Thu, 07 Aug 2008 09:44:24 -0800
"news totalview" on midnight
totalview
========================
Totalview
========================================
Totalview is available on midnight and can be used to debug MPI, OpenMP and
serial applications. Generally debugging should occur on compute nodes
through the use of an interactive PBS job. Totalview may be run on login
nodes to debug short serial applications or to inspect core files.
The instructions below are prefaced by a prompt corresponding to a system
name where the command should be run.
+ midnight% corresponds to a midnight login node
(i.e. midnight1 or midnight2).
+ midnight-compute% corresponds to a midnight compute node.
+ local% corresponds to the name of your local workstation.
I. Starting an interactive job with X11 forwarding enabled.
A) Log into midnight1 or midnight2 with X11 forwarding enabled.
local% ssh -X -Y username@midnight.arsc.edu
B) Run the "tunnelx" command to setup X11 forwarding for the
compute nodes.
midnight1% tunnelx
C) Start an interactive PBS job requesting the number of processors
required for your job.
# for an 8 task MPI job on 4way (i.e. X2200) nodes
midnight% qsub -l select=2:ncpus=4:node_type=4way -I
# for a 16 task OpenMP job on 16way (i.e. X4600) nodes
midnight% qsub -l select=1:ncpus=16:node_type=16way -I
# for a serial job
midnight% qsub -l select=1:ncpus=1:node_type=4way -I
When there are a sufficient number of nodes available, PBS will
start the job.
D) Once the interactive PBS job starts, rerun the "tunnelx"
command to initialize the X11 settings for your shell.
midnight-compute% tunnelx
Next, source the file ~/.tx_local which was generated by the tunnelx
command.
# bash/ksh syntax
midnight-compute% . ~/.tx_local
# csh/tcsh syntax
midnight-compute% source ~/.tx_local
You should be able to run X11 applications on the compute nodes.
E) For additional details on the "tunnelx" command see "man tunnelx"
II. Running totalview.
A) For MPI applications, start the application using the "-tv" and
"-timeout" flags.
midnight-compute% mpirun -tv -timeout 600 ./a.out
The "-tv" flag instructs mpirun to start the executable (a.out)
under the control of totalview. The "-timeout" flag sets the
maximum time the MPI environment should wait before generating
a timeout error. Setting the timeout limit makes it easier to
set up breakpoints, etc without totalview.
B) For OpenMP and serial applications, start the application using
totalview.
midnight-compute% totalview ./a.out
Additional hints:
1) Code should be compiled with -g. This makes it possible for
totalview to refer back to the source code. Code compiled without
-g will appear as assembly and you will not have meaningful access
to variable values.
2) You can view core files with totalview by passing the executable
and core file to totalview. A core file from an MPI application
can be viewed without using mpirun.
midnight% totalview ./a.out core.1234
3) The totalview command line version (i.e. totalviewcli) is also
available. This version can be used with MPI environment by
setting the TOTALVIEW environment variable before issuing the
mpirun command.
# bash/ksh syntax
midnight-compute% export TOTALVIEW=/usr/local/bin/totalviewcli
midnight-compute% mpirun -tv -timeout 600 ./a.out
# csh/tcsh syntax
midnight-compute% setenv TOTALVIEW /usr/local/bin/totalviewcli
midnight-compute% mpirun -tv -timeout 600 ./a.out
For more information, see http://www.totalviewtech.com
-Donald Bahls
Mon, 02 Jun 2008 09:47:51 -0800
These items were uploaded on 20100313 at 08:42:13. News items are available for other systems.
Arctic Region Supercomputing Center
PO Box 756020, Fairbanks, AK 99775 | voice: 907-450-8600 | email: 
home | search | about | support | news | science | resources