![[Menu Bar]](/images/header.gif)
ARSC news items for midnight
Contents
News Items
"news downtime" on midnight
downtime
========
This file contains a description of the next scheduled downtime
for preventive maintenance, and the most recent system downtime
and/or network interruption. All times are Alaska Time. Contact
consult@arsc.edu or 907-450-8602 for more information.
==========================================================================
ONGOING DOWNTIME(S)
-------------------
NONE
NEXT SCHEDULED DOWNTIME(S)
--------------------------
16 JUL 2008 SCHEDULED DOWNTIME
machine(s): midnight
start time: 17:00
end time: 23:00
reason: Scheduled maintenance for switch hardware changes.
MOST RECENT DOWNTIME(S)
-----------------------
08 JUL 2008 UNSCHEDULED DOWNTIME
machine(s): midnight
start time: 10:30
end time: 16:30
reason: $WORKDIR filesystem outage. Scans suggest that there
are no complications from this outage, however please
review the output from jobs running in $WORKDIR to
ensure the files are intact.
07 JUL 2008 UNSCHEDULED DOWNTIME
machine(s): midnight1 (login node only)
start time: 13:20
end time: 14:45
reason: Login node was rebooted to clear hang.
04 JUL 2008 UNSCHEDULED DOWNTIME
machine(s): midnight
start time: 08:08
end time: 11:00
reason: $WORKDIR filesystem outage.
02 JUL 2008 UNSCHEDULED DOWNTIME
machine(s): midnight
start time: 13:35
end time: 16:17
reason: Lustre filesystem error. Scans suggest that there
are no complications from this outage, however please
review the output from jobs running in $WORKDIR to
ensure the files are intact.
19 JUN 2008 UNSCHEDULED DOWNTIME
machine(s): midnight1
start time: 16:28
end time: 16:33
reason: Unexpected reboot.
19 JUN 2008 UNSCHEDULED DOWNTIME
machine(s): midnight1
start time: 10:40
end time: 01:55
reason: Emergency maintenance.
11 JUN 2008 SCHEDULED DOWNTIME
machine(s): seawolf ($ARCHIVE_HOME)
start time: 18:00
end time: 22:00
reason: Firmware upgrades.
27 MAY 2008 SCHEDULED DOWNTIME
machine(s): midnight1 and midnight2
start time: 10:00
end time: 10:00
reason: Update public interface to use IPv6 protocol. This
change should be transparent, however if you experience
problems, please contact user support.
MOST RECENT DOWNTIME(S)
17 MAY 2008 SCHEDULED DOWNTIME
machine(s): midnight
start time: 17:00 17 MAY 2008
end time: 17:00 18 MAY 2008
reason: The machine room will undergo power maintenance. System
maintenance will be performed before and/or after the
power maintenance is completed.
25 APR 2008 UNSCHEDULED DOWNTIME
machine(s): midnight
start time: 10:12
end time: 11:30
reason: There was a problem with the $WORKDIR filesystem. Due
to the nature of this problem the output from jobs running
during this outage may be incomplete. The login node
midnight1 (a.k.a. midnight1) had to be rebooted due to
a related issue with the $WORKDIR filesystem.
16 APR 2008 SCHEDULED DOWNTIME
machine(s): midnight
start time: 16:00
end time: 22:00
reason: Security Updates, System Testing, Voltaire Switch Updates,
Cisco Switch Updates, Lustre Hardware updates. As a
reminder $WORKDIR is not backed up. Be sure to copy files
you wish to have backed up to $ARCHIVE_HOME prior to
maintenance.
14 APR 2008 UNSCHEDULED DOWNTIME
machines(s): midnight1 ( /archive/u1/uaf mount )
start time: 14:27
end time: 15:59
reason: The QFS mount sea_u1 is currently hung. This only affects
users with $ARCHIVE_HOME directories in /archive/u1/uaf.
If you need immediate access to $ARCHIVE_HOME, please
use midnight2.
08 APR 2008 UNSCHEDULED DOWNTIME
machine(s): seawolf ($ARCHIVE)
start time: 14:20
end time: 15:20
reason: Reboot filesystem.
07 APR 2008 UNSCHEDULED DOWNTIME
machine(s): midnight
start time: 19:45 07 APR 2008
end time: 02:55 08 APR 2008
reason: There is a problem with the $WORKDIR filesystem on
midnight. Several data nodes were rebooted to
recover the system therefore the output from jobs
that were running during this time period may be
incomplete. If you notice anomalies in your output
please report them to User Support.
02 APR 2008 UNSCHEDULED DOWNTIME
machine(s): midnight
start time: 22:20 02 APR 2008
end time: 09:13 03 APR 2008
reason: There was a problem with the $WORKDIR filesystem. Due
to the nature of this problem the output from jobs running
during this outage may be incomplete.
13 MAR 2008 UNSCHEDULED DOWNTIME
machine(s): midnight
start time: 09:00
end time: 11:14
reason: There was an issue with the $HOME and /usr/local/ filesystems
on midnight.
12 MAR 2008 SCHEDULED DOWNTIME
machine(s): midnight
start time: 17:00
end time: 22:00
reason: Apply miscellaneous OS updates. Update IB switch
firmware. Update Sun Studio and PathScale
Programming Environments (see "news PrgEnv").
12 MAR 2008 SCHEDULED DOWNTIME
machine(s): seawolf ($ARCHIVE_HOME server)
start time: 18:00
end time: 22:00
reason: SL8500 firmware upgrade.
12 MAR 2008 UNSCHEDULED DOWNTIME
machine(s): seawolf ($ARCHIVE_HOME server)
start time: 14:25
end time: 14:35
reason: Unscheduled reboot.
11 FEB 2008 UNSCHEDULED DOWNTIME
machine(s): midnight
start time: 01:00
end time: 13:45
reason: Emergency $WORKDIR file system maintenance. Output for
jobs which during between 1 AM and noon should be checked
to verify results are correct.
09 JAN 2008 SCHEDULED DOWNTIME
machine(s): midnight
start time: 13:00
end time: 22:00
reason: System Maintenance. Enable NAT for selected applications.
The data queue will stopped during this maintenance.
System upgrades will be performed on Lustre nodes. Be sure
to save output from $WORKDIR to $ARCHIVE_HOME (i.e $ARCHIVE)
prior to maintenance to avoid potential data loss.
* For a log of past downtimes, type: news LOGS/downtime.log | more
Then press "space bar" to advance a page or "q" to quit.
==========================================================================
"news modules" on midnight
modules
=======
Using the Modules Package
=========================
The modules package is used to prepare the environment for various
applications before they are run. Loading a module will set the
environment variables required for a program to execute properly.
Conversely, unloading a module will unset all environment variables
that had been previously set. This functionality is ideal for
switching between different versions of the same application, keeping
differences in file paths transparent to the user.
Sourcing the Module Init Files
---------------------------------------------------------------------
For some jobs, it may be necessary to source these files, as they
may not be auto-sourced as with login shells.
Before the modules package can be used, its init file must first be
sourced.
To do this using tcsh or csh, type:
source /usr/share/modules/init/<shell>
To do this using bash, ksh, or sh, type:
. /usr/share/modules/init/<shell>
For either case, replace <shell> with the shell you are using.
If your shell is bash, for example:
. /usr/share/modules/init/bash
Once the modules init file has been sourced, the following commands
become available:
Command Purpose
---------------------------------------------------------------------
module avail - list all available modules
module load <pkg> - load a module file from environment
module unload <pkg> - unload a module file from environment
module list - display modules currently loaded
module switch <old> <new> - replace module <old> with module <new>
module purge - unload all modules
-Alec Bennett, 2 March 2007
[This news item valid until March 2008]
"news mpi" on midnight
MPI Environment
===============
Midnight uses the Voltaire versions of the MVAPICH MPI stack. A list of
MVAPICH specific features is available on the ARSC webpage here:
http://www.arsc.edu/support/howtos/usingsun.html#mpi
Changes to MPI Stack
====================
The following are a list of changes to the MPI stack on midnight.
2008-05-19:
Task affinity is now enabled by default for MPI jobs. It can be disabled
by using "-noaff". See 2008-04-30 changes for details.
2008-04-30:
The mpirun command now includes an option to enable task affinity (-aff).
This option helps decrease the variability in run time for many applications,
however it should not be used for hybrid applications (e.g. MPI/OpenMP).
The "-noaff" option can be used do disable task affinity. This option should
be used for hybrid MPI/OpenMP or MPI/pthread applications.
Task affinity will be enabled for all MPI applications not specify "-noaff"
during maintenance on May 17-18th.
MPI Feature Support
===================
The following MPI features are not currently supported on midnight.
1) MPI-2 and MPI-IO. MPI-2 features including MPI-IO are not currently
available on midnight.
2) MPMD Support. The MVAPICH stack does not support the multiple program
multiple data (MPMD) programming model used by some codes (e.g. CCSM).
-Donald Bahls, 30 April 2008
[This news item valid until April 2009]
"news pin" on midnight
New SecurID card PINs to be 5 digit
===================================
In order to improve the security and effectiveness of our authentication
system, ARSC will begin issuing 5 digit SecurID card PINs as of 1
May, 2007. Existing SecurID cards will NOT have their PINs updated
at this time. However, if a PIN reset is needed, the replacement
PIN will be 5 digit. All new SecurID cards issued after 1 May will
be set to 5 digit PINs. If you have any questions please contact
the ARSC Help Desk.
-Derek Bastille, 27 April 2007
[This news item valid until April 2008]
"news PrgEnv" on midnight
Programming Environments on midnight
====================================
Compiler and MPI Library versions on midnight are controlled via
the modules package. New accounts load the "PrgEnv" module by
default. This module adds the Pathscale compilers and Voltaire
MPI compilers to the PATH.
Should you experience problems with a compiler or library in
many cases a new programming environment may be available.
Below is a description of available Programming Environments:
Module Name Description
=============== ==============================================
PrgEnv Programming environment using Pathscale
compilers and MPI stack (default version).
PrgEnv.new programming environment using the latest available
version of the Pathscale compilers.
PrgEnv.path Alternate name for PrgEnv
PrgEnv.path.new Alternate name for PrgEnv.new
PrgEnv.gcc Programming environment using GNU compilers
and MPI stack.
PrgEnv.pgi Programming environment using PGI compilers
and MPI stack.
PrgEnv.sun Programming environment using Sun Studio compilers
and MPI stack.
For a list of the latest available Programming Environments, run:
mg56 % module avail PrgEnv
--------------------- /usr/local/pkg/modulefiles ----------------
PrgEnv PrgEnv.new PrgEnv.pgi
PrgEnv.gcc PrgEnv.path PrgEnv.sun
PrgEnv.gcc.new PrgEnv.path.new PrgEnv.sun.new
ARSC Policy for Maintaining Programming Environment Module Versions
===================================================================
ARSC maintains a "current" programming environment, the last
programming environment and the most recently released version using
the "modules" facility. These policies explain the naming scheme
for the programming environment modules as well as the schedule for
changing the versions referenced by those modules.
- These policies pertain to the PathScale, PGI, and SunStudio compilers.
PathScale is our "official" compiler, hence it is pointed to by the
special names PrgEnv, PrgEnv.old and PrgEnv.new reflecting the
"current", "old" and "new" versions respectively. These are aliases for:
PrgEnv == PrgEnv.path
PrgEnv.old == PrgEnv.path.old
PrgEnv.new == PrgEnv.path.new
There will be "current", "old" and "new" versions for the other
compilers as well
PrgEnv.pgi, PrgEnv.pgi.old, PrgEnv.pgi.new
PrgEnv.sun, PrgEnv.sun.old, PrgEnv.sun.new
and in the discussion that follows, it is understood that policies that
apply to "PrgEnv", "PrgEnv.old" and "PrgEnv.new" apply to all three
compilers.
- The above module names are aliases for modules named by their version,
e.g., PrgEnv.path-3.0.
- When versions newer than "PrgEnv.new" are made available, we will
assign them a version-specific module name and notify users of their
availability. We will install new Programming Environment versions as
soon as practical, balanced against the benefits of the new version as
judged by ARSC staff.
- Under normal circumstances, i.e., no "extreme" need to act sooner, we
will rotate "PrgEnv", "PrgEnv.old" and "PrgEnv.new" no more frequently
than twice annually, on or about October 1 and April 1. This rotation
will be done only if "PrgEnv.new" is actually a newer version than
"PrgEnv" and only if "PrgEnv.new" has been available for testing for at
least one month. The rotation keeps the old version and updates the
current version as follows:
"PrgEnv.old" <-- "PrgEnv"
"PrgEnv" <-- "PrgEnv.new"
"PrgEnv.new" <-- a newer version (if it exists)
- Under extreme circumstances (e.g., a version that we cannot tolerate
being used or a significant system upgrade), ARSC staff may elect to
rotate these versions sooner or remove a version entirely.
- Insofar as it is feasible, we will maintain old versions (older than
"PrgEnv.old") as long as possible and will always maintain at least the
"PrgEnv.old" version.
- Any changes to "PrgEnv.old", "PrgEnv" and "PrgEnv.new" will be
preceded by at least two weeks notice to users.
Programming Environment Changes
================================
The following is a table of recent additions and changes to the
Programming Environment on midnight.
Date Module Name Description
---------- --------------------- -----------------------------------
2008-05-05 PrgEnv.pgi.new Latest release of the PGI compiler
PrgEnv.pgi-7.1.6 Suite.
2007-12-05 PrgEnv.pgi New Programming Environment using
PrgEnv.pgi-7.0.2 PGI Compiler Suite
2007-11-19 PrgEnv.path-3.1 New Programming Environment using
PathScale 3.1 compilers.
2007-09-10 PrgEnv.sun-2007-06 New Programming Environment using
Sun Studio 2007-06 compilers.
Pending Updates
===============
During system maintenance on March 12, 2008 the PathScale and Sun Studio
Programming Environments will be updated as follows:
Module Name Alias to Notes
--------------- ------------------- -----------------------
PrgEnv.old PrgEnv.path-2.5 new module
PrgEnv.path.old PrgEnv.path-2.5 new module
PrgEnv PrgEnv.path-3.0 was PrgEnv.path-2.5
PrgEnv.path PrgEnv.path-3.0 was PrgEnv.path-2.5
PrgEnv.new PrgEnv.path-3.1 was PrgEnv.path-3.0
PrgEnv.path.new PrgEnv.path-3.1 was PrgEnv.path-3.0
PrgEnv.sun.old PrgEnv.sun-2006-08 new module
PrgEnv.sun PrgEnv.sun-2006-12-r2 unchanged
PrgEnv.sun.new PrgEnv.sun-2007-06 was PrgEnv.sun-2006-12-r2
-Donald Bahls, 27 Feb 2008
[This news item valid until Feb 2009]
"news projects" on midnight
Instructions for Users with Multiple Projects
=============================================
This news item is intended for users that are members of more than one
project. Users in a single project will automatically have use
charged against the allocation for their primary group (i.e. project).
Users in more than one project can select an alternate project to
charge use to by using the "-W group_list" PBS option. If the
"-W group_list" option is not specified the account number will default
to your primary group (i.e. project).
Below is an example "-W group_list" statement.
e.g.
#PBS -W group_list=proja
The "-W group_list" option can also be used on the command line.
e.g.
mg56 % qsub -Wgroup_list=proja script.bat
Each project has a corresponding UNIX group, therefore the groups
command will show all projects and other groups of which you are a
member.
e.g.
mg56 % groups
proja projb
In this case use would be charged to proja by default, but could be
charged to projb by setting "-W group_list=projb" in the PBS script.
If you have questions about this news item, please contact the ARSC
help desk (consult@arsc.edu).
'show_usage' Available
=======================
Project utilization information is now available via the BCT Compliant 'show_usage'
command. 'show_usage' with no parameters will display the remaining
allocation for each project of which you are a member.
e.g.
mg56 % show_usage
ARSC - Subproject Usage Information (in CPU Hours)
As of 01:11:21 hours ADT 11 Nov 2007
For Fiscal Year 2008 (01 October 2007 - 30 September 2008)
Percentage of Fiscal Year Remaining: 88.77%
Hours Hours Hours Percent Background
System Subproject Allocated Used Remaining Remaining Hours Used
========== ============== ========== ========== ========== ========= ==========
midnight projectA 100000.00 39663.48 60336.52 60.34% 0.00
midnight projectB 5000.00 0.00 5000.00 100.00% 0.00
Project utilization for all allocated systems can be obtained by using 'show_usage'
with the '-s' option as in 'show_usage -s'.
mg56 % show_usage -s
ARSC - Subproject Usage Information (in CPU Hours)
As of 13:00:00 hours ADT 11 Nov 2007
For Fiscal Year 2008 (01 October 2007 - 30 September 2008)
Percentage of Fiscal Year Remaining: 88.77%
Hours Hours Hours Percent Background
System Subproject Allocated Used Remaining Remaining Hours Used
========== ============== ========== ========== ========== ========= ==========
iceberg projectA 30000.00 309.84 29690.16 98.97% 0.00
iceberg projectB 10000.00 3873.46 6126.54 61.27% 0.00
midnight projectA 100000.00 39663.48 60336.52 60.34% 0.00
midnight projectB 5000.00 0.00 5000.00 100.00% 0.00
-Donald Bahls, 11 November 2007
[This news item valid until November 2008]
"news queues" on midnight
Midnight Queues
===============
The queue configuration is as described below. It is subject to
review and further updates.
Queues:
===============
Specify one of the following queues in your PBS qsub script
(e.g., "#PBS -q standard"):
Queue Name Purpose of queue
------------- ------------------------------
standard General use by all allocated users.
debug Quick turnaround queue for debugging work.
data Provides access to long term storage (i.e. $ARCHIVE_HOME)
background For projects with little or no remaining allocation.
This queue has the lowest priority, however projects
running jobs in this queue do not have allocation deducted.
The number of running jobs or processors available to this
queue may be altered based on system load.
special For jobs which do not fit into normal queue limits. (Limited Access)
high For jobs requiring higher priority than normal. (Special Access Only)
urgent For jobs requiring highest priority. (Special Access Only)
challenge For challenge projects only.
See 'qstat -q' for a complete list of system queues. Note, some
queues are not intended for general use.
Maximum Walltimes:
===================
The maximum allowed walltime for a job is dependant on the number of
processors requested. The table below describes maximum walltimes for
each queue.
Queue Min Max Max
Procs Procs Walltime Notes
--------------- ----- ----- --------- ------------
standard
1 16 84:00:00 See (A)
17 256 16:00:00
257 512 12:00:00
challenge
1 16 96:00:00 See (B)
17 256 96:00:00 See (C)
257 512 12:00:00
background 1 512 12:00:00
debug 1 32 01:00:00 See (D)
data 1 1 12:00:00 See (E)
NOTES:
(A) The max walltime for standard_sm jobs (i.e. 1 to 16 processors) was
increased from 42 hours to 84 hours on October 5th 2007.
(B) The max walltime for challenge_sm jobs (i.e. 1 to 16 processors) was
increased from 48 hours to 96 hours on October 5th 2007.
(C) The max walltime for challenge_med jobs (i.e. 17 to 256 processors) was
increased from 24 hours to 96 hours on November 28th 2007.
(D) As of January 9th 2007 the debug queue now has 4 dedicated nodes for
quick turn around debugging work. 2- 4way nodes and 2- 16way nodes.
(E) Do not specify a node_type when submitting jobs to the data queue.
Please refer to the sample PBS data staging script below for an example.
PBS Commands:
=============
Below is a list of common PBS commands. Additional information is
available in the man pages for each command.
Command Purpose
-------------- -----------------------------------------
qsub submit jobs to a queue
qdel delete a job from the queue
qsig send a signal to a running job
qmap display a map of the system (ARSC only)
Running a Job:
==============
To run a batch job, create a qsub script which, in addition to
running your commands, specifies the processor resources and time
required. Submit the job to PBS with the following command. (For
more PBS directives, type "man qsub".)
qsub <script file>
Sample PBS scripts:
--------------
#### Beginning of MPI Example Script using Sun Fire x2200 nodes ############
#!/bin/bash
#PBS -q standard
#PBS -l select=8:ncpus=4:node_type=4way
#PBS -l walltime=8:00:00
#PBS -j oe
cd $PBS_O_WORKDIR
mpirun -np 32 ./myprog
#### End of Sample Script ##################
#### Beginning of MPI Example Script using Sun Fire x4600 nodes ############
#!/bin/bash
#PBS -q standard
#PBS -l walltime=8:00:00
#PBS -l select=2:ncpus=16:node_type=16way
#PBS -j oe
cd $PBS_O_WORKDIR
mpirun -np 32 ./myprog
#### End of Sample Script ##################
#### Beginning of OpenMP Example Script using Sun Fire x4600 nodes ############
#!/bin/bash
#PBS -q standard
#PBS -l select=1:ncpus=16:node_type=16way
#PBS -l walltime=8:00:00
#PBS -j oe
cd $PBS_O_WORKDIR
export OMP_NUM_THREADS=16
export PSC_OMP_AFFINITY=TRUE
./myprog
#### End of Sample Script ##################
#### Beginning of Data Staging Example Script ############
#!/bin/bash
#PBS -q data
#PBS -l walltime=4:00:00
#PBS -l select=1:ncpus=1
#PBS -j oe
cd $PBS_O_WORKDIR
cp -r $ARCHIVE_HOME/mydataset/* . || exit 1
qsub mpi_job.pbs
#### End of Sample Script ##################
Resource Limits:
==================
The only resource limits users should specify are walltimes and select,
ncpus, and node_type limits. The "select" statement requests a job be
allocated a number of chunks with the given "ncpus" size. The "node_type"
limit is used to explicitly request a job be placed on x2200 or x4600 nodes.
Tracking Your Job:
==================
To see which jobs are queued and/or running, execute this
command:
qstat -a
Current Queue Limits:
=====================
Queue limits are subject to change and this news item is not always
updated immediately. For a current list of all queues, execute:
qstat -Q
For all limits on a particular queue:
qstat -Q -f <queue-name>
Maintenance
============
Scheduled maintenance activities on Midnight use the Dedicated Time
functionality of PBS to reserve all available nodes on the system.
This reservation keeps PBS from scheduling jobs which would still be
running during maintenance. This allows the queues to be left running
until maintenance. Because walltime is used to determine whether
or not a job will complete prior to maintenance, using a shorter
walltime in your job script may allow your job to be run sooner.
e.g.
If maintenance begins at 6PM and it is currently 4PM, jobs specifying
walltimes of 2 hours or less will start if there are available nodes.
CPU Usage
==========
Only one job may run per node on midnight (i.e. jobs may not share nodes).
If your job uses fewer than the number of available processors on a node
the job will be charged for all processors on the node.
* A job running on a X2200 node (i.e. node_type=4way) consumes 4 CPU hours
of allocation per hour regardless of the number of tasks running on each
node.
* A job running on a X4600 node (i.e. node_type=16way) consumes 16 CPU hours
of allocation per hour regardless of the number of tasks running on each
node.
PBS Scripts - Default Walltimes
===============================
As of January 22, 2008, any PBS script submitted to the scheduler not
including a walltime for the job will be given a default walltime of 8
hours. The only exception to this is the debug queue, which will default
to 1 hour. To indicate a walltime, add the following line to your
PBS script:
#PBS -l walltime=1:30:00
This command will request a walltime of 1 hour and 30 minutes.
-Oralee Nudson, 2 March 2007
[This news item valid until March 2008]
"news samples_home" on midnight
Baseline Configuration Sample Code Repository
=============================================
Filename: INDEX.txt
Description: This file contains the name,location, and brief
explanation of "samples" included in this Sample
Code Repository. There are several subdirectories within
this code repository containing frequently-used procedures,
routines, scripts, and code used on this allocated system,
midnight. This sample code repository can be
accessed from midnight by changing directories to
$SAMPLES_HOME, or changing directories to the following
location: mg56% /usr/local/pkg/samples. This particular
file can be viewed from the internet at:
http://www.arsc.edu/cgi/news.cgi
******************************************************************************
Directory: applications
Description: This directory includes example uses of the software
packages available on midnight. Each directory name
corresponds to the name of the software package example.
The contents of each subdirectory may include the
following information:
* sample input files
* how to access a software license (if required)
* instructions for executing a sample test using both
PBS and the interactive job option
* applicable module and/or environment
information (if required)
* sample scripts for submitting job types using the
software
* additional resources and references available
Contents: abaqus
cobalt
gaussian
matlab
nwchem
*****************************************************************************
Directory: dataManagement
Description: This directory contains information about data management
techniques including use of the data queue, storing files in
$WORKDIR, $ARCHIVE, and proper data retrieval methods.
Contents: storageManagement
storagePolicies
usingArchive
*****************************************************************************
Directory: debugging
Description: This directory contains basic information on how to start up
and use the available debuggers on midnight.
Contents: totalview
*****************************************************************************
Directory: jobSubmission
Description: This directory contains sample PBS batch scripts
and helpful commands for monitoring job progress.
Examples include options to submit a jobs such as
declaring which group membership you belong to
(for allocation accounting), how to request a particular
software license, etc.
Contents: dataStaging
MPI_OpenMP_scripts
MPI_scripts
OpenMP_scripts
*****************************************************************************
Directory: libraries
Description: This directory contains a summary of the various libraries
available and examples of how to compile a program linking to
those libraries. Example source code, Makefiles,
and data files may be included in this directory.
Contents: scalapack
papi
*****************************************************************************
Directory: parallelEnvironment
Description: This directory contains sample code and scripts containing
compiler options for common parallel programming practices
including code profiling.
Contents: auto_tau_pi
autoloops_tau_pi
manual_tau_pi
tau_papi_counters
*****************************************************************************
-Oralee Nudson, 29 October 2007
[This news item valid until October 2008]
"news software" on midnight
software
========
ncl: ncl-5.0.0 (2008-06-19)
The latest release of the NCAR Command Language is
now available on midnight. To use the latest release
run:
module load ncl-5.0.0
This version includes both NCL and NCAR Graphics.
idv: idv-2.5 (2008-06-19)
The Integrated Data Viewer (IDV) has been installed on
midnight in /usr/local/pkg/idv/idv-2.5 This software is
available by loading the idv module with the following:
module load idv-2.5
and then running the command "IDV".
pgi: PGI 7.1.6 (2008-05-05)
The latest version of the PGI compiler is available for
serial or parallel codes by loading the PrgEnv.pgi.new
module (or PrgEnv.pgi-7.1.6).
cobalt 4.2:
The latest release of cobalt is now available on midnight.
NCAR Graphics 4.4.1 (PathScale):
The PathScale version of NCAR Graphics library is now
available in /usr/local/pathscale/lib.
ezViz:
ezViz has been available on midnight for some time
now. There is now a module available which will
add the ezViz tools to your PATH. To load ezViz,
run "module load ezViz". For more informations on
ezViz, see:
https://visualization.hpc.mil/wiki/EzViz_Batch_Mode
gaussian: gaussian-03.E.01 & gaussview-4.1 (2008-03-03)
Gaussian 03 vE.01 is now available in
/usr/local/pkg/gaussian. Additionally, GaussView
has been installed in /usr/local/pkg/gaussview and
is available by running "module load gaussview" and
"gview". This software is group controlled, but please
contact consult@arsc.edu if you would like to use this
software. Also, there is now a sample file in the
$SAMPLES_HOME/applications/gaussian directory.
totalview: totalview-8.4.0-0 (2008-02-26)
The latest version of the totalview debugger is now
available via the modules environment. To use the
latest version run "module load totalview-8.4.0-0"
mpscp: mpscp-1.2b (2008-01-04)
During scheduled maintenance on January 9, 2008. The
default version of mpscp on midnight will be upgraded
from version 1.2 to 1.2b.
pgi: PGI 7.0.2 (2007-12-05)
The PGI compilers have been installed on midnight.
Additionally, the following packages have been built to
support this compiler suite:
acml-3.6.0, blacs-1.1.3, fftw-2.1.5, fftw-3.1.2,
hdf-4.2r1, ncarg-4.4.1, netcdf-3.6.1, scalapack-1.8.0
Additionally, modules "PrgEnv.pgi" and "PrgEnv.pgi-7.0.2" are
available for ease of use.
pathscale: Pathscale 3.1 (2007-11-19)
The latest version of the PathScale compiler has been
installed. Use the "PrgEnv.path-3.1" module to access this
environment.
ncl-4.3.1: NCAR Command Language 4.3.1 (2007-08-28)
New version of ncl is available via the "ncl-4.3.1" module.
sunstudio: Sun Studio 2007-06 (2007-09-14)
The latest release of the Sun Studio Compiler suite
is available as of Sept 14, 2007. The new version is
available via the "PrgEnv.sun-2007-06" module.
pathscale: Pathscale (2.5 & 3.0) (2007-08-09)
The pathscale compiler defaults file will be installed
and active as of Aug 23rd at 8:00am AST. This sets defaults
relating to include/lib paths as well as search path, and
shared library paths. This should only affect new builds,
and should not impact executables and objects compiled
prior to Aug 23rd, 2007.
ncl-4.3.0: NCAR Command Language 4.3.0 (2007-06-27)
New version of ncl is available via the "ncl-4.3.0" module.
svn: Subversion 1.4.3 installed (2007-06-01)
subversion (svn) is a versioning control systems. See
'man svn' or 'svn help' for more information. This
version was built with SSL enabled to allow access to
https servers as well as standard http and ssh.
-Alec Bennett, 7 June 2007
[This news item valid until June 2008]
"news storage" on midnight
Midnight Storage
================
The environment variables listed below represent paths. They are
expanded to their actual value by the shell, and can be used in
commands (i.e. ls $ARCHIVE_HOME). From the command prompt, the
expanded path and the variable are usually interchangeable. However,
in non-shell settings like ftp, you will need to use the actual path,
not the variable.
In the listing below, $USER is an environment variable holding
your ARSC username.
Filesystem Purpose Quota
------------------ ------------------------ -----------
$HOME dotfiles, sm. files 500 MB
/u1/uaf/$USER
/u2/wes/$USER
/u2/red/$USER
$WORKDIR do work here 100 GB (Initial)
/wrkdir/$USER (Purged)
$ARCHIVE_HOME long-term remote storage unlimited
/archive/$HOME
$SCRATCH local drive each node no quota
/scratch/$USER (Purged)
-- $HOME: Home directories are intended primarily for basic account
info (e.g. dotfiles). Please use $WORKDIR (your /wrkdir/$USER
directory) for compiles, inputs, outputs, etc. Files in the
$HOME are backed up periodically. Quotas are enabled on this
filesystem. Use the command "quota -v" to show your current
$HOME use.
-- $ARCHIVE_HOME: Long-term backed up storage is only available in
your $ARCHIVE_HOME directory. As this is an NFS-mounted
filesystem from seawolf, files will be temporarily unavailable
when seawolf goes down for maintenance. I/O performance in this
directory will be much slower. Compiles in $ARCHIVE_HOME are not
recommended. $ARCHIVE_HOME is not available from compute nodes.
The "data" queue provides access to $ARCHIVE_HOME for copying
data to and from $WORKDIR.
-- $WORKDIR: Short term, not backed up, purged filesystem. This is
a large fast local disk. The $WORKDIR filesystem is available
to all nodes on midnight. This is the recommended location
for input, output, and temporary files. The $ARCHIVE_HOME
filesystem is available for long term storage.
-- $SCRATCH: drives are local to each node. For example, the
$SCRATCH on midnight1 is only available to that system and is
not available to any other compute node in the midnight cluster.
You must copy any data you wish to keep from $SCRATCH to $WORKDIR
at the completion of your job. When possible $WORKDIR should
be preferred over $SCRATCH as files in $WORKDIR are available
from all compute nodes.
The $SCRATCH directory is not created by default. If you wish to
use $SCRATCH, you must create the directory on each compute node
that you will be using it on. Please contact the ARSC Help desk
for further details.
Updates
=========
Apr 12, 2007- All $WORKDIR directories have been moved from /lustre/$USER to
/lustre/wrkdir/$USER. If you have scripts which used
/luster/$USER rather than /wrkdir/$USER, those scripts will
need to be updated to reflect the new design.
We highly recommend using $WORKDIR or /wrkdir/$USER rather
than /lustre/wrkdir/$USER in scripts.
Apr 26, 2007- As of April 13, 2007 quotas are enabled on $HOME. Default
quotas are 500 MB.
Aug 01, 2007- On August 22, 2007 $WORKDIR file purging will begin.
All files which have not been accessed for 30 days will be
removed. Files being actively accessed in $WORKDIR will not
be removed. Thus, it is a good idea to backup your work
regularly to $ARCHIVE_HOME.
See http://www.arsc.edu/support/howtos/storage.html for more information
on storage policies at ARSC.
-Donald Bahls, 12 April 2007
[This news item valid until April 2008]
"news totalview" on midnight
Totalview
=========
Totalview is available on midnight and can be used to debug MPI, OpenMP and
serial applications. Generally debugging should occur on compute nodes
through the use of an interactive PBS job. Totalview may be run on login
nodes to debug short serial applications or to inspect core files.
The instructions below are prefaced by a prompt corresponding to a system
name where the command should be run.
+ midnight% corresponds to a midnight login node
(i.e. midnight1 or midnight2).
+ midnight-compute% corresponds to a midnight compute node.
+ local% corresponds to the name of your local workstation.
I. Starting an interactive job with X11 forwarding enabled.
A) Log into midnight1 or midnight2 with X11 forwarding enabled.
local% ssh -X -Y username@midnight.arsc.edu
B) Run the "tunnelx" command to setup X11 forwarding for the
compute nodes.
midnight1% tunnelx
C) Start an interactive PBS job requesting the number of processors
required for your job.
# for an 8 task MPI job on 4way (i.e. X2200) nodes
midnight% qsub -l select=2:ncpus=4:node_type=4way -I
# for a 16 task OpenMP job on 16way (i.e. X4600) nodes
midnight% qsub -l select=1:ncpus=16:node_type=16way -I
# for a serial job
midnight% qsub -l select=1:ncpus=1:node_type=4way -I
When there are a sufficient number of nodes available, PBS will
start the job.
D) Once the interactive PBS job starts, rerun the "tunnelx"
command to initialize the X11 settings for your shell.
midnight-compute% tunnelx
Next, source the file ~/.tx_local which was generated by the tunnelx
command.
# bash/ksh syntax
midnight-compute% . ~/.tx_local
# csh/tcsh syntax
midnight-compute% source ~/.tx_local
You should be able to run X11 applications on the compute nodes.
E) For additional details on the "tunnelx" command see "man tunnelx"
II. Running totalview.
A) For MPI applications, start the application using the "-tv" and
"-timeout" flags.
midnight-compute% mpirun -tv -timeout 600 ./a.out
The "-tv" flag instructs mpirun to start the executable (a.out)
under the control of totalview. The "-timeout" flag sets the
maximum time the MPI environment should wait before generating
a timeout error. Setting the timeout limit makes it easier to
set up breakpoints, etc without totalview.
B) For OpenMP and serial applications, start the application using
totalview.
midnight-compute% totalview ./a.out
Additional hints:
1) Code should be compiled with -g. This makes it possible for
totalview to refer back to the source code. Code compiled without
-g will appear as assembly and you will not have meaningful access
to variable values.
2) You can view core files with totalview by passing the executable
and core file to totalview. A core file from an MPI application
can be viewed without using mpirun.
midnight% totalview ./a.out core.1234
3) The totalview command line version (i.e. totalviewcli) is also
available. This version can be used with MPI environment by
setting the TOTALVIEW environment variable before issuing the
mpirun command.
# bash/ksh syntax
midnight-compute% export TOTALVIEW=/usr/local/bin/totalviewcli
midnight-compute% mpirun -tv -timeout 600 ./a.out
# csh/tcsh syntax
midnight-compute% setenv TOTALVIEW /usr/local/bin/totalviewcli
midnight-compute% mpirun -tv -timeout 600 ./a.out
For more information, see http://www.totalviewtech.com
-Donald Bahls, 2 June 2008
[This news item valid until June 2009]
These items were uploaded on 20080708 at 19:11:50. News items are available for other systems.
Arctic Region Supercomputing Center
PO Box 756020, Fairbanks, AK 99775 | voice: 907-450-8600 | email: 
home | search | about | support | news | science | resources