ARSC | Introduction to Using the Penguin Computing AMD Opteron Cluster

Introduction to Using the Penguin Computing AMD Opteron Cluster

Introduction
"pacman"
Operating System / Shells
System News, Status and RSS Feeds
Storage
Connecting to Pacman
Sample Code Repository ($SAMPLES_HOME)
Available Software
User Installed Software
Parallel Programming Models
Programming Environment
- Modules
- PGI - Compiling and Linking Fortran Programs
- PGI - Compiling and Linking C/C++ Programs
Libraries
Pre-Processing and Post-Processing on Login Nodes
Running Interactive Jobs
Running Batch Jobs
MPI Run Time options
Torque/Moab Script Examples
- MPI using 4 nodes, all 16 processors on each node
- MPI using 6 nodes, 2 processors on each node
- OpenMP using 1 node, all 16 processors
- Serial job using 1 processor on a shared node
- Transfer queue, copying files from $ARCHIVE
Charging Allocation Hours to an Alternate Project
Monitoring Project Allocation Usage
Online Manuals and Documentation
More Information

Introduction

The Arctic Region Supercomputing Center (ARSC) operates a Penguin Computing AMD Opteron cluster ( pacman ) running RedHat Enterprise Linux 5.7 and Scyld ClusterWare.

Pacman is a resource dedicated to University of Alaska affiliated academic users performing non-commercial, scientific research of Arctic interest.

"Pacman"

The ARSC Penguin Computing cluster consists of the following hardware:

12 Login Nodes
- 2- Six core 2.2 GHz AMD Opteron Processors
- 64 GB of memory per node. (64 GB per core)
- 1 Mellanox Infiniband DDR Network Card
256 Four Core Compute Nodes
- 2- Dual core 2.6 GHz AMD Opteron Processors
- 16 GB of memory per node (4 GB per core)
- Voltaire DDR Infiniband Network Card
88 Sixteen Core Compute Nodes
- 2 Eight core 2.3 GHz AMD Opteron Processors
- 64 GB of memory per node (4 GB per core)
- QLogic QDR Infiniband Network Card
- 250 GB local disk
20 Twelve Core Compute Nodes
- 2 Six core 2.2 GHz AMD Opteron Processors
- 32 GB of memory per node (2.6 GB per core)
- Mellanox Infiniband DDR Network Card
4 Large Memory Nodes
- 4 Eight core 2.3 GHz AMD Opteron Processors
- 256 GB of memory per node (8 GB per core)
- QLogic QDR Infiniband Network Card
- 1 TB local disk
- 80 GB solid state drive
2 GPU Nodes
- 2 Quad core 2.4 GHz Intel CPUs
- 64 GB of memory per node (8 GB per core)
- QLogic QDR Infiniband Network Card
- 2 M2050 nVidia Fermi GPU cards
QLogic QDR and Mellanox DDR Infiniband Interconnect
89 TB Panasas version 12 scratch file system

Operating System / Shells

The operating system on pacman is RedHat Enterprise Linux version 5.7.

The following shells are available on pacman:

sh (Bourne Shell)
ksh (Korn Shell)
bash (Bourne-Again Shell) default
csh (C Shell)
tcsh (Tenex C Shell)

If you would like to have your default login shell changed, please contact User Support .

System News & Status

System news is available via the news command when logged on to pacman. For example, the command "news queues" gives news about the current queue configuration. System status and public news items are available on the web.

Storage

Several environment variables are defined to point to the available storage on pacman.

Name	Notes	Default Quota ¹	Purge Policy	Back Up Policy
`$HOME`	`$HOME` directories are intended for locally compiled executables and libraries,dot files, and small data sets. ARSC recommends that you avoid accessing `$HOME` in parallel jobs. The $HOME environment variable expands to `/u1/uaf/$USER`.	4GB ³	not purged	backed up
`$CENTER`	`$CENTER` is a high performance Lustre file system available from all nodes on pacman. This is the preferred location to run large parallel jobs. This file system is also available on fish.arsc.edu. The $CENTER environment variable expands to `/center/w/$USER`.	750GB ³	30 day purge policy ⁴	not backed up
`$ARCHIVE`	`$ARCHIVE` directories are intended for long term storage of executables, large datasets, and compressed backups of important data. `$ARCHIVE` is only accessible by the pacman login nodes, therefore the `transfer` queue should be used to copy data from `$ARCHIVE` to `$WORKDIR` prior to batch job execution. The $ARCHIVE environment variable expands to `/archive/u1/uaf/$USER`.	no quota	not purged	backed up
`/center/d`	`/center/d` is an alternate directory for the storage of large data sets.	Based on request and availability.³	not purged	not backed up

NOTES:

Individual user's are responsible to monitor their data usage. If their quota exceeds the value stated above further writes will fail.
Requests for increased quotas should be sent to User Support.
The "show_storage" command will display current usage for $HOME, $ARCHIVE and $CENTER directories.
The $CENTER filesystem is not currently being purged on a daily basis, however it may be enabled at a future date. Data residing in $CENTER for inactive accounts may be purged without warning.
The "show_storage" command is run automatically on login to warn when disk usage is approaching set quotas. This automatic check can be disabled by running "touch $HOME/.hush_storage"

See http://www.arsc.edu/support/howtos/storage and for more information on the storage environment at ARSC.

Connecting to Pacman

Connections to pacman should be made using an SSH compliant client. Linux and Mac OS X systems normally include a command line "ssh" program. Persons using Windows systems to connect to pacman will need to install an ssh client (e.g. PuTTY). For additional details see the Connecting to ARSC Academic Systems page.

Here's an example connection command for Mac OS X and Linux command line clients:

% ssh -XY arscusername@pacman1.arsc.edu

File transfers to and from pacman should also use SSH protocol via the "scp" or "sftp" programs. Persons using Windows systems to connect to pacman will need to install an sftp or scp compatible Windows client (e.g. FileZilla, WinSCP).

Pacman has a number of login nodes available. The nodes "pacman1.arsc.edu" and pacman2.arsc.edu" are primarily intended for compilation and interacting with the batch system. Activities requiring significant CPU time or memory should occur on "pacman3.arsc.edu" through "pacman9.arsc.edu".

Login Node	Intended Purpose
`pacman1.arsc.edu`	Compiling and Batch Job Submission
`pacman2.arsc.edu`	Compiling and Batch Job Submission
`pacman3.arsc.edu` through `pacman9.arsc.edu`	Compute Intensive Interactive Work
`pacman10.arsc.edu` through `pacman12.arsc.edu`	Batch Data Transfer Work

Sample Code Repository ($SAMPLES_HOME)

The $SAMPLES_HOME directory on pacman contains a number of examples including, but not limited to:

Torque/Moab scripts for MPI, OpenMP and Hybrid applications
Examples for Abaqus, OpenFoam, Gaussian, NWChem and other installed applications.
Examples using common libraries

A description of items available in the Sample Code Repository is available on the web and in the file $SAMPLES_HOME/INDEX.txt on pacman, however you must login to pacman to access the examples.

% less $SAMPLES_HOME/INDEX.txt

If you would like to see additional samples or would like to contribute an example, please contact User Support .

Available Software

Open source and commercial applications have been installed on the system in /usr/local/pkg. In most cases, the most recent versions of these packages are easily accessible via modules. Additional packages may be installed upon request.

User Installed Software

The following directory on pacman is intended as a place for users to build third-party software packages and provide them for others to use:

/usr/local/unsupported

The purpose of this directory is to share your efforts with others. Packages built for personal use only should be installed in your $HOME directory. To request a subdirectory for your project, please contact the ARSC Help Desk with the following information:

the name of your requested subdirectory, which can be your project's name (e.g., UAFCLMIT) or the type of software you intend to install in the directory (e.g., "ClimateModels")
a general description of what you intend to install
a rough estimate of the amount of storage you will need (e.g., 100 MB)
the member of your group who will maintain this directory
- whether this person would like their email address listed for other users to contact them

An entry for your software directory will be placed in the /usr/local/unsupported/INDEX file. Entries take the following form:

DIRECTORY DESCRIPTION CONTACT

--------------- ---------------------------------------------- -----------------------
UAFCLMIT Climate Models and example runs johndoe@alaska.edu

Policies

Commercial & Export Controlled Software

Please do not install any commercial or export controlled software in the /usr/local/unsupported directory without explicit approval from the ARSC Help Desk.

Permissions

You have the option of sharing the packages you install with either:

the other members of your project
all other users on pacman

Access is controlled via Linux groups. For example:

The following command will grant read/execute access to members of your group:

chmod -Rh g+rX /usr/local/unsupported/UAFCLMIT/fftw-3.2.2.pgi

The following command will grant read/execute access to all users on pacman:

chmod -Rh go+rX /usr/local/unsupported/UAFCLMIT/fftw-3.2.2.pgi

While group-write access is allowed for these directories, please remember that world-write access is never permitted. For more information, please refer to the "Non-Home Directory/File Permissions" policies described on the following web page:

http://www.arsc.edu/support/policy/secpolicy/#other

Storage

Only the files belonging to a package's installation should be included in the /usr/local/unsupported directory. Input/output files, PBS scripts, etc., should be excluded from the installation directory unless they came with the package itself.

Due to the way the /usr/local/unsupported directory is configured, files installed in this directory will count toward your $HOME file system quota. If necessary, please send a request for an increased $HOME quota to the ARSC Help Desk.

Daily backups are performed on the /usr/local/unsupported directory.

Recommendations

Create a README File

If you plan to share your package installations with all other users on the system, we recommend that you create a README file in your group's root directory. E.g.,

/usr/local/unsupported/ClimateModels/README

In this file, you may choose to include descriptions of each package, how they were built, and contact information.

Library Naming Scheme

If you are building a library package from source, we recommend that you add a suffix to your installation directory indicating which compiler suite was used to build the library. This will help other users determine whether the installation is compatible with the compiler they are using.

The following compiler suite suffix conventions are recommended:

Compiler Suite	Suffix
Portland Group	`.pgi`
GNU	`.gnu`

For example, the directory structure for FFTW 3.2.2, built with the Portland Group compilers, might look like this:

/usr/local/unsupported/UAFCLMIT/fftw-3.2.2.pgi

Module Files

If you would like your package installations to be widely used on pacman, you may want to create a module to set up the proper environment to run the package. Please refer to the files in the following directory as examples of how to create a module:

/usr/local/pkg/modulefiles

To provide a module for your package, put the module file in a "modulefiles" subdirectory. E.g.:

/usr/local/unsupported/ClimateModels/fftw-3.2.2.pgi/modulefiles/fftw-3.2.2

Then, contact the ARSC Help Desk with the location of this module for it to be made available to all other users of the system via the "module" command.

More information on how to create module files is available via the "man modulefile" page.

Parallel Programming Models

Several types of parallelism can be employed on pacman using different programming models and methods. These are listed in the table below.

Hardware Level	Model	Description
Shared-memory node	Auto	Automatic shared-memory parallel executables can be compiled by and linked with the `-Mconcur=` option to `pgf90` `pgcc` or `pgCC` Since only a subset of loops can generally be parallelized this way OpenMP directives can furtherimprove performance.
Shared-memory node	OpenMP	This is a form of explicit parallel programming in which the programmer inserts directivesinto the program to spawn multiple shared-memory threads, typically at the loop level. It iscommon, portable, and relatively easy. On the downside, it requires shared memory which limitsscaling to the number of processors on a node. To activate OpenMP directives in your code, usethe `-mp` option to `pgf90` `pgcc` or `pgCC` OpenMP can be used in conjunction with autoparallelization.
Shared-memory node	pthreads	The system supports POSIX threads.
Distributed memory system	MPI	This is the most common and portable method for parallelizing codes for scalable distributedmemory systems. MPI is a library of subroutines for message passing, collective operations, andother forms of inter-processor communication. The programmer is responsible for implementingdata distribution, synchronization, and reassembly of results using explicit MPI calls. Using MPI, the programmer can largely ignore the physical organization of processors intonodes and simply treat the system as a collection of independent processors.

Programming Environment

The following programming tools are available on pacman.

Item	MPI compilers (*)	Portland Group	GNU Compilers v4.1.2
Fortran 77	`mpif77`	`pgf77`	`g77`
Fortran 90/95	`mpif90`	`pgf90`	`gfortran`
C compiler	`mpicc`	`pgcc`	`gcc / cc`
C++ compiler	`mpiCC`	`pgCC`	`g++ / c++`
Debuggers		`pgdbg`	`gdb`
Performance Analysis		`pgprof`	`gprof`
Default MPI module (*)	`PrgEnv-pgi` or `PrgEnv-gnu`	`PrgEnv-pgi`	`PrgEnv-gnu`
Batch queueing system	Torque/Moab

The PGI compilers and tools are the recommended set although the GNU compiler and tools are also available.

* The default environment for new accounts loads the PGI MPI compilers into the PATH via the PrgEnv-pgi module. Should you wish to use the GNU MPI compilers instead, load the PrgEnv-gnu module. For more information on the available programming environments see news PrgEnv and the modules section below.

Modules

pacman has the modules package installed. This tool allows a user to quickly and easily switch between different versions of a package (e.g. compilers). The module package sets common environment variables used by applications such as PATH , MANPATH , LD_LIBRARY_PATH , etc. The PrgEnv-pgi module is loaded by default in the shell skeleton files for new accounts. The PrgEnv-pgi module loads the PGI compilers and MPI compiler wrappers into your PATH Alternately the PrgEnv-gnu module is provided for users who prefer to use the MPI compiler wrappers with the GNU compilers.

Command	Example Use	Purpose
`module avail`	`module avail`	lists all available modules for the system.
`module load pkg`	`module load PrgEnv-pgi`	loads a module file from the environment
`module unload pkg`	`module unload PrgEnv-pgi`	unloads a module file from the environment
`module list`	`module list`	displays the modules which are currently loaded.
`module switch old new`	`module switch PrgEnv-pgi PrgEnv-gnu`	replaces the module old with module new in the environment
`module purge`	`module purge`	unload all module settings, restoring the environment to the state before anymodules were loaded.

See "man module" for more information on the module command. See news modules for more information on using modules at ARSC.

Typically modules only need to be loaded on login. To change the default modules, modify your .cshrc (csh/tcsh users) or .kshrc (ksh) or .bashrc (sh/bash). Note that only one PrgEnv should be loaded at any given time.

e.g.

####
# Run common module commands
#
# load the PrgEnv-pgi module 
# (uses PGI compilers for MPI environment) 
module load PrgEnv-pgi 

# alternately load the PrgEnv-gnu module 
# (uses GNU compilers for the MPI environment) 
#module switch PrgEnv-pgi PrgEnv-gnu 
# list currently loaded modules 

module list

PGI - Compiling and Linking Fortran Programs

The PGI Fortran 90 compiler is:

pgf90

The Fortran 90 compiler with MPI support is:

mpif90

NOTE: the PrgEnv-pgi module must be loaded to use this compiler.

Here's a sample compiler command showing several common options:

pacman% pgf90 -O3 -g -0 reduce reduce.f90

see man pgf90 for additional information.

Portland Group Fortran 90 Compiler Options

Option	Description
-c	Generate intermediate object file but does not attempt to link.
-g	Adds information for debugging to the object file and/or executable.
-I<directory>	Tells the preprocessor to search in directory for include or module files.
-L<directory>	Tells the linker to search in directory for libraries.
-r8	Promotes REALs from the default size of 4 bytes to 8 bytes.
-i8	Promotes INTEGERs from the default size of 4 bytes to 8 bytes.
-O3	Higher level of optimization than -O2 (the default optimization level).
-fast	Higher optimization level than -O3
-Mipa	Tells the compiler to perform interprocedural analysis. Can be very time consuming toperform. This flag should also be used in both compilation and linking steps.
-Mconcur	Enables autoparallelization. Additional options can be used with -Mconcur to provide morefine-grained control of autoparallelization, see `man pgf90` for details.
-Minfo	Instructs the compiler to report optimizations that are made.
-Mneginfo	Instructs the compiler to report optimizations that are not made.
-mp	Enables parallelization via OpenMP directives.

Many other compiler options are available. Check the online manuals and if you have difficulty finding specific information, or contact User Support for additional help.

PGI - Compiling and Linking C/C++ Programs

The PGI C compilers are:

pgcc or mpicc

C++ uses the command:

pgCC or mpiCC

Here are sample compiler commands showing common options:

pacman% pgcc -O3 -o prog prog.c

pacman% pgCC -O3 -o prog prog.cpp

Details concerning the examples (see " man pgcc or man pgCC for more information):

Option	Description
-c	Generate intermediate object file but does not attempt to link.
-g	Adds information for debugging to the object file and/or executable.
-I<directory>	Tells the preprocessor to search in directory for include or module files.
-L<directory>	Tells the linker to search in directory for libraries.
-O3	Higher level of optimization than -O2 (the default optimization level).
-fast	Higher level optimization (default is -O2). This flag should be used in both compilation andlinking steps.
-Mipa	Tells the compiler to perform interprocedural analysis. This option can be very timeconsuming to perform. This flag should be used in both compilation and linking steps.
-Mconcur	Enables autoparallelization. Additional options can be used with -Mconcur to provide morefine-grained control of autoparallelization, see `man pgcc` or `man pgCC` for details.
-Minfo	Instructs the compiler to report optimizations that are made.
-Mneginfo	Instructs the compiler to report optimizations that are not made.
-mp	Enables parallelization via OpenMP directives.

Libraries

Libraries on pacman are generally available for both the Portland Group and GNU compiler suites. Most current versions of libraries and include files are available in the following directories:

PGI Compilers	Libraries:	`/usr/local/pgi/lib`
	Include Files:	`/usr/local/pgi/include`
GNU Compilers	Libraries:	`/usr/local/lib`
	Include Files:	`/usr/local/include`

The following Libraries are currently available on pacman:

Library	Notes
ACML	AMD Core Math Library including BLAS, LAPACK, and FFT routines
	Current PGI Version:	`/usr/local/pkg/acml/acml-4.4.0.pgi/pgi64/lib` (single threaded) `/usr/local/pkg/acml/acml-4.4.0.pgi/pgi64_mp/lib` (multi-threaded)
	Current GNU Version:	`/usr/local/pkg/acml/acml-4.0.0.gnu/gfortran64/lib` (single threaded) `/usr/local/pkg/acml/acml-4.0.0.gnu/gfortran64_mp/lib` (multi-threaded)
	Alternate Versions:	`/usr/local/pkg/acml`
	Example Fortran Compile Statement	pgf90 -I/usr/local/pkg/acml/acml-4.4.0.pgi/pgi64/include test.f90 pgf90 -L/usr/local/pkg/acml/acml-4.4.0.pgi/pgi64/lib -lacml -lacml_mv test.o -o test.exe
BLACS	Basic Linear Algebra Communication Subprograms
	Current PGI Version:	`/usr/local/pgi/lib`
	Current GNU Version:	`/usr/local/lib`
	Alternate Versions:	`/usr/local/pkg/blacs`
BLAS	See ACML
FFTW-2 and FFTW-3	Library for computing the Discrete Fourier Transform
	Current PGI Version:	`/usr/local/pgi/lib`
	Current GNU Version:	`/usr/local/lib`
	Alternate Versions:	`/usr/local/pkg/fftw`
GSL	GNU Scientific Mathematic Library for numerical C and C++ programs
	Current PGI Version:	`/usr/local/lib` (Use GNU Version)
	Current GNU Version:	`/usr/local/lib`
	Alternate Versions:	`/usr/local/pkg/gsl`
HDF-5	Hierarchical Data Format for transferring graphical and numerical data among computers
	Current PGI Version:	`/usr/local/pgi/lib`
	Current GNU Version:	`/usr/local/lib`
	Alternate Versions:	`/usr/local/pkg/hdf5`
LAPACK	See ACML
NetCDF	network Common Data Form
	Current PGI Version:	`/usr/local/pgi/lib`
	Current GNU Version:	`/usr/local/lib`
	Alternate Versions:	`/usr/local/pkg/netcdf`
	Example Fortran Compile Statement	pgf90 -c -I/usr/local/pgi/include test.f90 pgf90 -L/usr/local/pgi/lib -lnetcdff -lnetcdf test.o -o test.exe
	Alternate Fortran Compile Statement	pgf90 test.f90 -I/usr/local/pgi/include -L/usr/local/pgi/lib -lnetcdff -lnetcdf -o test.exe
ScaLAPACK	Scalable LAPACK, Library of high-performance linear algebra routines
	Current PGI Version:	`/usr/local/pgi/lib`
	Current GNU Version:	`/usr/local/lib`
	Alternate Versions:	`/usr/local/pkg/scalapack`

Pre-Processing and Post-Processing on Login Nodes

Several pacman login nodes have been configured with higher CPU limits and memory Limits to allow for pre-processing and post-processing as well as code development, testing and debugging activities.

The login nodes pacman3.arsc.edu through pacman9.arsc.edu allow for greater memory and CPU time use. For codes requiring significant memory, please verify there is light system load on the login node being used prior to running applications. The "top" command will display current memory use.

To increase CPU limit in seconds run:

# bash/ksh (set the limit to 8 hours; maximum is 259200 seconds or 3 days)

ulimit -St 28800

# csh/tcsh (set the limit to 8 hours; maximum is 259200 seconds or 3 days

limit cputime 28800

To increase CPU limit in seconds run:

# bash/ksh (set the limit to 16 GB; maximum is 33554432 KB or 32 GB)

ulimit -Sv 16777216

# csh/tcsh (set the limit to 16 GB; maximum is 33554432 KB or 32 GB)

limit vmemoryuse 16777216

Use caution when adding limits to the ~/.cshrc, ~/.login, ~/.profile, ~/.bash_profile, or ~/.bashrc as these files will also affect memory limits on compute nodes.

Running Interactive Jobs

You are encouraged to submit jobs to the Torque/Moab batch system, but spawning interactive jobs is also available. To use the batch system for spawning an interactive job (with an X11 support enabled), type the following command:

qsub -q debug -l nodes=1:ppn=16 -X -I

Standard error and standard output are displayed within the terminal, redirected to a file, or piped to another command using appropriate Unix shell syntax.

After your job has started, you may then run interactive commands on the compute node(s) Torque/Moab assigned to your session.

Running Batch Jobs

All production work on pacman is run through the Torque/Moab batch scheduler . Jobs run through the batch system execute the programs on pacman's compute nodes. The scheduler assigns a unique jobid to each individual job submission, conveniently saves stdout/stderr to a file(s) for each run, and will allow jobs to run after the user logs off.

A batch job is a shell script prefaced by a statement of resource requirements and instructions which Torque/Moab will use to manage the job.

Torque/Moab scripts are submitted for processing with the command, qsub .

As outlined below, five steps are common in the most basic batch processing:

1. Create a batch script:

Normally, users embed all Torque/Moab request options within the batch script.

In a batch script comforming to Torque/Moab syntax, all Torque/Moab options must precede all shell commands. Each line containing a Torque/Moab option must commence with the character string, " #PBS followed by one or more spaces, followed by the option.

Torque/Moab scripts begin execution from your home directory and thus, the first executable shell script command is a usually a " cd " to the work or run directory. The environment variable PBS_O_WORKDIR is set to the directory from which the script was submitted.

Torque Script- MPI Example requesting 4 nodes (reserving all 16 processors on each node)

Script	Notes
#!/bin/bash #PBS -q standard_16 #PBS -l walltime=8:00:00 #PBS -l nodes=4:ppn=16 #PBS -j oe cd $PBS_O_WORKDIR mpirun --mca mpi_paffinity_alone 1 ./myprog	Select the shell. The -q option requests to run the job in the standard queue Walltime requests the job be allowed to run for a maximum of 8 hours. Requests 4 nodes and all 16 processes on each node. The -j option joins output and error messages into one file. Change to the initial working directory ($PBS_O_WORKDIR). Execute the program by calling mpirun and passing the executable name.

Script

Notes

#!/bin/bash
#PBS -q standard_16

#PBS -l walltime=8:00:00 

#PBS -l nodes=4:ppn=16

#PBS -j oe

cd $PBS_O_WORKDIR

mpirun --mca mpi_paffinity_alone 1 ./myprog

Select the shell.
The -q option requests to run the job 
in the standard queue
Walltime requests the job be allowed 
to run for a maximum of 8 hours.  
Requests 4 nodes and all 16 processes on 
each node.  
The -j option joins output and error messages 
into one file.
Change to the initial working directory 
($PBS_O_WORKDIR).
Execute the program by calling mpirun and passing 
the executable name.

Torque Script- MPI Example requesting 2 processors on each of 6 nodes

Script Notes

Script	Notes
#!/bin/bash #PBS -q standard_16 #PBS -l walltime=8:00:00 #PBS -l nodes=6:ppn=16 #PBS -j oe cd $PBS_O_WORKDIR mpirun -np 12 -npernode 2 ./myprog	Select the shell. The -q option requests to run the job in the standard queue Walltime requests the job be allowed to run for a maximum of 8 hours. Reserves all 16 processors on each of 6 nodes. The -j option joins output and error messages into one file. Change to the initial working directory. Execute the program by calling mpirun.Note this command only runs 2 tasks on each node despite all 16 processors being reserved. This is a technique used when individual processes demand very large memory resources. The `-np` flag specifies the total number of executable copies run on the reserved nodes.


#!/bin/bash
#PBS -q standard_16
#PBS -l walltime=8:00:00 
#PBS -l nodes=6:ppn=16
#PBS -j oe

cd $PBS_O_WORKDIR

mpirun -np 12 -npernode 2 ./myprog

Select the shell.

The -q option requests to run the
job in the standard queue Walltime
requests the job be allowed to run
for a maximum of 8 hours.  Reserves
all 16 processors on each of 6
nodes.  The -j option joins output
and error messages into one file.
 
Change to the initial working
directory.

Execute the program by calling
mpirun.Note this command only runs
2 tasks on each node despite all 16
processors being reserved.  This is
a technique used when individual
processes demand very large memory
resources. The 
-np flag specifies the
total number of executable copies
run on the reserved nodes.

Torque Script- OpenMP Example requesting all 16 processors on 1 node

Script	Notes
#!/bin/bash #PBS -q standard_16 #PBS -l walltime=8:00:00 #PBS -l nodes=1:ppn=16 #PBS -j oe cd $PBS_O_WORKDIR export OMP_NUM_THREADS=16 ./myprog	Select the shell. The -q option requests to run the job in the standard queue. Walltime requests the job be allowed to run for a maximum of 8 hours. Requests 1 node, all 16 processes. The -j option joins output and error messages into one file. Change to the initial working directory. Assign all 16 processes to be available for the OpenMP program. Execute the program.

Script

Notes


#!/bin/bash
#PBS -q standard_16
#PBS -l walltime=8:00:00 
#PBS -l nodes=1:ppn=16
#PBS -j oe

cd $PBS_O_WORKDIR
export OMP_NUM_THREADS=16

./myprog

Select the shell.

The -q option requests to run the
job in the standard queue.

Walltime requests the job be
allowed to run for a maximum of 8
hours.  Requests 1 node, all 16
processes.  The -j option joins
output and error messages into one
file.
 
Change to the initial working
directory.  Assign all 16 processes
to be available for the OpenMP
program.

Execute the program.

Torque Script- Serial Example requesting 1 processor on a shared node

Script	Notes
#!/bin/bash #PBS -q shared #PBS -l walltime=8:00:00 #PBS -l nodes=1:ppn=1 #PBS -j oe cd $PBS_O_WORKDIR ./myprog	Select the shell. The -q option requests to run the job in the shared queue. Walltime requests the job be allowed to run for a maximum of 8 hours. Requests 1 node and 1 processor. The -j option joins output and error messages into one file. Change to the initial working directory. Execute the serial program on a shared node.

Script

Notes


#!/bin/bash
#PBS -q shared
#PBS -l walltime=8:00:00 
#PBS -l nodes=1:ppn=1
#PBS -j oe

cd $PBS_O_WORKDIR

./myprog

Select the shell.

The -q option requests to run the
job in the shared queue.  Walltime
requests the job be allowed to run
for a maximum of 8 hours.  Requests
1 node and 1 processor.  The -j
option joins output and error
messages into one file.
 
Change to the initial working
directory.

Execute the serial program on a
shared node.

Torque Script- Transfer Queue Example copying data from $ARCHIVE

Script	Notes
#!/bin/bash #PBS -q transfer #PBS -l walltime=8:00:00 #PBS -l nodes=1:ppn=1 #PBS -j oe cd $PBS_O_WORKDIR batch_stage $ARCHIVE/mydataset/* cp -r $ARCHIVE/mydataset/* . \|\| exit 1 qsub mpi_job.pbs	Select the shell. The -q option requests to run the job in the transfer queue. Walltime requests the job be allowed to run for a maximum of 8 hours. Requests 1 node and 1 processor. The -j option joins output and error messages into one file. Change to the initial working directory. Bring $ARCHIVE files online with batch_stage. Copy files to working directory. Submit computational job.

Script

Notes


#!/bin/bash
#PBS -q transfer
#PBS -l walltime=8:00:00 
#PBS -l nodes=1:ppn=1
#PBS -j oe

cd $PBS_O_WORKDIR

batch_stage $ARCHIVE/mydataset/*
cp -r $ARCHIVE/mydataset/* . || exit 1

qsub mpi_job.pbs

Select the shell.

The -q option requests to run the
job in the transfer queue.
Walltime requests the job be
allowed to run for a maximum of 8
hours.  Requests 1 node and 1
processor.  The -j option joins
output and error messages into one
file.

Change to the initial working
directory.

Bring $ARCHIVE files online with
batch_stage.  Copy files to working
directory.

Submit computational job.

2. Submit a batch script to Moab/Torque, using qsub

The script file can be given any name. If the above sample were named myprog.pbs, it would be submitted for processing with the command:

qsub myprog.pbs

3. Monitor the job

To check the status of the submitted Torque/Moab job, execute this command:

qstat -a

4. Delete the job

Given its Torque/Moab identification number (returned when you " qsub the job and shown in " qstat -a output), you can delete the job from the batch system with this command:

qdel <PBS-ID>

5. Examine Output

When the job completes, Torque/Moab will save the stdout and stderr from the job to a file in the directory from which it was submitted. These files will be named using the script name and Torque/Moab identification number. For example,

myprog.pbs.o<PBS-ID>

Torque/Moab Queues

List all available queues with the command qstat -Q . List details on any queue, for instance, " standard " with the command qstat -Qf standard . You may also read news queues for information on all queues, but note that the most current information is always available using the qstat commands.

Currently, these are the available queues:

Queue	Purpose
background	For projects with little or no remaining allocation. This queue has the lowest priority,however projects running jobs in this queue do not have allocation deducted. Runs on 16core nodes only.
bigmem	Queue for jobs requiring significant memory. Runs on 32 cores nodes with 256GB of memory.
debug	Quick turn-around queue for debugging work. Runs on 12 core and 16 core nodes only..
gpu	Jobs run on nodes with GPU accelerators. Nodes have 8 cores and 1 NvidiaC2050 GPU card.
shared	For sharing a node with other serial or small parallel jobs. Runs on 12 core nodes only.
standard	General use by all allocated users routes to standard_16
standard_4	General use by all allocated users. Runs on 4 core nodes only.
standard_12	General use by all allocated users. Runs on 12 core nodes only.
standard_16	General use by all allocated users. Runs on 16 core nodes only.
transfer	For transferring data to and from $ARCHIVE. Be sure to bring all $ARCHIVE files online usingbatch_stage prior to the file copy.

MPI Run Time Options

The OpenMPI MPI stack has a number of run-time options which affect the run time behavior of MPI applications. Below are a few useful options

Enable CPU Affinity

Standard MPI applications not using OpenMP or pthreads may see performance improvement if CPU Affinity is enabled. The flag "--mca mpi_paffinity_alone 1" enables CPU affinity.

mpirun -np 16 --mca mpi_paffinity_alone 1 ./a.out

MPI jobs using the standard_4 queue may see performance may see noticable performance improvements by enabling CPU affinity.

Running Fewer Tasks Per Node

At times it may be adventageous to run with fewer tasks per node while still requesting all node on thenode with the torque script. The "--npernode" option allows for finer control of task placement.

mpirun -np 4 --npernode 2 ./a.out

Charging Allocation Hours to an Alternate Project

Users with membership in more than one project should select which project to charge allocation hours towards. The directive for selecting which project is the "-W group_list" Torque/Moab option. If the "-W group_list" option is not specified within a user's Torque/Moab script, the account charged will default to the user's primary group (i.e. project).

The following is an example "-W group_list" statement.

#PBS -W group_list=proja

The " -W group_list option can also be used on the command line, e.g.

pacman1 % qsub -W group_list=proja script.bat

Each project has a corresponding UNIX group, therefore the "groups" command will show all projects (or groups) of which you are a member.

pacman1 % groups proja projb

Without the "-W group_list" Torque/Moab option, allocation hours would be charged to proja by default, but could be charged to projb by setting " -W group_list=projb in the Torque/Moab script.

Monitoring Project Allocation Usage

Each project is allocated time on pacman. The "show_usage" command can be used to monitor allocation use for projects.

pacman1 % show_usage 

            ARSC - Subproject Usage Information (in CPU Hours)
                     As of 04:24:01 hours ADT 24 May 2012
          For Fiscal Year 2012 (01 October 2011 - 30 September 2012)
                  Percentage of Fiscal Year Remaining: 35.62% 

                          Hours      Hours      Hours      Percent  Background
System     Subproject     Allocated  Used       Remaining  Remaining Hours Used
========== ============== ========== ========== ========== ========= ==========
pacman     proja             20000.00       0.00   20000.00   100.00%       0.00
pacman     projb            300000.00  195887.32  104112.68    34.70%   11847.78

Online Manuals and Documentation

External Links

Portland Group Compilers: http://www.pgroup.com/resources/docs.htm
GNU Compilers: http://gcc.gnu.org/

News Items

Some documentation is available directly on pacman via the "news" command. This documentation cover a variety of topics, including availability of software, sample job scripts, and more.

Many news items available on the system through "news <topic>" are also available on our website:

pacman News Items

More Information

General information on ARSC and its other resources is available in a number of forms

Online, on the web:
The ARSC web site covers policies, research, events, and technical information.
The ARSC HPC Users' Newsletter
This bi-weekly newsletter is designed to inform users of HPC systems.
Training Classes
The ARSC training schedule is maintained on our web site and you are always welcome to schedule a one-on-one work session with ARSC specialists or consultants.
Consultants
Contact User Support for more information.

Introduction to Using the Penguin Computing AMD Opteron Cluster

Contents

Introduction

"Pacman"

Operating System / Shells

System News & Status

Storage

Connecting to Pacman

Sample Code Repository ($SAMPLES_HOME)

Available Software

User Installed Software

Policies

Commercial & Export Controlled Software

Permissions

Storage

Recommendations

Create a README File

Library Naming Scheme

Module Files

Parallel Programming Models

Programming Environment

Modules

PGI - Compiling and Linking Fortran Programs

Portland Group Fortran 90 Compiler Options

PGI - Compiling and Linking C/C++ Programs

Libraries

Pre-Processing and Post-Processing on Login Nodes

Running Interactive Jobs

Running Batch Jobs

Torque Script- MPI Example requesting 4 nodes (reserving all 16 processors on each node)

Torque Script- MPI Example requesting 2 processors on each of 6 nodes

Torque Script- OpenMP Example requesting all 16 processors on 1 node

Torque Script- Serial Example requesting 1 processor on a shared node

Torque Script- Transfer Queue Example copying data from $ARCHIVE

MPI Run Time Options

Charging Allocation Hours to an Alternate Project

Monitoring Project Allocation Usage

Online Manuals and Documentation

More Information