Introduction to Using the Cray XT5 (Pingo)

Contents

• Introduction
• "Pingo"
• Operating System / Shells
• System News & Status
• Storage
• Sample Code Repository ($SAMPLES_HOME)
• Parallel Programming Models
• Programming Environment
• Modules
• PGI - Compiling and Linking Fortran Programs
• PGI - Compiling and Linking C/C++ Programs
• PathScale - Compiling and Linking Fortran Programs
• PathScale - Compiling and Linking C/C++ Programs
• PathScale - Other Tools
• PathScale - Runtime Environment Variables
• Libraries
• Performance Analysis
• Running Interactive Jobs
• Running Batch Jobs
  • PBS Script- MPI
  • PBS Script- OpenMP
  • PBS Script- Data staging script
• Advance Reservation Service (ARS)
• Allocations
• Monitoring Project CPU Hour Usage
• Charging Allocation Hours to an Alternate Project
• Online Manuals and Documentation
• More Information

Introduction

The Arctic Region Supercomputing Center (ARSC) operates a Cray XT5 (pingo) running UNICOS/lc 2.1.

"Pingo"

• 2- Cray XT external service login nodes.
  • The external service login nodes are the preferred system login nodes and offer extended memory for intensive pre and post data processing.
  • The external service login nodes are called:
    • "pingob.arsc.edu"
    • "pingoc.arsc.edu"
  • Each external service login node has:
    • 128 GB of shared memory per node. (8 GB per core)
    • 4 - quad core AMD Opteron Processors
    • 1 - 10 Gigabit Ethernet Interconnect
• 2- Cray XT standard login nodes.
  • The standard login nodes are called:
    • "pingo1.arsc.edu"
    • "pingo2.arsc.edu"
  • Each standard login node has:
    • 8 GB of shared memory per node. (4 GB per core)
    • 1 - dual core AMD Opteron Processors
    • 1 - Seastar2+ Interconnect Module
• 432- Cray XT5 compute nodes with:
  • 32 GB of shared memory per node (4 GB per core)
  • 2 - quad core 2.3 GHz AMD Opteron processors per node.
  • 1 - Seastar2+ Interconnect Module per node.
• Cray Seastar2+ Interconnect between all compute and standard login nodes.
• A Lustre global shared filesystem accessible from all nodes.

Operating System / Shells

The operating system on pingo is UNICOS/lc 2.1 based on the SuSE Linux Enterprise Server.

The following shells are available on pingo:

• sh
• ksh
• bash
• csh
• tcsh

If you would like to have your login shell changed, please contact User Support.

System News & Status

System news is available via the news command when logged on to Pingo. For example, the command "news queues" gives news about the current queue configuration. System status and non sensitive news items are available on the web.

• Pingo System Status
• Pingo News Items

Storage

Several environment variables are defined to point to the available storage on Pingo.

Name	Notes	Quota1	Purge Policy	Back Up Policy
$HOME	$HOME directories are intended for locally compiled executables and libraries, dot files, and small data sets.	1GB2	not purged	backed up
$WORKDIR	$WORKDIR is a high performance parallel Lustre file system available from all nodes on pingo.	Currently there are no quotas enabled	30 day purge policy3	not backed up
$ARCHIVE_HOME	long-term storage, not available from compute nodes	none	not purged	backed up

NOTES:

1. Requests for increased quotas should be sent to User Support.
2. The qcheck command should be used to check storage usage and quota information.
3. The getPurgable command can be used to determine which files are susceptible to purging (see getPurgable -help for available options).

See http://www.arsc.edu/support/howtos/storage.html and for more information on the storage environment at ARSC.

Sample Code Repository ($SAMPLES_HOME)

The $SAMPLES_HOME directory on pingo contains a number of examples including, but not limited to:

• PBS scripts for MPI, OpenMP and Hybrid applications
• Examples using common libraries

An index of items available in the Sample Code Repository is available on the web, however you must login to pingo to access these examples.

If you would like to recommend a new example, please contact User Support.

Parallel Programming Models

Several types of parallelism can be exploited on pingo using different programming models and methods. These are listed in the table below.

Hardware Level	Model	Description
Shared-memory node	Auto	When enabled, automatic parallelization can be used on shared-memory programs. Basic loops can be automatically parallelized, but for additional performance improvements, try using OpenMP directives. This option is available with the PGI and Pathscale environments.
Shared-memory node	OpenMP	This is a form of explicit parallel programming in which the programmer inserts directives into the program to spawn multiple shared-memory threads, typically at the loop level. It is a common method of parallelization, portable, and relatively easy to implement. On the downside, it requires shared memory which limits scaling to the number of processors on just one node. OpenMP can be used in conjunction with autoparallelization and is available with the PGI, Pathscale, and GNU environments.
Distributed memory	MPI	This is the most common and portable method for parallelizing codes for scalable distributed memory systems. MPI is a library of subroutines for message passing, collective operations, and other forms of inter-processor communication. The programmer is responsible for implementing data distribution, synchronization, and reassembly of results using explicit MPI calls. Using MPI, the programmer can largely ignore the physical organization of processors into nodes and simply treat the system as a collection of independent processors. MPI is available with the PGI, Pathscale, and GNU environments.
Distribute memory	SHMEM	This library supports one-sided communications between tasks.

Programming Environment

Item	PGI	PathScale	GNU Compilers
Fortran 77/90/95 (Compute Node)	ftn	ftn	ftn
C compiler (Compute Node)	cc	cc	cc
C++ compiler (Compute Node)	CC	CC	CC
Fortran 77/90/95 (Login Node Only)	pgf90	pathf90	gfortran
C compiler (Login Node Only)	pgcc	pathcc	gcc
C++ compiler (Login Node Only)	pgCC	pathCC	g++
Debuggers	totalview	totalview	totalview
Performance Analysis	CrayPat	CrayPat	CrayPat
Default module (*)	PrgEnv-pgi (default)	PrgEnv-pathscale	PrgEnv-gnu
Batch queueing system	PBS Professional 9.2

The PGI programming environment is loaded by default for all accounts.

Modules

Pingo has the modules package installed. This tool allows a user to quickly and easily switch between different versions of a package (e.g. compilers). The module package sets common environment variables used by applications such as PATH, MANPATH, etc. The PrgEnv-pgi module is loaded by default for all accounts. The PrgEnv-pgi module loads the current version of the PGI compilers with MPI support into your PATH.

Command	Example Use	Purpose
module avail	module avail	lists all available modules for the system.
module load pkg	module load PrgEnv-pathscale	loads a module file from the environment
module unload pkg	module unload PrgEnv-pgi	unloads a module file from the environment
module list	module list	displays the modules which are currently loaded.
module switch old new	module switch PrgEnv-pgi PrgEnv-gnu	replaces the module old with module new in the environment
module purge	module purge	unload all module settings, restoring the environment to the state before any modules were loaded.

See "man module" for more information on the module command. See news modules for more information on using modules at ARSC.

Typically modules only need to be loaded on login. This can be done by placing module load commands in your .cshrc (csh/tcsh users), .bashrc (bash) or .profile (ksh).

PGI - Compiling and Linking Fortran Programs

The Fortran 90 compiler is:

ftn

This compiler has OpenMP, MPI, and Shmem support.

pingo% ftn -O3 reduce.f90

Portland Group Fortran 90 Compiler Options

Option	Description
-c	Generate intermediate object file but does not attempt to link.
-g	Adds information for debugging to the object file and/or executable.
-I<directory>	Tells the preprocessor to search in directory for include or module files.
-L<directory>	Tells the linker to search in directory for libraries.
-r8	Promotes REALs from the default size of 4 bytes to 8 bytes.
-i8	Promotes INTEGERs from the default size of 4 bytes to 8 bytes.
-default64	Passes the -i8 and -r8 options to the compiler.
-O3	Higher level of optimization than -O2 (the default optimization level).
-fast	Higher optimization level than -O3
-tp barcelona-64	Use optimizations for the AMD Barcelona Quad Core processor (default).
-Mipa	Tells the compiler to perform interprocedural analysis. Can be very time consuming to perform. This flag should also be used in both compilation and linking steps.
-Mconcur	Enables autoparallelization. Additional options can be used with -Mconcur to provide more fine-grained control of autoparallelization, see man pgf90 for details.
-mp=nonuma	Enables parallelization via OpenMP directives.
-Minfo	Displays useful information to stderr, see man pgf90 for details.
-Mneginfo	Displays information on why a particular optimization was not performed.

Many other compiler options are available. View the man pages for additional information:

• man pgf90
• man ftn

PGI - Compiling and Linking C/C++ Programs

The C compilers are called using the following command:

cc

Compiling in C++ requires the command:

CC

These compilers have OpenMP, MPI, Shmem and Pthreads support.

Here are sample compiler commands showing common options:

pingo% cc -O3 -o prog prog.c

pingo% CC -O3 -o prog prog.cpp

Option	Description
-c	Generate intermediate object file but does not attempt to link.
-g	Adds information for debugging to the object file and/or executable.
-I<directory>	Tells the preprocessor to search in directory for include or module files.
-L<directory>	Tells the linker to search in directory for libraries.
-O3	Higher level of optimization than -O2 (the default optimization level).
-fast	Higher level optimization (default is -O2). This flag should be used in both compilation and linking steps.
-Mipa	Tells the compiler to perform interprocedural analysis. This option can be very time consuming to perform. This flag should be used in both compilation and linking steps.
-Mconcur	Enables autoparallelization. Additional options can be used with -Mconcur to provide more fine-grained control of autoparallelization, see man pgcc or man pgCC for details.
-mp=nonuma	Enables parallelization via OpenMP directives.
-Minfo	Displays useful information to stderr, see man pgf90 for details.
-Mneginfo	Displays information on why a particular optimization was not performed.

Many other compiler options are available. View the man pages for additional information:

• man pgcc
• man pgCC
• man cc
• man CC

PathScale - Compiling and Linking Fortran Programs

The Fortran 90 compiler is:

ftn

This compiler has OpenMP, MPI, Shmem and Pthreads support.

Here's a sample compiler command showing several common options:

pingo% ftn -O3 reduce.f90

PathScale Fortran 90 Compiler Options

Option	Description
-show-defaults	List default compiler options for the compiler and exits.
-c	Generate intermediate object file but does not attempt to link.
-g	Adds information for debugging to the object file and/or executable.
-I<directory>	Tells the preprocessor to search in directory for include or module files.
-L<directory>	Tells the linker to search in directory for libraries.
-r8	Promotes REALs from the default size of 4 bytes to 8 bytes.
-i8	Promotes INTEGERs from the default size of 4 bytes to 8 bytes.
-default64	Passes the -i8 and -r8 options to the compiler.
-O3	Higher level of optimization than -O2 (the default optimization level).
-cpp	Preprocess files with the C preprocessor. Enabled by default for files ending in .F,.F90, or .F95.
-ftpp	Preprocess files with the Fortran preprocessor. Useful when portions of the Fortran code could be misinterpreted as C preprocessor directives (e.g. "//")
-O3	Higher level of optimization than -O2 (the default optimization level).
-O3 -OPT:Ofast	Higher optimization level than -O3
-ipa	Tells the compiler to perform interprocedural analysis. Can be very time consuming to perform. This flag should also be used in both compilation and linking steps. Not recommended for programs over 100,000 lines for the current compiler release.
-intrinsic=PGI	Enables intrinsic functions that are available in the PGI compiler which are not ANSI standard (e.g. rand)
-apo	Enables autoparallelization.
-mp	Enables parallelization via OpenMP directives.

Many other compiler options are available. View the man pages for additional information:

• man pathf90
• man eko
• man ftn

PathScale - Compiling and Linking C/C++ Programs

The C compilers are called using the following command:

cc

Compiling in C++ requires the command:

CC

This compiler has OpenMP, MPI, Shmem and Pthreads support.

Here are sample compiler commands showing common options:

pingo% cc -O3 -o prog prog.c

pingo% CC -O3 -o prog prog.cpp

Option	Description
-show-defaults	List default compiler options for the compiler and exits.
-c	Generate intermediate object file but does not attempt to link.
-g	Adds information for debugging to the object file and/or executable.
-I<directory>	Tells the preprocessor to search in directory for include or module files.
-L<directory>	Tells the linker to search in directory for libraries.
-O3	Higher level of optimization than -O2 (the default optimization level).
-Ofast	Higher level optimization (default is -O2). This flag should be used in both compilation and linking steps.
-ipa	Tells the compiler to perform interprocedural analysis. This option can be very time consuming to perform. This flag should be used in both compilation and linking steps. Not recommended for programs over 100,000 lines for the current compiler release.
-apo	Enables autoparallelization.
-mp	Enables parallelization via OpenMP directives.

Many other compiler options are available. View the man pages for additional information:

• man pathcc
• man pathCC
• man eko
• man cc
• man CC

PathScale - Other Tools

The following tools are available with the PathScale compiler suite. 

Command	Notes
assign	Alters the way Fortran I/O is performed.
explain	Provides extended information for compiler error and diagnostic messages.
pathhow-compiled	Shows the compiler options used to produce an object file. e.g.   pingo% pathhow-compiled sample.o

PathScale - Runtime Environment Variables

Below are some environment variables which affect the run time function of executables compiled with the PathScale compilers. 

Environment Variable	Languages	Description
PSC_OMP_AFFINITY	OpenMP codes	PSC_OMP_AFFINITY specifies whether or not OpenMP tasks should be bound to particular processors. Valid values are TRUE or FALSE
PSC_FDEBUG_ALLOC	Fortran codes	PSC_FDEBUG_ALLOC specifies the initial value to be placed in Fortran memory allocations for debugging purposes. Valid values are ZERO, NaN or NaN8
FILENV	Fortran codes	FILENV specifies the file name which contains assign options to be used by a Fortran program.

Libraries

Libraries on pingo are generally available for the PathScale, Portland Group and GNU compiler suites. Most current versions of libraries and include files are available in the following directories:

Pathscale Compilers	Libraries:	/usr/local/pathscale/lib
	Include Files:	/usr/local/pathscale/include
PGI Compilers	Libraries:	/usr/local/pgi/lib
	Include Files:	/usr/local/pgi/include
GNU Compilers	Libraries:	/usr/local/gnu/lib or /usr/local/lib
	Include Files:	/usr/local/gnu/include or /usr/local/include

Performance Analysis

The preferred performance analysis tool on Pingo is CrayPat. Here are the basic steps required to build an instrumented executable

1. Load the "xt-craypat" module
   
   pingo1% module load xt-craypat
   
   
2. Recompile the code as you normally would to generate an executable.
   
   pingo1% ftn mycode.f90 -o mycode
   
   
3. Use the pat_build command to generate an instrumented executable.
   
   pingo1% pat_build -g mpi -u mycode
   
   This generates an instrumented executable called mycode+pat. Here the "-g" option enables "mpi" tracegroup. See "man pat_build" for available tracegroups.
   
   
4. Run the instrumented executable with aprun via PBS.
   
   aprun -n 80 ./mycode+pat
   
   This generates an instrumented output file (e.g. mycode+pat+2007-12tdt.xf).
   
   
5. Use pat_report to display the statistics from the output file
   
   pingo1% pat_report mycode+pat+2007-12tdt.xf > mycode.pat_report
   
   

Additional profiling options are available. See "man pat_build" for additional instrumentation options.

Running Interactive Jobs

You may use the batch system to spawn an interactive job by using the following command:

qsub -q debug -l mppwidth=16 -I

Once your job is started, you may run interactive commands on the compute node(s) PBS assigned your session. For example:

aprun -n 16 ./myprog

Running Batch Jobs

All production work on pingo is run through the PBS batch scheduler. The batch system offers greater memory and wall time limits than on interactive nodes. Jobs run through the batch system also convieniently save stdout/stderr to a file(s) for each run, and will continue after logging off.

A batch job is a shell script prefaced by a statement of resource requirements and instructions which PBS will use to manage the job.

As outlined below, five steps are common in the most basic batch processing:

1. Create a batch script:
   

   Normally, users embed all PBS request options within the batch script.

   In a batch script comforming to PBS syntax, all PBS options must precede all shell commands. Each line containing a PBS option must commence with the character string, "#PBS" followed by one or more spaces, followed by the option.

   PBS scripts begin execution from your home directory and thus, the first executable shell script command is a usually a "cd" to the work or run directory. The environment variable PBS_O_WORKDIR is set to the directory from which the script was submitted.

PBS Script- MPI Example

Script	Notes
#!/bin/bash #PBS -q standard #PBS -l mppwidth=32 #PBS -l walltime=8:00:00 #PBS -j oe cd $PBS_O_WORKDIR aprun -n 32 ./myprog	Select the shell. The -q option requests the queue to run in. Requests 32 cores. Requests that the job be allowed to run 8 hours. The -j option joins output and error files. Change to the directory to initial working directory. Run the mpi program. A copy of this script is also available on Pingo at $SAMPLES_HOME/jobSubmission

Open Script in New Window


PBS Script- OpenMP Example

Script	Notes
#!/bin/bash #PBS -q standard #PBS -l mppwidth=8 #PBS -l walltime=8:00:00 #PBS -j oe cd $PBS_O_WORKDIR export OMP_NUM_THREADS=8 aprun -n 1 -d 8 ./myprog	Select the shell. The -q option requests the queue to run in. Requests 8 cores. Requests that the job be allowed to run 8 hours. The -j option joins output and error files. Change to the directory to initial working directory. Set the number of OpenMP threads to use. Run the OpenMP program using aprun. A copy of this script is also available on Pingo at $SAMPLES_HOME/jobSubmission

Open Script in New Window


PBS Script- Data Staging Script

Script

Notes

#!/bin/bash #PBS -q transfer #PBS -l walltime=4:00:00 #PBS -j oe   cd $PBS_O_WORKDIR   find $ARCHIVE_HOME/mydataset/ -offline | batch_stage -i   cp -r $ARCHIVE_HOME/mydataset/* . || exit 1   qsub mpi_job.pbs

Select the shell. The -q option requests the queue to run in. Requests that the job be allowed to run 4 hours. The -j option joins output and error files.   Change to the directory to initial working directory.   Bring offline files from tape back online to disk.   Copy files from long term storage to current working directory.   Submit compute job A copy of this script is also available on Pingo at $SAMPLES_HOME/jobSubmission Please note that $ARCHIVE_HOME is not available from the compute nodes. This example script accesses files from $ARCHIVE_HOME by running on the login node.

Open Script in New Window


2. Submit a batch script to PBS, using qsub:
   

   The script file can be given any name. If the above sample were named myprog.pbs, it would be submitted for processing with the command

   qsub myprog.pbs

3. Monitor the job
   

   To check the status of the submitted PBS job, execute this command:

   qstat -a

4. Delete the job
   

   Given its PBS identification number (returned when you "qsub" the job and shown in "qstat -a" output), you can delete the job from the batch system with this command:

   qdel <PBS-ID>

5. Examine Output
   

   When the job completes, PBS will save the stdout and stderr from the job to a file in the directory from which it was submitted. These files will be named using the script name and PBS identification number. For example,

   myprog.pbs.o<PBS-ID>

PBS Queues

List all available queues with the command qstat -Q. List details on any queue, for instance, "standard," with the command qstat -Qf standard. You may also read news queues for information on all queues, but note that the most current information is always available using the qstat commands.

Currently, these are the available queues:

Queue	Purpose
background	For projects with little or no remaining allocation. This queue has the lowest priority, however projects running jobs in this queue do not have allocation deducted.
challenge	For projects with challenge status. (Limited Access)
transfer	Provides access to long term storage (i.e. $ARCHIVE_HOME)
debug	Quick turn-around queue for debugging work.
high	(Limited Access)
special	For jobs which do not fit into the normal queue limits for the system. (Limited access)
standard	General use by all allocated users.
urgent	(Limited Access)

 

Advance Reservation Service (ARS)

The Advance Reservation Service (ARS) is available to HPCMP users with non ARSC.EDU principles. The ARS allows users to reserve a portion of the system for future debugging, development, or production work. Projects are charged for all CPU time used by a reservation.
Reservations may be created using the ARS Website.

Other References:

• news ARS
• https://reservation.hpc.mil

 

Allocations

Projects are allocated in CPU hours. Jobs running in all queues except the "background" queue will be charged for all CPU time dedicated to a job.

Monitoring Project CPU Hour Usage

The show_usage command displays the remaining CPU Hour allocation for each project a user belongs to:

pingob % show_usage 

            ARSC - Subproject Usage Information (in CPU Hours)
                     As of 04:4:01 hours ADT 4 May 2010
          For Fiscal Year 2010 (01 October 2009 - 30 September 2010)
                  Percentage of Fiscal Year Remaining: 41.10% 

                          Hours      Hours      Hours      Percent  Background
System     Subproject     Allocated  Used       Remaining  Remaining Hours Used
========== ============== ========== ========== ========== ========= ==========
pingo      projecta         50000.00   25750.61   24249.39    48.50%       0.15
pingo      projectb          1000.00       0.00    1000.00   100.00%       0.00

 

Charging Allocation Hours to an Alternate Project

Users with membership in more than one project should select which project to charge allocation hours towards. The directive for selecting which project is the "-W group_list" PBS option. If the "-W group_list" option is not specified within a user's PBS script, the account charged will default to the user's primary group (i.e. project).

The following is an example "-W group_list" statement.

   #PBS -W group_list=proja

The "-W group_list" option can also be used on the command line, e.g.

   pingo % qsub -W group_list=proja script.bat

Each project has a corresponding UNIX group, therefore the "groups" command will show all projects (or groups) of which you are a member.

   pingo % groups
   proja projb

Without the "-W group_list" PBS option, allocation hours would be charged to proja by default, but could be charged to projb by setting "-W group_list=projb" in the PBS script.

Online Manuals and Documentation

External Links

• Online Cray Documents: http://docs.cray.com
• PGI Compilers: http://www.pgroup.com/resources/docs.htm
• PathScale Compilers: http://pathscale.com/docs.html

News Items

Some documentation is available directly on pingo via the "news" command. This documentation cover a variety of topics, including availability of software, sample job scripts, and more.

Many news items available on the system through "news <topic>" are also available on our website:

• Pingo News Items

More Information

General information on ARSC and its other resources is available in a number of forms

• Online, on the web:

  The ARSC web site covers policies, research, events, and technical information.

• The ARSC HPC Users' Newsletter

  This bi-weekly newsletter is designed to inform users of HPC systems. The current and all past issues are available at: http://www.arsc.edu/support/news/HPCnews.shtml.

• Training Classes

  The ARSC training schedule is maintained on our web site and you are always welcome to schedule a one-on-one work session with ARSC specialists or consultants.

• Questions?

  Contact User Support for more information.

Arctic Region Supercomputing Center
PO Box 756020, Fairbanks, AK 99775 | voice: 907-450-8600 | email:

home | search | about | support | news | science | resources