![[Menu Bar]](/images/header.gif)
The Arctic Region Supercomputing Center (ARSC) operates an IBM p655+/p690+ supercomputer (iceberg). This system is available to allocated ARSC users only and access requires proof of citizenship for all users regardless of country.
The two p690+ nodes are ideal for large memory OpenMP programs. The bulk of the computing capability consists of p655+ servers. These servers can be used as small 8-way shared memory systems, but can also be used for large message passing jobs using multiple servers per job.
The operating system on iceberg is IBM's Unix-variant, AIX. The following shells are available on iceberg:
sh (Bourne Shell)ksh (Korn Shell) defaultbash (Bourne-Again Shell)csh (C Shell)tcsh (Turbo C Shell)If you would like to have your login shell changed, please contact User Support.
System news is available via the news command when logged on to iceberg. System status and news items are
available on the web.
On all ARSC systems, several storage related environment variables are set by default. Please use these environment variables:
| Name | Purpose | Quota1 | Purge Policy | Back Up Policy |
|---|---|---|---|---|
| $HOME | "dot" and other small files | 100MB | not purged | backed up |
| $WORKDIR2 $WRKDIR3 |
compilation, program execution, storage for active program data files | 20GB Initial | purged | not backed up |
| $ARCHIVE_HOME2 $ARCHIVE3 |
long-term storage | none | not purged | backed up |
$WRKDIR and $ARCHIVE are
deprecated versions of the environment variables and may be phased out in the near future.Requests for quota increases should be sent to User Support.
See http://www.arsc.edu/support/howtos/storage.html for more information on storage policies at ARSC.
The $SAMPLES_HOME directory
on iceberg contains a number of examples including, but not limited to:
A description of items available in the Sample Code Repository is available on the web, however you must login to iceberg to access these examples.
If you would like to see additional samples or would like to contribute an example, please contact User Support.
The following programming models are available through compiler directives and/or libraries.
Hardware Level Model Description Shared-memory node OpenMP This is a form of explicit parallel programming in which the programmer inserts directives into the program to spawn multiple shared-memory threads, typically at the loop level. It is common, portable, and relatively easy. Shared-memory node pthreads The system supports POSIX threads. Distributed memory system MPI This is the most common and portable method for parallelizing codes for scalable distributed memory systems. MPI is a library of subroutines for message passing, collective operations, and other forms of interprocessor communication. The programmer is responsible for implementing data distribution, synchronization, and reassembly of results using explicit MPI calls.
Using MPI, the programmer can largely ignore the physical organization of processors into nodes and simply treat the system as a collection of independent processors.
Distributed memory system SHMEM This implements one-sided data passing, and other communication between tasks. Like MPI, SHMEM is a library of subroutines and the programmer must find and code all parallel work. SHMEM is supported on the IBM through the TurboMP library. Distributed memory system Hybrid Many of the models/methods described above can be combined in a single application. In particular:
- MPI and threads are explicitly combined by a growing number of programmers. For example one MPI task could be launched per p655+ node each of which would then spawn 8 threads to use all 8 processors on the node. Thread models available on iceberg include: pthreads and OpenMP.
man xlf, man xlc or
man xlC for additional options.
Compilers Standard Thread-Safe MPI Thread-Safe MPI Fortran 77 compiler xlf xlf_r mpxlf mpxlf_r Fortran 90 compiler xlf90 xlf90_r mpxlf90 mpxlf90_r Fortran 95 compiler xlf95 xlf95_r mpxlf95 mpxlf95_r C compiler xlc xlc_r mpcc mpcc_r C++ compiler xlC xlC_r mpCC mpCC_r NOTE: Thread-Safe compilers are recommended for all parallel code whether threads are explicitly used or not.
Other Programming Tools Executable Debuggers totalview, totalviewcli Performance analysis prof, gprof, tprof, xprofiler Batch queuing system LoadLeveler
Iceberg has the modules package installed. This tool allows a user to quickly switch between different versions of a package (e.g. compilers).
Before the modules package can be used on iceberg, the init file must first be sourced.
To do this using tcsh or csh, type:
source /usr/local/pkg/modules/init/<shell>To do this using bash, ksh, or sh, type:
. /usr/local/pkg/modules/init/<shell>For either case, replace <shell> with the shell you are using. If your shell is bash, for example:
. /usr/local/pkg/modules/init/bash
Once the modules init file has been sourced, the following commands become available:
Command Example Use Purpose module availmodule availlists all available modules for the system. module load pkgmodule load PrgEnvloads a module file from the environment module unload pkgmodule unload PrgEnvunloads a module file from the environment module listmodule listdisplays the modules which are currently loaded. module switch old newmodule switch PrgEnv PrgEnv.gccreplaces the module old with module new in the environment module purgemodule purgeunload all module settings, restoring the environment to the state before any modules were loaded.
The Fortran 90 compilers are:
Here are sample compiler commands showing several common options:
xlf90(standard Fortran Compiler)
mpxlf90(Fortran MPI Compiler)
mpxlf90_r(Thread Safe Fortran MPI Compiler)See table above or
man xlffor additional options.
xlf90_r program.f -o program(Compilation of a serial FORTRAN program)
xlf90_r program.f90 -o program(Compilation of a serial FORTRAN program with .f90 suffix)
xlf90_r openmp_prog.f -qsmp=omp -o openmp_prog(Compilation of a simple OpenMP program)
mpxlf90_r program.f -o program(Compilation of a simple MPI program)
See "man xlf" for more information and additional compiler options not listed here.
Common Fortran Compiler Directives: xlf option Description -q64 Enables 64 bit compilation. -qwarn64 Produces warnings for possible 32 bit data size issues. Useful when porting 32 bit code to 64 bit environment. -O3 High level of optimization. Note additional levels of optimization are available. See man xlffor more details-qhot Attempts to perform high order transformations. -qautodbl=dbl Promotes REALs to 64 bit double precision REALs. This can be useful when porting codes from systems where REALs are 64 bit by default. See man xlffor additional options.-qstrict Ensures that the semantics of a program are not altered when using -O3 -qmaxmem=<num> Specifies the memory limit in kilobytes used by space intensive optimizations. The special value -1 is used to indicate there is no limit to memory used by such optimizations. -bmaxdata:0x70000000 Specifies the number of bytes to reserve for the heap (dynamic memory allocation). In this case the data segment is 0x70000000 bytes or 1.75GB. Recommended values are 0x20000000 (0.50 GB) to 0x70000000 (1.75 GB). Programs requiring greater than 1.75 GB for the data segment should be compiled in 64 bit mode (-q64). -bmaxstack:0x70000000 Specifies the number of bytes to reserve for the stack**. In this case the stack is 0x70000000 bytes or 1.75 GB. Recommended values are 0x20000000 (0.50 GB) to 0x70000000 (1.75 GB). Programs requiring greater than 1.75 GB of stack space should be compiled in 64 bit mode (-q64). -g Produces debugging information. -p Adds profiling support code. -pg Adds profiling support code with BSD profiling support. -qfullpath Full path to source and include files is included in output files for debugging or profiling purposes. prog.f Name of the input Fortran source file. (File name extensions have an effect on compilation; the .F extension cause the cpp pre-processor to be invoked while .f does not.)
Many other compiler options are available. Check the man page for the compiler for more details. If you can't find something you'd expect, contact User Support for additional help.
The C compiler is accessible by the commands:
xlc (Standard C Compiler)
mpcc (C MPI Compiler)
mpcc_r (Thread Safe C MPI Compiler)
C++ uses the command:
xlC (Standard C++ Compiler)
mpCC (MPI C++ Compiler)
mpCC_r (Thread-Safe MPI C++ Compiler)
Sample Compilation:
xlc_r test.c -o test (Compilation of a serial C program)
xlc_r -qsmp=omp openmp_test.c -o openmp_test (Compilation of an OpenMP C program)
mpcc_r mpi_prog.c -o mpi_prog (Compilation of a MPI C program)
See "man xlc" for more information and additional compiler options.
Common C/C++ Compiler Directives: xlc/xlC option Description -q64 Enables 64 bit compilation. -qwarn64 Produces warnings for possible 32 bit data size issues. Useful when porting 32 bit code to 64 bit environment. -O3 High level of optimization. Note additional levels of optimization are available. See man xlffor more details-qstrict Ensures that the semantics of a program are not altered when using -O3 -qmaxmem=<num> Specifies the memory limit in kilobytes used by space intensive optimizations. The special value -1 is used to indicate there is no limit to memory used by such optimizations. -bmaxdata:0x70000000 Specifies the number of bytes to reserve for the heap (dynamic memory allocation). In this case the data segment is 0x70000000 bytes or 1.75GB. Recommended values are 0x20000000 (0.50 GB) to 0x70000000 (1.75 GB). Programs requiring greater than 1.75 GB for the data segment should be compiled in 64 bit mode (-q64). -bmaxstack:0x70000000 Specifies the number of bytes to reserve for the stack. In this case the stack is 0x70000000 bytes or 1.75 GB. Recommended values are 0x20000000 (0.50 GB) to 0x70000000 (1.75 GB). Programs requiring greater than 1.75 GB of stack space should be compiled in 64 bit mode (-q64). -g Produces debugging information. -p Adds profiling support code. -pg Adds profiling support code with BSD profiling support. (use with gprof and xprofiler) -qfullpath Full path to source and include files is included in output files for debugging or profiling purposes.
Many other compiler options are available. Check the man page for the compiler for more details. If you can't find something you'd expect, contact User Support for additional help.
-g -pg -qfullpath. This will add code to support profiling, and debugging.
Be sure to include the -pg flags on the linking line for you code.
xlf90 program.f90 -pg -g -qfullpath -qsuffix=f=f90 -o profiled-program
gmon.out), containing output statistics for the run.
./profiled-program
gprof > profiled-program.gprof_report
more profiled-program.gprof_report
See ARSC HPC Users' Newsletter 275 for an overview of xprofiler.
You are encouraged to use the LoadLeveler batch system, but may run interactive jobs which require less than 15 minutes of CPU time and run on a single node. An interactive command is simply typed at the prompt in a terminal window. Standard error and standard output may displayed on the terminal, redirected to a file, or piped to another command using appropriate Unix shell syntax.
Here are some examples of running serial and parallel jobs interactively.
./a.out
./a.out -procs 4 -hostfile host
The hostfile "host" has the names of the nodes to use. In this case it might contain the node b1n2 on four separate lines. The hostfile specifies which nodes to use in the interactive environment, so it is not necessary for jobs submitted through LoadLeveler.
poe ./a.out -procs 4 -hostfile host
All production work on iceberg is run through the LoadLeveler batch scheduler. Many users find that it is convenient to run short test jobs this way because memory, run-time, and CPU limits are larger, stdout/stderr are saved to file(s) for each run, and the test will continue even if the user logs off.
A batch job is a shell script (which could be as simple as two commands: a cd to the working directory and a run command like ./a.out) prefaced by a statement of resource requirements and instructions which LoadLeveler will use to manage the job.
LoadLeveler scripts are submitted for processing with the command llsubmit.As outlined below, five steps are common in the most basic batch processing:
In a batch script conforming to LoadLeveler syntax, all LoadLeveler options must precede all shell commands. Each line
containing a LoadLeveler option must commence with the character string, "# @" (spaces and tabs do not matter) followed by an
option. In addition the last LoadLeveler keyword must be "# @ queue".
This demonstrates the most common LoadLeveler options for running a parallel job. See the Using Loadleveler page for a more in depth description of these and other Loadleveler keywords.
#!/bin/csh
#
# @ environment = MP_SHARED_MEMORY=yes; COPY_ALL
# @ error = $(executable).$(jobid).err
# @ output = $(executable).$(jobid).out
# @ notification = error
# @ job_type = parallel
# @ node = 1
# @ tasks_per_node = 8
# @ network.MPI = sn_single,shared,us
# @ class = standard
# @ queue
./a.out
The script file can be given any name. If the above sample were named myprog.7.cmd, it would be submitted for processing with the command
llsubmit myprog.7.cmd
To check the status of the submitted LoadLeveler job, execute this command:
llq
Given its LoadLeveler job identification number (shown when you use "llq"), you can delete the job from the batch system with this command:
llcancel <LoadLeveler Job-ID>
When the job terminates, LoadLeveler will save the stdout and stderr from the job to a file in the directory from which it was submitted to the filenames listed in the "#@ error" and "#@ output" lines of your LoadLeveler script. These files will be named according to the naming convention provided in the loadleveler script. For example the stdout for a run of myprog using the example LoadLeveler script might be,
myprog.7102.out
LoadLeveler Classes
List all available classes (also called commonly called queues) with the command llclass. List details on any class, for instance, "standard," with the command llclass -l standard. You may also read news queues for information on all queues, but note that the most current information is always available using the llclass commands.ß
See Introduction to Using LoadLeveler for more example LoadLeveler scripts
for iceberg.
There are three different memory settings which need to be set appropriately to make best use of system memory on iceberg.
AIX can have memory settings included within the executable. The
-bmaxdataand-bmaxstackcan be used during linking to set the maximum space that the executable is allowed to use for the heap and stack. If these flags are not used the heap and stack will be limited to 256MB for 32 bit executables. Generally the heap setting (i.e.-bmaxdata) causes problems more frequently than the stack setting (i.e.-bmaxstack).
Example Use:# set maxdata to 1.75 GB xlf90_r -bmaxdata=0x70000000 test.o -o testThe
Example Use:-bmaxdataand-bmaxstackvalues for an existing executable can be altered without recompiling using theldeditcommand. (See also: ARSC HPC Users Newsletter issue 293 )# change the maxdata setting to 1.75 GB for an executable. ldedit -bmaxdata=0x70000000 testNOTE: The-bmaxdataand-bmaxstacksettings should not be used for programs compiled with-q64.
Even thought the
limit(csh/tcsh) andulimit(sh/ksh/bash) can be used to alter the soft and hard memory limits for a process, they should not be used for jobs run through Loadleveler. The shell memory limits within Loadleveler are set appropriately to allow all available memory on a node to be used by a single process, therefore there should not be a need to includelimitandulimitwithin Loadleveler script. Memory limits for batch jobs are enforced via Workload Manager.
WLM is enabled on iceberg to prevent jobs from using excessive amounts of memory. When a Loadleveler script is submitted a
ConsumableMemoryvalue will be automatically added to the script. For more information see:news memory
Users in more than one project can select an alternate project to charge use to by using the account_no
Loadleveler keyword. If the account_no keyword is not specified the account number will default to your primary
group (i.e. project). Users in a single project do not need to specify an account_no.
For Example:
# @ account_no = proja
This statement should be added to your Loadleveler script prior to the queue keyword.
Each project has a corresponding UNIX group, therefore the groups command will show all projects and other groups
of which you are a member.
For Example:
iceberg2 1% groups
proja projb
account_no = projb in the Loadleveler script.
All projects on iceberg have an allocation of CPU time on the system. CPU allocation and utilization information can be viewed
using the show_usage command. Please contact User Support with any
questions about project allocation.
For more information on the show_usage command see the Resource
Accounting page.
There are several debuggers available on iceberg including totalview, totalviewcli, gdb, dbx, and pdbx.
From news totalview on iceberg:
Totalview
=========
Totalview 6.4 is currently available on iceberg. This version will work with
MPI and openMP programs.
Totalview can be used in two ways.
* Using the front-end nodes on iceberg.
* Via Loadleveler using the 'debug' class.
I. Using the front-end nodes.
To use totalview on the front-end nodes, a host file is necessary:
1) Create a host file. This contains a list of hosts where totalview
will start your jobs (1 entry per processor). In this case, processes
will start on 'b1n2' so this file should contain one line of 'b1n2' for
each debug process you would like to start. Keep in mind that there are
8 processors on the front-end nodes and potentially multiple users
on the machine. Should you need to use more than 4 tasks (or threads)
you might consider using Loadleveler.
An example host file, for running no more than 4 debug processes:
iceberg1 1% cat Hostfile
b1n2
b1n2
b1n2
b1n2
2) Set the environment variable MP_HOSTFILE with the full path to your
host file. Below are example usage for both csh/tcsh and ksh.
csh/tcsh:
setenv MP_HOSTFILE ~/Hostfile
ksh:
export MP_HOSTFILE=~/Hostfile
You may want to put this setting in your .profile or .cshrc file.
3) Example four-processor run:
totalview poe -a ./a.out -procs 4
Note: the totalview flag -a tells totalview to pass all following
arguments to the user executable (in this case poe).
single processor (non-MPI) code can be started without poe:
totalview ./a.out
==========================================================================
II. Interactive using the debug class with Loadleveler.
1) To use Loadleveler you must provide a loadleveler command
file specifying the resources that will be used.
Note: The 'debug' class must be used when using totalview.
ARSC's current totalview license allows for up to 32 tasks.
iceberg1 2% cat lltotalview
#!/bin/csh
#
# @ job_type = parallel
# @ node = 2
# @ tasks_per_node = 8
# @ network.MPI = sn_single,shared,us
# @ class = debug
# @ wall_clock_limit = 0:30:00
# @ queue
##############################################
# Note: no executable need to be specified
# in this loadleveler script.
#
##############################################
2) Invoke totalview:
You will need to specify the Loadleveler command when starting poe.
The command file can be specified on the command line or using
the environment variable MP_LLFILE:
a) Command line version:
totalview poe -a ./my_mpi_exe -llfile ./lltotalview
The flag -llfile specifies the name of the Loadleveler command
file from step 1.
b) Using the environment variable MP_LLFILE
ksh:
export MP_LLFILE=./lltotalview
csh/tcsh:
setenv MP_LLFILE ./lltotalview
Once the environment variable is set you can start totalview
without using the -llfile flag.
totalview poe -a ./my_mpi_exe
NOTE: If there are not enough nodes available to start your job,
totalview will launch, but an error will be echoed to the
terminal.
For example:
ERROR: 0031-365 LoadLeveler unable to run job, reason:
LoadL_negotiator: 2544-870 Step b1n1.12345.0 was not considered
to be run in this scheduling cycle due to its relatively low
priority or because there are not enough free resources.
3) When you are done using totalview close the application. If loadleveler
doesn't release the nodes, you may need to cancel your job.
For example:
iceberg1 3% llq
Id Owner Submitted ST PRI Class Running On
------------------------ ---------- ----------- -- --- ------------ -----------
b1n1.12345.0 username 11/01 08:00 R 50 debug b7n1
llcancel b1n1.12345.0
Additional hints:
1) code should be compiled with -g. This makes it possible for
totalview to refer back to the source code. Code compiled without
-g will appear as assembler and you will not have meaningful access
to variable values.
2) if you compile your code in one location and run the executable in
another location you should also add the flag -qfullpath.
3) when starting jobs through poe, totalview will initially come up
with a screenful of assembler. Do not despair -- this is just the
poe executable. Click on 'go' to start the job. poe will start the
main run. A dialog box will appear asking whether you wish to stop the job
or continue. Stopping in this case means letting you set breakpoints, etc.
in your main code. Not stopping will run the code on the number of
processors you've specified until it either completes or one of the
processors halts at an error.
4) totalview works best with the _r compilers (i.e. mpxlf90_r)
5) source, object, and executable files must be in the same directory for
totalview to refer back to source lines by default. Otherwise:
a) add the directory/directories to the PATH environment variable
b) add paths with the File > Search Path command within totalview
6) you can view core files with totalview by passing the executable and
core file to totalview.
totalview ./a.out core -a -hostfile hostfile
For additional Documentation, see:
Totalview
- Documentation Sources
- Release 6, User Guide [Opens pdf in new window]
Active users may type the command:
iceberg1 25% news documentsTopics include:
Documentation is also available directly from IBM. Below are some links which may be of interest to iceberg users:
C / C++
- IBM C for AIX: C/C++ Language Reference, Version 6.0 (SC09-4958-00)
- VisualAge C++ for AIX Compiler Reference (SC09-4959-00)
- C for AIX Compiler Reference (SC09-4960-00)
- VisualAge C++ Standard C++ Library Reference (SC09-4949-00)
Fortran
- XL Fortran for AIX: User’s Guide, Version 8.1.1 (SC09-4946-01)
- XL Fortran for AIX: Language Reference, Version 8.1.1 (SC09-4947-01)
ESSL & PESSL
- Engineering and Scientific Subroutine Library for AIX, Version 4 Release 1 Guide and Reference (SA22-7904-01)
- Parallel Engineering and Scientific Subroutine Library for AIX, Version 3 Release 1, Guide and Reference (SA22-7906-01)
Loadleveler
Parallel Operating Environment (poe)
General information on ARSC and it's other resources is available in a number of forms
The ARSC web site covers policies, research, events, and technical information.
ARSC relies on news items and message of the day to relay system information and changes. Unread news items will be listed below the message of the day when logging into iceberg. All unread news items can be read by entering the command news at the prompt. Selected news items can be viewed using news news-item (e.g. news downtime and news queues).
This bi-weekly newsletter is designed for users of HPC systems. The current and all past issues are available at: http://www.arsc.edu/support/news/HPCnews.shtml.
The ARSC main library is located in West Ridge Research building, room 109A. We also maintain a library in the Duckering building.
ARSC offers several courses for users. The schedule is maintained on our website.
Please contact User Support if you have additional questions.
Arctic Region Supercomputing Center
PO Box 756020, Fairbanks, AK 99775 | voice: 907-450-8600 | email: 
home | search | about | support | news | science | resources